Mellivora Software boasts its stellar staff in each of the core expertise directions. In this brief interview, we decided to ask our lead Data Architect & CTO, Alexander Dolnik, what skills should a person have in order to become a data engineer. Is a specialized education a must? Does one need a college or university degree to become a data engineer? All the answers uncovered!
Question 1. What is Data Engineering? What tasks does Data Engineering solve?
Data engineering is a very broad area. We typically face tasks like database architecture creation, managing data, data migration, working with master data, identifying new data, data mining, data analysis, etc. I prefer the term “data management”, because all that we have is data, and whatever we do with it, is us basically manipulating that data to reach some specific goal.
It depends on the task that you are going to solve. For instance, if you work with a simple creation of a database, you should know such a programmatic language as SQL. I prefer an academic background in data engineering, because you should understand the whole business area, you should understand the task definition clearly, and also you should understand how your data will interact with the entire system. Data engineers should understand industry domain and business models clearly, because only then they will be able to implement architecture in the best way, to satisfy business needs of their customers.
Regarding the skills, it really depends on the task. For instance, for some cases, you might only know what relative database is and know SQL, but in general you should know much more, for example:
- How to build graph databases
- How to run clusters
- How to build ETL (extract, transform, load) process
- How to build DWH (data warehouses)
- Script languages (Python)
- Java, Scala, SQL
- Understanding of tech and business requirements
Question 3. What steps should one take to become a data engineer?
The first question is: why? 🙂 IT people should clearly understand why they want to move to data engineering, and what tasks they are going to solve. Because, as I already mentioned, data engineering is a broad area. For instance, data engineer for Hadoop cluster systems should know how cluster systems work, what is a general approach, maybe find the most popular tech stack and understand what tool would work best for that tech stack. Also, it will be useful to understand how ETL works and how to build it, what a relative database is, and have a general IT background.
If a person defined their area of interest, the basic steps they could take are as follows:
- Define a direction
- Define a tech stack you need to master (for that direction)
- Master the skills you need (online & offline courses, mentorship etc)
- Gain practical experience in that field (practice, pet projects, production projects etc)
Question 4. Does data engineering have a future? Will it be in high demand in the next 10 years?
It’s a great question. When we talk about data engineering, we should understand that it is basically a term for how to prepare the data, how to store the data, and how to extract the data from different sources. And I strongly believe that if we want to process the data, we do need data engineers. I can say definitely that in the next 10 years?.. We will still be needing data engineers.
We are working with data that can be stored in different formats, in different databases, in different systems, and I do believe that we need specialists who will understand how to work with data, how to operate with different systems, and how to move the data from one system to another. I think that the main data formats we operate with might change though: it might be visual and video information mostly.
Data engineering always faces new challenges, new tasks and new business areas. I am definitely sure that data engineering will be in high demand in the next decade.
Watch full episode here:
Video. How To Become A Data Engineer In 2020
Liked the article?.. Stay tuned for our next articles on Data Engineering in Mellivora Software’s blog!