Big DataBlog

How To Become A Data Engineer In 2020

Mellivora Software boasts its stellar staff in each of the core expertise directions. In this brief interview, we decided to ask our lead Data Architect & CTO, Alexander Dolnik, what skills should a person have in order to become a data engineer. Is a specialized education a must? Does one need a college or university degree to become a data engineer? All the answers uncovered!

 

Question 1. What is Data Engineering? What tasks does Data Engineering solve?

Data engineering is a very broad area. We typically face tasks like database architecture creation, managing data, data migration, working with master data, identifying new data, data mining, data analysis, etc. I prefer the term “data management”, because all that we have is data, and whatever we do with it, is us basically manipulating that data to reach some specific goal.

 

Question 2. What skills should a data engineer have? 

It depends on the task that you are going to solve. For instance, if you work with a simple creation of a database, you should know such a programmatic language as SQL. I prefer an academic background in data engineering, because you should understand the whole business area, you should understand the task definition clearly, and also you should understand how your data will interact with the entire system. Data engineers should understand industry domain and business models clearly, because only then they will be able to implement architecture in the best way, to satisfy business needs of their customers.

Regarding the skills, it really depends on the task. For instance, for some cases, you might only know what relative database is and know SQL, but in general you should know much more, for example:

  • How to build graph databases
  • How to run clusters
  • How to build ETL (extract, transform, load) process
  • How to build DWH (data warehouses)
  • Script languages (Python)
  • Java, Scala, SQL
  • Understanding of tech and business requirements

 

Question 3. What steps should one take to become a data engineer?

The first question is: why? 🙂 IT people should clearly understand why they want to move to data engineering, and what tasks they are going to solve. Because, as I already mentioned, data engineering is a broad area. For instance, data engineer for Hadoop cluster systems should know how cluster systems work, what is a general approach, maybe find the most popular tech stack and understand what tool would work best for that tech stack. Also, it will be useful to understand how ETL works and how to build it, what a relative database is, and have a general IT background.

If a person defined their area of interest, the basic steps they could take are as follows:

  • Define a direction
  • Define a tech stack you need to master (for that direction)
  • Master the skills you need (online & offline courses, mentorship etc)
  • Gain practical experience in that field (practice, pet projects, production projects etc)

 

Question 4. Does data engineering have a future? Will it be in high demand in the next 10 years?

It’s a great question. When we talk about data engineering, we should understand that it is basically a term for how to prepare the data, how to store the data, and how to extract the data from different sources. And I strongly believe that if we want to process the data, we do need data engineers. I can say definitely that in the next 10 years?.. We will still be needing data engineers. 

We are working with data that can be stored in different formats, in different databases, in different systems, and I do believe that we need specialists who will understand how to work with data, how to operate with different systems, and how to move the data from one system to another. I think that the main data formats we operate with might change though: it might be visual and video information mostly.

Data engineering always faces new challenges, new tasks and new business areas. I am definitely sure that data engineering will be in high demand in the next decade.

 

Watch full episode here:

Video. How To Become A Data Engineer In 2020

Liked the article?.. Stay tuned for our next articles on Data Engineering in Mellivora Software’s blog!

3,144 thoughts on “How To Become A Data Engineer In 2020

  1. Hello there, I do think your web site may be having browser compatibility problems. Whenever I look at your web site in Safari, it looks fine however, when opening in Internet Explorer, it has some overlapping issues. I merely wanted to give you a quick heads up! Besides that, great website!

  2. Oh my goodness! a fantastic short article dude. Thank you However I am experiencing concern with ur rss. Don?t understand why Incapable to subscribe to it. Is there anyone obtaining the same rss issue? Anybody that recognizes kindly react. Thnkx

  3. Greetings I am so excited I found your webpage,
    I really found you by error, while I was searching on Yahoo
    for something else, Regardless I am here now and would just like to say thanks for
    a incredible post and a all round interesting blog (I also love the theme/design),
    I don’t have time to read through it all at the minute but I have bookmarked it
    and also added in your RSS feeds, so when I have time I will be back to read more, Please do keep up the awesome
    job.

  4. Have you ever thought about publishing an e-book or guest authoring
    on other websites? I have a blog based upon on the same
    ideas you discuss and would love to have you share some
    stories/information. I know my visitors would value your work.

    If you’re even remotely interested, feel free to send me an email.

  5. I blog frequently and I truly appreciate your information. The article has really peaked my interest. I am going to book mark your website and keep checking for new information about once a week. I subscribed to your Feed too.

  6. Hey There. I found your blog using msn. This is a really well written article.
    I will make sure to bookmark it and come back to read more of your useful info.
    Thanks for the post. I will certainly comeback.