Challenges and Essential Skills to Become a Successful Data Engineer
Data Engineering is a relatively new field compared to other roles in the technology sector. Universities do not yet have a formal path to train a Data Engineer.
Although there are many bootcamps and courses available, there is still no established path for our field.
Becoming a Data Engineer can be challenging for this reason; those who decide to pursue it must be aware that much of the necessary knowledge will be acquired through independent research. We must get used to being self-taught in various related areas and new technologies that emerge daily.
If you are a professional transitioning from related fields such as Software Engineering, Data Science, Data Analysis, or others, your transition will certainly be smoother.
That said, we can divide the skills into two groups:
Interpersonal/Business Skills
Technical Skills
Among the Interpersonal Skills, I highlight the following:
The Ability to Communicate with Non-Technical Professionals
Many times in my career, a project either didn’t happen or didn’t turn out ideally due to communication failures between the business teams and the Data Engineering team.
Understanding what a stakeholder needs goes far beyond endless alignment meetings. Often, people don’t know what they want. Here lies the danger of delivering a product that was requested but doesn’t meet the team’s actual needs.
This is why it is important to know the business you are working in and its nuances. This allows us to put ourselves in the shoes of the people involved and shape what they really want.
Another important point is to focus on the result, not the tools, when sharing an achievement. Few vice presidents or CEOs care about the number of Spark Clusters you used or which language you employed to perform the data transformation.
Focus on results!
“Our optimized pipeline reduced cloud costs for this process by 25%.”
“The data modeling we chose reduced the sales dashboard update time by 30 seconds.”
Taking this approach was a turning point in my career. Knowing how to sell yourself is extremely important in any professional field.
Save the technical terms for your Tech Lead or teammates.
Knowing How to Say No!
As mentioned earlier, many people don’t know what they want. This can lead to conflicts and constant changes in a project. It’s essential to have a well-defined deliverable and move forward with it until it’s at least minimally viable to make changes. If we keep listening to endless requests, we’ll never finish a project.
That’s why it’s important to know how to say no and focus your work on things that will truly add value.
Many companies have well-structured product teams, which minimizes this problem. However, I believe that in most companies, engineers deal directly with the end users of their data products. In these companies, especially, knowing how to say no is even more essential!
Understand Agile Methodologies
Familiarity with these methodologies can save you from exhaustive work without controlled deliverables. Especially in smaller companies, sometimes it’s necessary to take on the role of a Product Manager and oversee the progress of projects. This isn’t the ideal scenario—I believe Data Engineers should focus on developing data solutions—but having this knowledge can save you some headaches!
Control Costs
Anyone working with cloud servers knows how easy it is to blow up costs if we’re not attentive to certain details. Always aim to deliver exceptional value at the lowest possible cost. After all, a company thrives on profits, and the cheaper a process is, the better for them.
Here, we must be careful to differentiate between subpar, sloppy work and exceptional work done at a low cost. Always strive for excellence, but aim to do so with the cheapest tool possible.
With the rise of the Big Data era, many companies are killing ants with a cannon!
They use extremely expensive tools to handle small volumes of data. Be careful with this! Simplicity is sometimes the ideal path.
Always Learn
New tools that can help us perform Data Engineering tasks emerge every day. Stay up to date with market trends. But above all, build a strong foundation. In my next post, I will talk precisely about these essential skills and how they impact and help a Data Engineer’s life!
See you next time!