Senior Databricks Data Engineer

Stefanini EMEA

Job Location:

Bucharest - Romania

Monthly Salary: Not Disclosed

Posted on: 7 hours ago

Vacancies: 1 Vacancy

Job Summary

Job Description

We are looking for a skilled Senior Databricks Data Engineer to join our team.The position is focused on the development implementation and optimization of complex Data Warehouse (DWH) and Data Lakehouse solutions using the Databricks platform (including Delta Lake Unity Catalog and Spark) to ensure a scalable high-performance and governed data foundation for analysis reporting and Machine Learning.

Responsibilities

Databricks Development and Architecture

Advanced Design and Implementation: Designing and implementing robust scalable and high-performance ETL/ELT data pipelines using PySpark/Scala and Databricks SQL on the Databricks platform.
Delta Lake: Expertise in implementing and optimizing the Medallion architecture (Bronze Silver Gold) using Delta Lake to ensure data quality consistency and historical accuracy.
Lakehouse Platform: Efficient implementation of the Lakehouse architecture on Databricks combining best practices from Data Warehousing (DWH) and Data Lakes.
Performance Optimization: Optimizing Databricks clusters Spark operations and Delta tables (e.g. Z-ordering Compaction Tuning Queries) to reduce latency and computing costs.
Streaming: Designing and implementing real-time/quasi-real-time data processing solutions using Spark Structured Streaming and Delta Live Tables (DLT).

Governance and Security

Unity Catalog: Implementing and administering
Unity Catalog for centralized data governance fine-grained security (row/column-level security) and data lineage.
Data Quality: Defining and implementing data quality standards and rules (e.g. using DLT or Great Expectations) to maintain data integrity.

Operations and Collaboration

Orchestration: Developing and managing complex workflows using Databricks Workflows (Jobs) or external tools (e.g. Azure Data Factory Airflow) for pipeline automation.
DevOps/CI/CD: Integrating Databricks pipelines into CI/CD processes using tools like Git Databricks Repos and Bundles.
Collaboration: Working closely with Data Scientists Analysts and Architects to understand business requirements and provide optimal technical solutions.
Mentorship: Providing technical guidance (mentorship) to junior developers and promoting best practices.

Required Knowledge (Expert Level)

Databricks Platform: Proven expert-level experience with the entire Databricks ecosystem (Workspace Cluster Management Notebooks Databricks SQL).
Apache Spark: In-depth knowledge of Spark architecture (RDD DataFrames Spark SQL) and advanced optimization techniques.
Delta Lake: Expertise in implementing and managing Delta Lake (ACID properties Time Travel Merge Optimize Vacuum).
Programming Languages: Advanced/expert-level proficiency in Python (with PySpark) and/or Scala (with Spark).
SQL: Advanced/expert-level skills in SQL and Data Modeling (Dimensional 3NF Data Vault).
Cloud: Solid experience with a major Cloud platform (AWS Azure or GCP) particularly with storage services (S3 ADLS Gen2 GCS) and networking.

Additional Knowledge (Major Advantage)

Unity Catalog: Practical experience with implementing and managing Unity Catalog.
Lakeflow: Experience with Delta Live Tables (DLT) and Databricks Workflows.
ML/AI Concepts: Understanding of basic MLOps concepts and experience with MLflow to facilitate integration with Data Science teams.
DevOps: Experience with Terraform or equivalent for Infrastructure as Code (IaC).
Certifications: Databricks certifications (e.g. Databricks Certified Data Engineer Professional) are a significant advantage.

Education and Experience

Education: Bachelors degree in Computer Science Engineering Mathematics or a relevant technical field.
Professional Experience: Minimum of 5 years of experience in Data Engineering with at least 3 years using Databricks and Spark at scale.

The preceding job description had been designed to indicate the general nature and level of work performed by employees within this classification. It is not designed to contain or be interpreted as a comprehensive inventory of all duties and responsibilities required of employees assigned to this job.

What we offer:

You will find here not only a challenging and interesting workplace but also a rewarding work experience with competitive compensation and benefit packages:

Soft skills and technical training for current and emerging products and technologies;
Career development: opportunity to grow within the team;
Special referral bonuses for recommending your friends;
Private Medical subscription;
A monthly budget for flexible benefits program that includes public transportation reimbursement and meal tickets sport & wellness telecom & electronics children & parents holiday & culture tourism;
Flexible working hours and work from home aligned with project needs;
Friendly team who is eager to meet you.

Whats next:

Its best to apply today because job postings can be taken down and we wouldnt want you to miss this case you need further information just send us a message at and well be happy to assist!

Diversity & Inclusion

Here at the Stefanini Group we value plurality and equity regardless of race sexual orientation disability age ancestry religion gender and nationality. We understand and encourage the importance of being you!

About us

We are the Stefanini group a global tech consulting company of Brazilian origin that believes in the power of people to transform businesses through technology.

We are present in over 40 countries and operate with the purpose of co-creating solutions TOGETHER WITH OUR CLIENTS that accelerate results and improve the experience of people and organizations.

Here we like to say that technology is not the end but the means: what really matters are the people who drive it all.

Our mindset is AI First meaning we invest in cutting-edge technology in everything we do focusing on results for our clients.

We are a company A GROUP that breathes collaboration and offers a dynamic environment where you will learn by doing grow alongside the team and have space to contribute with ideas and projects.

More than just talking about digital transformation we believe in real transformation that starts with people and impacts real businesses.

If you are looking for a place to develop innovate and be part of something bigger the Stefanini Group is your place.

We want to inform you that there are currently scams targeting job seekers by falsely using our companys name Stefanini. We sincerely apologize for any confusion or inconvenience this may have caused.

Please remember that legitimate job offers from Stefanini will always come through official channels including direct communication with our trained recruiters. If you receive any unsolicited messages requesting payment or personal information please disregard them.

If you suspect youve been targeted please contact us immediately at for verification.

Key Points to Remember:

* Legitimate job offers only follow interviews conducted with our hiring managers or clients.

* We will never ask for payment at any stage of the recruitment process.

Stay vigilant and feel free to reach out for verification. Your safety and security are our top priorities. Thank you for your understanding and cooperation.

#LI-HYBRID

Required Experience:

Senior IC

Job DescriptionWe are looking for a skilled Senior Databricks Data Engineer to join our team.The position is focused on the development implementation and optimization of complex Data Warehouse (DWH) and Data Lakehouse solutions using the Databricks platform (including Delta Lake Unity Catalog and S...