Databricks Engineer, Data Platform & Data Science Tooling

Job descriptions may display in multiple languagesbased on your language selection.

Job Responsibilities Must Have

• Non-negotiable skill: 3 years hands-on: PySpark Delta Lake Workflows Unity Catalog Databricks
• Design develop and implement efficient and reliable data pipelines using Databricks Delta Live Tables.
• Develop pipelines for structured and unstructured data (i.e. documents JSON Parquet Excel) supporting AI and ML consumption downstream.
• Implement and extend data models (i.e. fact/dimension tables domain data marts) following designs defined by the Senior DE and AI team.
• Write clean modular reusable PySpark and SQL transformation logic that is testable documented and deployable via CI/CD
• Write and optimize complex SQL queries using Databricks SQL for data extraction transformation and loading.
• Implement and manage data governance and metadata using Databricks Unity Catalog to ensure data quality and discoverability.
• Collaborate with data scientists analysts and other engineers to understand data requirements and deliver appropriate data solutions.
• Monitor troubleshoot and optimize existing data pipelines and data infrastructure.
• Stay up-to-date with the latest trends and technologies in data engineering and the Databricks ecosystem.
• Contribute to the semantic layer that powers Power BI dashboards.

Orchestration and Data Ops- Should Have

• Build and manage Databricks Workflows: configuring task dependencies retry policies and failure alerting
• Monitor & manage workspaces as Workspace admins.
• Follow and contribute to CI/CD practices: version control pull requests automated testing and deployment to Dev/QA/Prod environments using Azure DevOps or GitHub Actions.
• Understanding of infrastructure as a service Terraform and willingness to learn terraform for automation at infrastructure level.
• Package and deploy reusable logic as Python libraries following team standards
• Monitor pipeline health investigate failures and resolve data issues within SLA

FinOps Awareness Should Have

• Write cost-conscious PySpark avoiding unnecessary full scans optimizing joins using appropriate cluster types
• Apply Delta table best practices (i.e. VACUUM OPTIMIZE compaction) to manage storage costs
• Follow cluster policies defined by platform leads and flag unusual resource consumption
  

Experience/QUALIFICATIONS

• 4-6 years of overall data engineering experience
• 2 years of hands-on Azure Databricks experience in production environments
• Demonstrated ability to build and deliver pipelines not just maintain or support them
• Experience working within a defined architecture and contributing to its improvement
• Comfortable working with multiple data source types relational file-based API

At Magna we believe that a diverse workforce is critical to our success. Thats why we are proud to be an equal opportunity employer. We hire on the basis of experience and qualifications and in consideration of job requirements regardless of in particular color ancestry religion gender origin sexual orientation age citizenship marital status disability or gender identity. Magna takes the privacy of your personal information seriously. We discourage you from sending applications via email or traditional mail to comply with GDPR requirements and your local Data Privacy Law.

As part of our commitment to a fair consistent and efficient recruitment process we may use artificial intelligence (AI) tools to assist in the initial screening of applications submitted through our Workday system. These tools help identify qualifications and experience that align with the role requirements. Please note that AI is used solely to support our recruiters. Final decisions are always made by the hiring manager and the hiring team. Importantly no applicant data is shared externally through these AI tools. All information remains securely within our systems and is handled in accordance with our privacy and data protection policies.

If you have any questions or concerns about this process feel free to contact our Talent Attraction team.