Intermediate Senior Data Engineer (Databricks)
Share
Job Location:
Centurion - South Africa
Monthly Salary:
Not Disclosed
Posted on:
30+ days ago
Vacancies:
1 Vacancy
Job Summary
We are looking for a Data Engineer who is certified in Databricks (required) to join our this role you will be designing developing and optimizing scalable data pipelines and workflows on Databricks. The engineer will work closely with stakeholders to make certain data reliability performance and alignment with business requirements.
Scope of Work
Data Pipeline Development:
- Building efficient ETL/ELT pipelines using Databricks and Delta Lake for structured semi-structured and unstructured data.
- Transforming raw data into consumable datasets for analytics and machine learning.
Data Optimization:
- Improving performance by implementing best practices like partitioning caching and Delta Lake optimizations.
- Resolving bottlenecks and ensuring scalability.
Data Integration:
- Integrating data from various sources such as APIs databases and cloud storage systems (e.g. AWS S3 Azure Data Lake).
Real-Time Streaming:
- Designing and deploying real-time data streaming solutions using Databricks Structured Streaming.
Data Quality and Governance:
- Implementing data validation schema enforcement and monitoring to ensure high-quality data delivery.
- Using Unity CatLog to manage metadata access permissions and data lineage.
Collaboration and Documentation:
- Collaborating with data analysts data scientists and other stakeholders to meet business needs.
- Documenting pipelines workflows and technical solutions.
Responsibilities
Fully functional and documented data pipelines.
Optimized and scalable data workflows on Databricks.
Real-time streaming solutions integrated with downstream systems.
Detailed documentation for implemented solutions and best practices.
Skills and Qualifications
Proficiency in Databricks(certified) Spark and Delta Lake.
Strong experience with Python SQL and ETL/ELT development.
Familiarity with real-time data processing and streaming.
Knowledge of cloud platforms (e.g. AWS Azure GCP).
Experience with data governance and tools like Unity CatLog.
Assumptions
Access to necessary datasets and cloud infrastructure will be provided.
Timely input and feedback from stakeholders.
Success Metrics
Data pipelines deliver accurate and consistent data.
Workflows meet performance benchmarks.
Real-time streaming solutions operate with minimal latency.
Stakeholders are satisfied with the quality and usability of the solutions.
Key Skills
- Counselling
- ABB
- Fiber Optic
- Client Services
- Business Analytics
- Brokerage
AI AutoApply
Apply to 100+ jobs with one click
AI Resume Builder
Create an ATS-ready CV in minutes
AI Cover Letter
Write a personalized letter instantly
Related Jobs
Senior Project Manager – Wheeling (Renewable Energy)
Johannesburg Metropolitan Municipality
