Data Engineer Databricks, Python
Share
Job Location:
Monthly Salary:
Experience Required:
Posted on:
4 days ago
Vacancies:
1 Vacancy
Job Summary
Job Title
Data Engineer Databricks & Python
Location
Mississauga
Experience
46 years of professional experience in data engineering or data management
Key Responsibilities
Design build and maintain data pipelines using Databricks PySpark and Python
Support data warehousing initiatives including data integration transformation and cleansing
Develop and maintain data quality rules validations and mappings to ensure data accuracy and consistency
Write and optimize complex SQL queries for data extraction transformation and analysis
Troubleshoot and resolve data-related production issues ensuring system stability and data integrity
Work with cloud-based data platforms to support scalable data processing solutions
Collaborate with cross-functional teams to follow software development lifecycle (SDLC) best practices
Participate in Agile/Scrum ceremonies including sprint planning stand-ups and retrospectives
Support CI/CD pipelines for data workloads and automation
Provide ongoing operational support and continuous improvement of data systems
Required Skills & Qualifications
46 years of experience in data engineering data management or software development
Strong experience with Databricks Python and PySpark
Proficiency in writing complex SQL
Experience with data warehousing data integration and data cleansing
Working knowledge of cloud technologies
Experience troubleshooting data issues in Oracle and MS SQL environments (preferred)
Familiarity with CI/CD pipelines and automation for data workflows
Understanding of SDLC and Agile/Scrum methodologies
Strong problem-solving skills and ability to work effectively in a team environment
Nice-to-Have Skills
Experience with Perl
Exposure to DevOps practices for data platforms
Experience supporting production environments
Company Industry
IT Services and IT Consulting
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
