Data Engineer
Data Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
Job Title:Data Engineer
Location:Mumbai Vikhroli.
Job Type:Fulltime
Position Summary:We are seeking a skilled Data Engineer with expertise in Databricks on Azure (ADF ADLS) to join our data team. As a Data Engineer you will work with both structured and unstructured data. You will be responsible for designing building maintaining scalable and reliable data pipelines that support business intelligence data analytics and machine learning efforts. You will collaborate closely with data scientists analysts and crossfunctional teams to ensure data is available accurate and optimized for processing and storage.
Job Responsibilities:
Data Pipeline Development: Design develop and optimize data pipelines to ingest process and transform data from various sources (e.g. APIs databases files) into the data warehouse.
Data Integration:Integrate data from various structured and unstructured sources into the Databricks Lakehouse environment ensuring data accuracy and reliability
Data Lakehouse & storage Management:Design and maintain data warehouse solutions using medallion architecture practices optimizing storage cloud utilization costs and query performance
Collaboration with Data Teams: Work closely with data scientists analysts to understand requirements translate them into technical solutions and implement data solutions.
Data Quality and Monitoring: Cleanse transform and enrich data. Implement data quality checks and establish monitoring processes to ensure data integrity and accuracy. Implement monitoring for data pipelines and troubleshoot any issues or failures promptly to ensure data reliability.
Optimization and Performance Tuning:Optimize data processing workflows for performance reliability and scalability including tuning spark jobs caching and partitioning data appropriately.
Data Security and Privacy:Manage and organize data lakes using Unity catalog ensuring proper governance security rolebased access and compliance with data management policies
Preferred Qualifications:
- Education:Bachelors degree in Computer Science Engineering/MCA or a related field (Masters preferred)
- 3 years of experience as a Data Engineer with handson experience in Databricks.
Technical Skills:
- Proficiency with Databricks Lakehouse platform Delta Lake Genie ML Flow (e.g. Databricks Certified Data Engineer Associate) is a plus.
- SQL and NoSQL: Experienced working with both SQL and NoSQL data sources (e.g. MySQL PostgreSQL MongoDB etc.
- Strong knowledge of Spark especially in PySpark or Scala for data transformation.
- Proficiency in Python R and other programming languages used in data processing.
- Experience with cloud platforms like Azure AWS particularly Azure storage & services
- Knowledge of ML Pipelines data streaming platforms (e.g. Apache Kafka AWS Kinesis).
- Familiarity with data visualization tools (e.g. Tableau Power BI Looker).
Required Experience:
Manager
Employment Type
Full Time
