Data Engineer (Databricks)
Experience Level5 8 Years
Gurugram Bengaluru Hyderabad
Good To Have Skills
Change Management tool DevOps
Must Have Skills
Pyspark Azure ADF Databricks ETL SQL
Job Description
- Design and build data pipelines using SparkSQL and PySpark in Azure Databricks
- Design and build ETL pipelines using ADF
- Build and maintain a Lakehouse architecture in ADLS / Databricks.
- Perform data preparation tasks including data cleaning normalization deduplication type conversion etc.
- Work with DevOps team to deploy solutions in production environments.
- Control data processes and take corrective action when errors are identified.
- Corrective action may include executing a work around process and then identifying the cause and solution for data errors.
- Participate as a full member of the global Analytics team providing solutions for and insights into data related items.
- Collaborate with your Data Science and Business Intelligence colleagues across the world to share key learnings leverage ideas and solutions and to propagate best practices.
- You will lead projects that include other team members and participate in projects led by other team members.
- Apply change management tools including training communication and documentation to manage upgrades changes and data migrations.
MustHave Skills:
Azure Databricks Azure Data Factory PySpark. Spark SQL
ADLS