Databricks Engineer
Databricks Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Title:- Databricks Senior Data Engineer
Location: NYC NY (2 days/ week on-site)
As a Senior Data Engineer within this department you will play a critical role in designing and developing robust ETL pipelines to integrate diverse data sources. This role requires expertise in creating efficient data pipelines handling both streaming and batch data processing and ensuring data integrity throughout the ETL lifecycle. The engineer will implement monitoring solutions optimize ETL jobs for performance and provide comprehensive support from data ingestion to final output. With in-depth knowledge of the Databricks platform and strong analytical skills this position will significantly enhance the departments ability to deliver high-quality data-driven insights and solutions.
Key Skills
- Expertise in designing and developing ETL code to integrate various data sources.
- Proficiency in creating efficient data pipelines on the Databricks platform.
- Experience in developing maintaining and optimizing ETL pipelines.
- Ability to handle both streaming and batch data processing.
- Provide comprehensive support for ETL processes from data ingestion to final output.
- Ensure data integrity and consistency throughout the ETL lifecycle.
- Write and execute unit tests and integration test cases for ETL code.
- Ensure high-quality code through rigorous testing practices.
- Implement monitoring solutions for ETL jobs to ensure timely and successful data processing.
- Proactively identify and resolve issues in ETL workflows.
- Optimize ETL jobs for maximum performance and efficiency.
- Conduct performance tuning and troubleshooting of ETL processes.
- Minimum 10 years of experience in this field
Preferred Skills
- In-depth knowledge of Databricks platform and technologies Delta Lake Databricks SQL and Databricks Workflows.
- Experience with Azure cloud platforms
- Familiarity with Azure Data Lake cloud storage cloud storage
- Knowledge of data warehousing data modeling and best practices.
- Proficiency in programming languages such as Python SQL Scala or R.
- Experience with big data technologies such as Apache Spark Hadoop or Kafka.
- Knowledge of working with Hadoop for migration projects from Hadoop to Databricks
- Familiarity with DevOps practices and tools such as CI/CD Git etc.
- Knowledge of infrastructure as code (IaC) tools like Terraform
- Experience with implementing data governance and security measures in a cloud environment.
Communication and Soft Skills
- Strong verbal and written communication skills.
- Ability to explain complex technical concepts to non-technical stakeholders.
- Team player with the ability to work collaboratively in a cross-functional team.
- Experience working in an Agile/Scrum environment.
- Strong analytical and problem-solving skills.
- Ability to troubleshoot and resolve issues in a timely manner.
- Flexibility to adapt to changing project requirements and priorities.
- Willingness to learn new technologies and continuously improve skills.
Employment Type
Full-time
