SeniorLead Data Engineer Databricks

JDSenior Data Engineer:-

Senior/Lead Data Engineer - Databricks We are seeking a highly skilled and experienced Data Engineering Lead with a strong background in the retail domain and exceptional programming abilities. As a Lead you will play a pivotal role in implementing and optimizing data architecture to support our retail business operations and analytics initiatives. Your expertise in Spark programming optimization techniques and familiarity with Databricks and CI/CD practices will be instrumental in ensuring the efficient and effective management of our data ecosystem.

Responsibilities:-

• Design and develop data models data integration processes and data pipelines to capture transform and load structured and unstructured data from various retail sources.
• Hands-on programming in Spark to develop and optimize data processing applications and analytics workflows.
• Apply optimization techniques to enhance the performance and efficiency of data processing and analytical tasks.
• Evaluate and implement appropriate tools and technologies including Databricks to streamline data operations and ensure scalability and reliability.
• Work closely with other team members to ensure data integrity consistency and accessibility across the organization.
• Define and enforce best practices for data governance and data management including data quality metadata management and data security.
• Collaborate with DevOps teams to establish and maintain CI/CD pipelines for data engineering and analytics workflows.
• Peer Review of team members deliverables
• Stay updated with the latest advancements and trends in the retail domain data architecture and programming languages to drive continuous improvement.

Requirements:-

• At least 5 years of experience in the data engineering domain.
• Proven experience as a senior Data Engineer /Lead preferably within the retail industry.
• Strong programming skills with expertise in PySpark programming and optimization techniques.
• Hands-on experience with Databricks Deltalake and its components for data processing and analytics.
• Hands-on experience in data modelling data integration and ETL/ELT processes.
• Experience in working with Gitlab pipelines and an in-depth understanding of CI/CD pipeline designs.
• Experience with data governance data quality and metadata management.
• Strong analytical and problem-solving abilities with a detail-oriented mindset.
• Excellent communication and collaboration skills to work effectively with cross-functional teams.
• Ability to adapt to a fast-paced and evolving environment while managing multiple priorities.
• Good to have experience in at least one of the Cloud Vendor (AWS / Azure / GCP)
• Good to have experience with streaming technologies as well

• Professional Development and Mentorship.
• Hybrid work mode with remote friendly workplace. (6 times in a row Great Place To Work Certified).
• Health and Family Insurance.
• 40 Leaves per year along with maternity & paternity leaves.
• Wellness meditation and Counselling sessions.