Databricks

Job Summary:We are seeking a highly skilled Senior Databricks Engineer to join our data team. The ideal candidate will have strong expertise in Python programming ETL processes and in-depth knowledge of Medallion Architecture in modern Data Lakehouse environments. You will play a key role in designing developing and optimizing scalable data pipelines and analytics solutions on the Databricks Responsibilities:Design develop and maintain robust ETL pipelines using Databricks and and optimize the Medallion Architecture (Bronze Silver Gold layers) within our Data Lakehouse with data engineers data scientists and business stakeholders to translate business requirements into scalable data data ingestion transformation cleansing and enrichment from various structured and unstructured data Spark jobs for performance and cost-efficiency on best practices for data governance security and quality within the data junior team members and contribute to improving team processes and and resolve data pipeline and platform-related issues Skills & Qualifications:10 years of experience in data engineering or a related proficiency in Python programming and libraries related to data processing (PySpark preferred).Hands-on experience with Databricks platform and Apache understanding of ETL concepts and implementation in large-scale data in Medallion Architecture and Data Lakehouse design with data storage technologies like Delta Lake Parquet and cloud data platforms (AWS Azure or GCP).Familiarity with SQL and performance tuning of Spark SQL problem-solving skills and attention to communication and collaboration Qualifications:Experience with containerization (Docker/Kubernetes) and orchestration tools (Airflow Azure Data Factory).Knowledge of CI/CD pipelines for data to machine learning pipelines and MLOps..