Data Engineering with Medallion Architecture
Designing pipelines that move data from Bronze Silver Gold Implementing schema evolution CDC and data quality checks
Understanding standards for each layer (e.g. Bronze raw Gold curated KPis)
Spark and SQL Expertise
Writing performant Spark SQL and PySpark transformations
Optimizing joins window functions and aggregations
Handling large-scale data with partitioning and caching strategies
Delta Live Tables (DLT) and Workflows
Lakeflow Declarative Pipelines
DevOps and CI/CD
Using Databricks Asset Bundles (DABs) for deployment
Integrating with Git Bitbucket and Jenkins for version control and automation