Senior Azure Data Engineer

Location: Redmond WA 98052
Work Setup: Hybrid
Duration: 12 Months
Experience: 810 Years

We are looking for an experienced Senior Azure Data Engineer with 810 years of experience in data engineering and strong hands-on expertise in Python SQL Apache Spark PySpark and Azure Databricks.

The candidate will be responsible for developing and supporting scalable data pipelines transforming large datasets and optimizing data processing solutions using Azure cloud technologies.

• Strong experience with Python and SQL
• Hands-on experience with Apache Spark PySpark and Spark SQL
• Strong experience with Azure Databricks including:
  • Notebooks
  • Jobs/Workflows
  • Clusters
• Experience with Azure Synapse
• Strong knowledge of Delta Lake
• Experience working with Parquet
• Hands-on experience with ADLS Gen2
• Strong understanding of data engineering and ETL/ELT concepts

• Experience with Scala
• ETL/ELT development
• Experience with MERGE/Upsert operations
• Knowledge of data partitioning and partition pruning
• Strong Spark performance tuning experience
• Experience handling data skew shuffles caching broadcast joins and partition optimization
• Experience with Git
• Experience with Azure DevOps and CI/CD

• Develop and maintain scalable ETL/ELT data pipelines
• Extract transform clean and load data from various source systems
• Build data solutions using Azure Databricks Spark Delta Lake Synapse and ADLS Gen2
• Implement MERGE/Upsert logic using Delta Lake
• Design effective partitioning strategies for large datasets
• Optimize Spark jobs for performance and cost
• Manage data skew shuffles caching partitions and cluster resources
• Collaborate with development and data teams to deliver reliable data solutions
• Follow best practices for Git Azure DevOps and CI/CD where applicable