Data Engineer with Data Bricks with Pyspark and SQL knowledge
Job Location:
Job Location:
Bellevue, WA - USA
Monthly Salary:
Not Disclosed
Monthly Salary:
Posted on:
4 hours ago
Posted on:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
Data Bricks with Pyspark
SQL knowledge
Telecom
Must have Skills Data Engineering Data Pipeline Apache Spark Databricks Big Data Data Processing SQL
Skill 1 7Yrs of Exp Databricks Pyspark
Skill 2 7Yrs of Exp SQL Spark BigData
Skill 3 5Yrs of Exp in Telecom Domain
Good To have Skills
Skill 1 3 Yrs of Exp
Skill 2 3 Yrs of Exp
Skill 3 3 Yrs of Exp
Skill 4 3 Yrs of Exp -
Mandatory if Applicable
Domain Experience (If any ) Telecom
Number of days onsite 3
SQL knowledge
Telecom
Must have Skills Data Engineering Data Pipeline Apache Spark Databricks Big Data Data Processing SQL
Skill 1 7Yrs of Exp Databricks Pyspark
Skill 2 7Yrs of Exp SQL Spark BigData
Skill 3 5Yrs of Exp in Telecom Domain
Good To have Skills
Skill 1 3 Yrs of Exp
Skill 2 3 Yrs of Exp
Skill 3 3 Yrs of Exp
Skill 4 3 Yrs of Exp -
Mandatory if Applicable
Domain Experience (If any ) Telecom
Number of days onsite 3
JOB SUMMARY
This role builds and maintains scalable data pipelines and lakehouse infrastructure on Microsoft Azure to support efficient extraction transformation and loading of data across batch and real-time workloads. It involves implementing and managing the Medallion Architecture (Bronze Silver Gold) using Azure Data Factory Databricks-PySpark and Azure SQL Database and Databricks unity catalogue.
The role requires ensuring SLA-adherent data quality standards. Success is measured by pipeline reliability data freshness SLA compliance and the quality of Gold-layer datasets powering Power BI executive dashboards.
The work supports organizational decision-making by delivering trusted well-governed data to business executives and analytics consumers.
This role builds and maintains scalable data pipelines and lakehouse infrastructure on Microsoft Azure to support efficient extraction transformation and loading of data across batch and real-time workloads. It involves implementing and managing the Medallion Architecture (Bronze Silver Gold) using Azure Data Factory Databricks-PySpark and Azure SQL Database and Databricks unity catalogue.
The role requires ensuring SLA-adherent data quality standards. Success is measured by pipeline reliability data freshness SLA compliance and the quality of Gold-layer datasets powering Power BI executive dashboards.
The work supports organizational decision-making by delivering trusted well-governed data to business executives and analytics consumers.
Required Skills:
Experience building and optimizing big data pipelines using Azure Data Factory PySpark and SQL across structured and semi-structured data sets
Hands-on experience implementing Medallion Architecture (Bronze/Silver/Gold)
Experience with Delta Lake - ACID transactions incremental loading schema evolution partitioning strategies
Experience performing root cause analysis on pipeline failures and data quality issues to resolve SLA breaches and identify platform improvement opportunities
Experience building and optimizing big data pipelines using Azure Data Factory PySpark and SQL across structured and semi-structured data sets
Hands-on experience implementing Medallion Architecture (Bronze/Silver/Gold)
Experience with Delta Lake - ACID transactions incremental loading schema evolution partitioning strategies
Experience performing root cause analysis on pipeline failures and data quality issues to resolve SLA breaches and identify platform improvement opportunities
Azure Foundational Services :
Working knowledge of: Azure Data Factory (ADF) ADLS Gen2 Azure SQL Database Azure Blob Storage Azure Key Vault Azure Monitor / Log Analytics Azure Event Hubs Microsoft Fabric Lakehouse Azure Active Directory / Entra ID (RBAC Service Principals)
Working knowledge of: Azure Data Factory (ADF) ADLS Gen2 Azure SQL Database Azure Blob Storage Azure Key Vault Azure Monitor / Log Analytics Azure Event Hubs Microsoft Fabric Lakehouse Azure Active Directory / Entra ID (RBAC Service Principals)
Programming Languages:
Proficiency in Python and PySpark for data transformation pipeline automation and large-scale distributed processing; strong SQL skills including window functions CTEs and query optimization across relational and lakehouse engines
Proficiency in Python and PySpark for data transformation pipeline automation and large-scale distributed processing; strong SQL skills including window functions CTEs and query optimization across relational and lakehouse engines
Data Architecture:
Solid understanding of Medallion Architecture dimensional modeling (Star Schema SCD Types 1/2/3) and the trade-offs between lakehouse data warehouse and data lake patterns
Solid understanding of Medallion Architecture dimensional modeling (Star Schema SCD Types 1/2/3) and the trade-offs between lakehouse data warehouse and data lake patterns
Pipeline Engineering:
Ability to build robust ADF pipelines with ForEach Lookup Copy Activity and Data Flows; incremental loading via watermark or CDC; error handling retry logic and dead-letter patterns
Data Quality Experience:
Experience implementing SLA-based data quality checks (freshness completeness row count) monitoring via Azure Monitor and ADF diagnostic logs and defining data quality agreements with business stakeholders.
Ability to build robust ADF pipelines with ForEach Lookup Copy Activity and Data Flows; incremental loading via watermark or CDC; error handling retry logic and dead-letter patterns
Data Quality Experience:
Experience implementing SLA-based data quality checks (freshness completeness row count) monitoring via Azure Monitor and ADF diagnostic logs and defining data quality agreements with business stakeholders.
DevOps for Data:
Experience with Git-based workflows ADF Git integration CI/CD pipeline promotion across Dev/Test/Prod using Azure DevOps or GitHub Actions
Experience with Git-based workflows ADF Git integration CI/CD pipeline promotion across Dev/Test/Prod using Azure DevOps or GitHub Actions
Reporting Layer Awareness:
Understanding of how Gold-layer data feeds Power BI - DirectQuery vs. Import mode trade-offs dataset refresh patterns and semantic model collaboration with BI teams
Ability to manage work across multiple concurrent pipeline projects prioritize by business impact and communicate status clearly to technical and non-technical stakeholders
Understanding of how Gold-layer data feeds Power BI - DirectQuery vs. Import mode trade-offs dataset refresh patterns and semantic model collaboration with BI teams
Ability to manage work across multiple concurrent pipeline projects prioritize by business impact and communicate status clearly to technical and non-technical stakeholders
Good to have skills:
Experience with Microsoft Fabric (Lakehouse Notebooks OneLake Fabric Pipelines) - active migration or greenfield project
Experience with real-time / streaming workloads using Azure Event Hubs or Structured Streaming in PySpark
Experience delivering data platforms for executive-level reporting via Power BI semantic models
Experience with Microsoft Fabric (Lakehouse Notebooks OneLake Fabric Pipelines) - active migration or greenfield project
Experience with real-time / streaming workloads using Azure Event Hubs or Structured Streaming in PySpark
Experience delivering data platforms for executive-level reporting via Power BI semantic models
