Lead Data Engineer
Lead Data Engineer
Posted on :
03-09-2025
Employer Active
1 Vacancy
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Monthly Salary
Not Disclosed
Salary Not Disclosed
Vacancy
1 Vacancy
Posted on :
03-09-2025
Job Description
Job Title: Data Engineer
Location: Greenfield IN (Onsite)
Duration: Long term contract/ FTE
10-15 years
4 years of experience in Azure Databricks with PySpark.
2 years of experience in Databricks workflow & Unity catalog.
3 years of experience in ADF (Azure Data Factory).
3 years of experience in ADLS Gen 2.
3 years of experience in Azure SQL.
5 years of experience in Azure Cloud platform.
2 years of experience in Python programming & package builds.
2 years of experience in Databricks workflow & Unity catalog.
3 years of experience in ADF (Azure Data Factory).
3 years of experience in ADLS Gen 2.
3 years of experience in Azure SQL.
5 years of experience in Azure Cloud platform.
2 years of experience in Python programming & package builds.
Key technical skills :
Data management experience handling Analytics workload covering design development and maintainenance of lakehouse solutions sourcing data from platforms such as ERP sources API sources Relational stores NoSQL and on-prem sources using Databricks/PySpark as distributed /big data management servicesupporting batch and near-real-time ingestion transformation and processing.
Ability to optimize Spark jobs and manage large-scale data processing using RDD/DataFrame expertise in partitioning strategies file format optimization (Parquet/Delta) and Spark SQL with Databricks runtime versions cluster policies libraries and workspace management.
Skilled in governing and manage data access for Azure Data lakehouse with Unity Catalog. Experience in configuring data permissions object lineage and access policies with Unity of integrating Unity Catalog with Azure AD external metastores and audit trails.
Experience in building efficient orchestration solutions using Azure data factory Databricks to design modular reusable workflows using tasks triggers and in using dynamic expressions parameterized pipelines custom activities and triggers.
Familiarity with integration runtime configurations pipeline performance tuning and error handling strategies.
Strong experience in implementing secure hierarchical namespace-based data lake storage for structured/semi-structured data aligned to bronze-silver-gold layers with ADLS Gen2. Hands-on experience with lifecycle policies access control (RBAC/ACLs) and folder-level security. Understanding of best practices in file partitioning retention management and storage performance optimization.
Capable of developing T-SQL queries stored procedures and managing metadata layers on Azure SQL.
Comprehensive experience working across the Azure ecosystem including networking security monitoring and cost management relevant to data engineering of VNets Private Endpoints Key Vaults Managed Identities and Azure to DevOps tools for deployment automation (e.g. Azure DevOps ARM/Bicep/Terraform).
Experience in writing modular testable Python code used in data transformations utility functions and packaging reusable with Python environments dependency management (pip/Poetry/Conda) and packaging to write unit tests using PyTest/unittest and integrate with CI/CD pipelines.
Lead solution design discussions mentor junior engineers and ensure adherence to coding guidelines design patterns and peer review to prepare Design documents for development and guiding the team technically. Experience preparing technical design documents HLD/LLDs and architecture with code quality tools (e.g. SonarQube pylint) and version control workflows (Git).
Demonstrates strong verbal and written communication proactive stakeholder engagement and a collaborative attitude in cross-functional to articulate technical concepts clearly to both technical and business in working with product owners QA and business analysts to translate requirements into deliverables.
Data management experience handling Analytics workload covering design development and maintainenance of lakehouse solutions sourcing data from platforms such as ERP sources API sources Relational stores NoSQL and on-prem sources using Databricks/PySpark as distributed /big data management servicesupporting batch and near-real-time ingestion transformation and processing.
Ability to optimize Spark jobs and manage large-scale data processing using RDD/DataFrame expertise in partitioning strategies file format optimization (Parquet/Delta) and Spark SQL with Databricks runtime versions cluster policies libraries and workspace management.
Skilled in governing and manage data access for Azure Data lakehouse with Unity Catalog. Experience in configuring data permissions object lineage and access policies with Unity of integrating Unity Catalog with Azure AD external metastores and audit trails.
Experience in building efficient orchestration solutions using Azure data factory Databricks to design modular reusable workflows using tasks triggers and in using dynamic expressions parameterized pipelines custom activities and triggers.
Familiarity with integration runtime configurations pipeline performance tuning and error handling strategies.
Strong experience in implementing secure hierarchical namespace-based data lake storage for structured/semi-structured data aligned to bronze-silver-gold layers with ADLS Gen2. Hands-on experience with lifecycle policies access control (RBAC/ACLs) and folder-level security. Understanding of best practices in file partitioning retention management and storage performance optimization.
Capable of developing T-SQL queries stored procedures and managing metadata layers on Azure SQL.
Comprehensive experience working across the Azure ecosystem including networking security monitoring and cost management relevant to data engineering of VNets Private Endpoints Key Vaults Managed Identities and Azure to DevOps tools for deployment automation (e.g. Azure DevOps ARM/Bicep/Terraform).
Experience in writing modular testable Python code used in data transformations utility functions and packaging reusable with Python environments dependency management (pip/Poetry/Conda) and packaging to write unit tests using PyTest/unittest and integrate with CI/CD pipelines.
Lead solution design discussions mentor junior engineers and ensure adherence to coding guidelines design patterns and peer review to prepare Design documents for development and guiding the team technically. Experience preparing technical design documents HLD/LLDs and architecture with code quality tools (e.g. SonarQube pylint) and version control workflows (Git).
Demonstrates strong verbal and written communication proactive stakeholder engagement and a collaborative attitude in cross-functional to articulate technical concepts clearly to both technical and business in working with product owners QA and business analysts to translate requirements into deliverables.
Communication Skills:
Communicate effectively with internal and customer stakeholders
Communication approach: verbal emails and instant messages
Interpersonal Skills:
Strong interpersonal skills to build and maintain productive relationships with team members
Provide constructive feedback during code reviews and be open to receiving feedback on your own code.
Problem-Solving and Analytical Thinking:
Capability to troubleshoot and resolve issues efficiently.
Analytical mindset.
Task/ Work Updates
Prior experience in working on Agile/Scrum projects with exposure to tools like Jira/Azure DevOps.
Provides regular updates proactive and due diligent to carry out responsibilities.
Communication approach: verbal emails and instant messages
Interpersonal Skills:
Strong interpersonal skills to build and maintain productive relationships with team members
Provide constructive feedback during code reviews and be open to receiving feedback on your own code.
Problem-Solving and Analytical Thinking:
Capability to troubleshoot and resolve issues efficiently.
Analytical mindset.
Task/ Work Updates
Prior experience in working on Agile/Scrum projects with exposure to tools like Jira/Azure DevOps.
Provides regular updates proactive and due diligent to carry out responsibilities.
We are seeking a highly skilled Data Engineering specialist with above mentioned mentioned Primary Skills to join our dynamic team who are at the forefront of enabling enterprises in Healthcare sectors.
The ideal candidate should be passionate about working on Data Engineering on Azure cloud with strong focus on DevOps practices in building product for our customers.
Effectively Communicate and Collaborate with internal teams and customer to build code leveraging or building low level design documents aligning to standard coding principles and guidelines.
The ideal candidate should be passionate about working on Data Engineering on Azure cloud with strong focus on DevOps practices in building product for our customers.
Effectively Communicate and Collaborate with internal teams and customer to build code leveraging or building low level design documents aligning to standard coding principles and guidelines.
Good to have Azure Entra/AD skills and GitHub Actions.
Good to have orchestration experience using Airflow Dagster LogicApp.
Good to have expereince working on event-driven architectures using Kafka Azure Event Hub.
Good to have exposure on Google Cloud Pub/Sub.
Good to have experience developing and maintaining Change Data Capture (CDC) solutions preferrably using Debezium.
Good to have hands-on experience on data migration projects specifically involving Azure Synapse and Databricks Lakehouse.
Good to have eperienced in managing cloud storage solutions on Azure Data Lake Storage . Experience with Google Cloud Storage will be an advantage.
Good to have orchestration experience using Airflow Dagster LogicApp.
Good to have expereince working on event-driven architectures using Kafka Azure Event Hub.
Good to have exposure on Google Cloud Pub/Sub.
Good to have experience developing and maintaining Change Data Capture (CDC) solutions preferrably using Debezium.
Good to have hands-on experience on data migration projects specifically involving Azure Synapse and Databricks Lakehouse.
Good to have eperienced in managing cloud storage solutions on Azure Data Lake Storage . Experience with Google Cloud Storage will be an advantage.
Employment Type
Full-time
Key Skills
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
