Senior Data Engineer (Azure Databricks)
Senior Data Engineer (Azure Databricks)
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
About the Role
Talpro is seeking a Senior Data Engineer (Azure Databricks) for an internal position to be deployed at a client location. This is a 6-month contract-to-hire role if you excel during the contract there is potential to transition into a permanent position. You will be on Talpro s payroll and work with the client s team in Bangalore. The work model is hybrid (mix of on-site and remote as needed) and aligned to UK business hours (approximately 2 PM to 7 PM IST).
Key Responsibilities
Data Pipeline Development: Design and implement highly performant metadata-driven data ingestion and transformation pipelines from multiple sources using Azure Databricks and Apache Spark. This includes building both streaming and batch data processes to handle large-scale data in real time and in bulk.
Framework & Optimization: Develop scalable and reusable frameworks for data ingestion and transformation of large datasets. Perform Spark performance tuning and optimization to ensure data pipelines are efficient and cost-effective. Provide technical guidance on complex data problems (including handling geospatial data) using Spark DataFrames and other big data tools.
Data Quality & Integration: Design and implement data quality checks and processes to ensure data integrity is maintained through the pipeline. Integrate end-to-end data pipelines from source systems to target data repositories (data lakes/data warehouses) ensuring consistency quality and reliability of data at all times.
Collaboration & Tooling: Work closely with cross-functional teams (data analysts data scientists application developers) to support additional project components such as reporting tools APIs and search applications. Evaluate various tools and technologies (e.g. Azure Data Factory Azure Databricks features etc.) against project requirements to recommend the best fit.
Agile Delivery: Operate within an Agile/DevOps delivery environment. Participate in iterative sprint planning deliver proof-of-concepts (POCs) and guide the project from concept to production. Implement continuous integration/continuous deployment (CI/CD) practices for data pipelines and ensure solutions are delivered in a timely iterative manner.
Requirements
Required Qualifications
Experience: 6 8 years of overall Data Engineering experience preferably in building data solutions on Microsoft Azure. Within this at least 3 4 years of hands-on Databricks experience including delivering 2 end-to-end Databricks projects from concept to production.
Azure Databricks Expertise: Proven experience in designing Databricks-based data platforms with a strong understanding of Spark internals performance tuning and best practices for building data lakes/warehouses on Databricks. Familiarity with addressing non-functional requirements (scalability reliability security) in a data platform design.
Data Pipeline Skills: Hands-on experience building data pipelines using Azure Data Factory and Azure Databricks. Proficient in developing data integration workflows using Python (including PySpark) for ETL/ELT processes.
Certifications: Databricks Certified Data Engineer Associate (at least one Databricks certification is required). Microsoft Azure Data Engineer Associate certification is highly desired.
Data Warehousing & ETL: Experience designing and delivering solutions on the Azure Data Analytics stack (e.g. Azure Data Lake Azure Synapse Analytics). Background in building data warehouse solutions using ETL/ELT tools such as Informatica or Talend.
Data Management: Strong understanding of data management best practices data profiling data cleansing data transformation and data quality assurance. Experience implementing routines for data standardization deduplication and matching to ensure high data quality.
Preferred Qualifications (Nice to Have)
DevOps & CI/CD: Experience working in a DevOps-oriented environment using tools like Azure DevOps (formerly VSTS) Chef Puppet or Terraform for infrastructure automation and deployment. Knowledge of setting up CI/CD pipelines for data projects is a plus.
Diverse Data Experience: Exposure to working with structured and unstructured data including experience handling geospatial data or large imaging datasets. Ability to incorporate such data types into pipelines and ensure performance.
NoSQL & Big Data Tech: Familiarity with open-source NoSQL databases and graph databases (e.g. MongoDB Cassandra Neo4j) and understanding of how to integrate or migrate data to/from these systems.
Streaming & IoT: Experience with streaming data platforms and event ingestion services such as Azure Event Hub Azure IoT Hub Apache Kafka or Apache NiFi. Ability to design pipelines for real-time data processing and event-driven architectures.
Why Join Talpro
Innovative Projects: Work on cutting-edge data engineering projects using Azure Databricks and the modern Azure data stack tackling challenges in big data and analytics.
Contract-to-Hire Opportunity: The 6-month C2H model allows you to prove your skills with the possibility of a long-term role. You ll get a foot in the door with a leading client and the stability of Talpro s backing.
Flexible Work Environment: Enjoy a hybrid work setup with the flexibility of remote work combined with collaborative on-site days. Align with UK timing for better work-life balance if you prefer later start times.
Professional Growth: Leverage your expertise and certifications while also having opportunities to learn new technologies (cloud services DevOps tools streaming platforms). Talpro supports continuous learning and certification for our employees.
Note: This position is for Bangalore-based candidates (or those willing to relocate) as on-site presence is required on a case-to-case basis. Immediate joiners are preferred due to project urgency.
If you are a seasoned data engineer passionate about building scalable data platforms with Azure Databricks and meet the above qualifications we encourage you to apply. Join Talpro and contribute to transformative data solutions in a dynamic collaborative environment.
procurement, supplier portal/isupplier, sourcing modules
Employment Type
Full Time
