Data Engineer(ETL, SQL, SSIS, Python, AWS)

Virtusa

Job Location:

Chennai - India

Monthly Salary: Not Disclosed

Posted on: 6 hours ago

Vacancies: 1 Vacancy

Job Summary

Job Description: Senior Data Engineer (AWS & Databricks)
Role Overview
We are looking for a Senior Data Engineer to architect and execute the migration of our data estate from SQL Server to an AWS-based Databricks Lakehouse. You will lead the transition from legacy stored procedures and manual processes into scalable PySpark pipelines. This role is critical for establishing our Medallion Architecture (Bronze/Silver/Gold) on AWS.
Key Responsibilities
Migration Leadership: Lead the technical migration strategy for moving data and logic from on-prem/RDS SQL Server to AWS S3 and Databricks.
Pipeline Engineering: Build high-performance ETL/ELT pipelines using PySpark and Scala leveraging Databricks Workflows and Unity Catalog.
AWS Integration: Configure and optimize data ingestion patterns using AWS Glue (Crawlers) S3 and IAM for secure data access.
Code Quality: Set standards for Python/Scala development including unit testing and CI/CD integration via AWS CodePipeline or GitHub Actions.
Required Skills & Qualifications
8 years in Data Engineering with at least 3 years focused on AWS.
Expert level: PySpark and Databricks (Delta Lake Photon engine Liquid Clustering).
Strong Proficiency: SQL Server (T-SQL) for reverse-engineering legacy logic and optimizing source extraction.
Languages: Mastery of Python and SQL; professional experience with Scala.
Cloud Infrastructure: Hands-on experience with AWS S3 IAM and Secrets Manager.
Nice to Have
Experience reading/decoding SSIS packages to extract business logic.
Databricks Certified Data Engineer Professional.

Required Experience: