Senior Data Engineer
Senior Data Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
Senior Data Engineer - Quality Engineering
Experience Range: 7-10 Years
Location: Bangalore (Hybrid Mode)
Resillion a leading quality engineering company with offices around the world is seeking a talented Data Engineer to join our growing India team. In this role you will play a critical part in building and maintaining the data infrastructure that supports our AI-powered testing tools and analytics solutions. You will have technical responsibility for the entire data lifecycle from data acquisition and ingestion to storage processing and analysis.
Responsibilities:
Collaborate with GenAI engineers Software Engineers Test automation engineers and other stakeholders within Resillion to understand data needs and translate them into technical solutions including designing data pipelines for training & deploying models and data pre-processing for AI including generative AI applications.
Design implement and advise on data migration testing strategies and data quality assurance strategies for Resillion customers ensuring a smooth transition of data to new customer systems.
Design develop and implement scalable data pipelines using cloud-based data engineering tools and technologies with a focus on both Microsoft Azure solutions (e.g. Azure Data Factory Azure Databricks) and Google Cloud Platform (GCP) solutions (e.g. Google Cloud Dataflow Google Cloud Dataproc).
Write efficient and maintainable code to extract transform and load data from various sources leveraging your expertise in Azure Data Lake Storage and other relevant Azure services as well as Google Cloud Storage and other relevant GCP services.
Build and manage data warehouses and data lakes for quality engineering data utilizing your knowledge of Azure Synapse Analytics or similar technologies and Google BigQuery or similar technologies.
Develop and implement data quality checks and monitoring procedures to ensure data integrity using Azure Data Catalog or other appropriate tools as well as Google Cloud Data Catalog or other appropriate tools.
Automate data engineering tasks and workflows using Azure automation tools and GCP automation tools (e.g. Cloud Functions Cloud Composer).
Set up quality intelligence dashboards for quality assurance data using Microsoft Power BI to provide stakeholders with clear and actionable insights.
Stay up-to-date with the latest data engineering tools and technologies including advancements in AI Machine Learning (MLOps) and generative AI for data processing in both the Azure and GCP environments.
Advise on and implement test data generation strategies and solutions for various testing needs.
Qualifications :
Minimum 7 years of experience in data engineering or a related field
Proven experience in designing developing and deploying data pipelines
Strong understanding of data warehousing data lakes and data modelling concepts
Expertise in SQL and scripting languages (e.g. Python)
Experience with cloud platforms (specifically Microsoft Azure and Google Cloud Platform) and their data engineering services (Azure Data Factory Databricks Data Lake Storage Synapse Analytics and Google Cloud Dataflow Dataproc Cloud Storage BigQuery etc.)
Experience working with data formats commonly used in software development (e.g. logs code repositories) is a plus
Experience with data migration strategies and data migration testing
Experience in defining and implementing data testing strategies and data quality assurance processes particularly in data migration projects.
Experience in advising on and implementing test data generation techniques and tools.
Experience with MLOps practices and tools for deploying and monitoring machine learning models.
Experience with generative AI concepts and their application in data processing and analysis.
Experience with Microsoft Power BI for data visualization
Excellent communication and collaboration skills
Ability to work independently and as part of a team
Additional Information :
Why Join Us
- Work in a dynamic and rapidly evolving field.
- Competitive compensation benefits and growth opportunities.
- Make a significant impact on the organizations overall posture.
Remote Work :
No
Employment Type :
Full-time
Employment Type
Full-time
