H-GAG Data Engineer
Share
Job Location:
Houston, MS - USA
Monthly Salary:
Not Disclosed
Posted on:
10 hours ago
Vacancies:
1 Vacancy
Job Summary
Job Summary:
The Data Engineer will be responsible for building and maintaining data ingestion and transformation pipelines to migrate and process structured and semi-structured data from on-premise SQL servers and file systems into an Azure Data Lakehouse environment. This role involves configuring data factories developing ETL/ELT pipelines implementing data cleansing and validation logic optimizing performance and coordinating with the BI team to ensure dataset readiness.
The Data Engineer will be responsible for building and maintaining data ingestion and transformation pipelines to migrate and process structured and semi-structured data from on-premise SQL servers and file systems into an Azure Data Lakehouse environment. This role involves configuring data factories developing ETL/ELT pipelines implementing data cleansing and validation logic optimizing performance and coordinating with the BI team to ensure dataset readiness.
Location:Houston Texas United States
Responsibilities:
- Configure Azure Data Factory / Fabric Data Factory for batch and incremental ingestion.
- Develop ETL/ELT pipelines for on-prem SQL sources.
- Implement data cleansing validation and schema evolution logic.
- Optimize performance through partitioning and parallel loads.
- Coordinate with the BI team for Gold-layer dataset readiness.
- Document pipeline configurations and error-handling mechanisms.
- Configure Azure Data Factory / Fabric Data Factory for batch and incremental ingestion.
- Develop ETL/ELT pipelines for on-prem SQL sources.
- Implement data cleansing validation and schema evolution logic.
- Optimize performance through partitioning and parallel loads.
- Coordinate with the BI team for Gold-layer dataset readiness.
- Document pipeline configurations and error-handling mechanisms.
Required Skills & Certifications:
- 5 years of experience in Azure Data Factory Synapse or Fabric Data Engineering.
- Strong SQL and Python skills for data transformations.
- Experience with Medallion or Kimball modeling patterns.
- Familiarity with Azure Monitor and pipeline logging frameworks.
- 5 years of experience in Azure Data Factory Synapse or Fabric Data Engineering.
- Strong SQL and Python skills for data transformations.
- Experience with Medallion or Kimball modeling patterns.
- Familiarity with Azure Monitor and pipeline logging frameworks.
Preferred Skills & Certifications:
- Not specified.
- Not specified.
Special Considerations:
- Not specified.
- Not specified.
Scheduling:
- Not specified.
- Not specified.
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
