Data Engineer
Share
Job Location:
Columbus, NE - USA
Monthly Salary:
Not Disclosed
Posted on:
30+ days ago
Vacancies:
1 Vacancy
Job Summary
We are seeking a skilled and motivated Data Engineer to play a key role in building scalable data pipelines and preparing data for advanced AI use cases. This role will focus on leveraging Azure DataBricks and implementing the medallion architecture to support robust data engineering workflows.
Key Responsibilities
Design develop and maintain scalable data pipelines using Azure DataBricks.
Implement and optimize the medallion architecture (bronze silver gold layers) to support AI/ML use cases.
Collaborate with data scientists analysts and business stakeholders to understand data requirements.
Ensure data quality integrity and governance across all layers of the architecture.
Automate data ingestion transformation and enrichment processes.
Monitor and troubleshoot data workflows and performance issues.
Document data engineering processes and contribute to best practices.
Key Responsibilities
Design develop and maintain scalable data pipelines using Azure DataBricks.
Implement and optimize the medallion architecture (bronze silver gold layers) to support AI/ML use cases.
Collaborate with data scientists analysts and business stakeholders to understand data requirements.
Ensure data quality integrity and governance across all layers of the architecture.
Automate data ingestion transformation and enrichment processes.
Monitor and troubleshoot data workflows and performance issues.
Document data engineering processes and contribute to best practices.
Required Skills & Qualifications
Proven experience with Azure DataBricks and the medallion architecture.
Strong proficiency in Python SQL and Spark.
Experience preparing data for AI/ML models including feature engineering and data normalization.
Familiarity with Delta Lake Azure Synapse and Azure Data Factory.
Understanding of data governance security and compliance principles.
Excellent problem-solving and communication skills.
Bachelor s degree in Computer Science Engineering or related field (or equivalent experience).
Proven experience with Azure DataBricks and the medallion architecture.
Strong proficiency in Python SQL and Spark.
Experience preparing data for AI/ML models including feature engineering and data normalization.
Familiarity with Delta Lake Azure Synapse and Azure Data Factory.
Understanding of data governance security and compliance principles.
Excellent problem-solving and communication skills.
Bachelor s degree in Computer Science Engineering or related field (or equivalent experience).
Preferred Qualifications
Experience with MLflow Unity Catalog or other MLOps tools.
Knowledge of CI/CD pipelines for data engineering.
Familiarity with real-time data processing and streaming architectures.
Experience with MLflow Unity Catalog or other MLOps tools.
Knowledge of CI/CD pipelines for data engineering.
Familiarity with real-time data processing and streaming architectures.
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
