Engineering

Job DescriptionOverallmso-spacerun:yes>  6 8 years of relevant be flexible to adapt to project requirements Responsibilities:Data Pipeline Development: Design build and optimize robust scalable and efficient ETL/ELT data pipelines using Python and PySpark primarily within Azure Databricks and Azure Data Ingestion & Processing: Develop and manage processes for ingesting data from various sources (e.g. transactional databases APIs streaming sources) and transforming it into clean usable formats for downstream Quality & Monitoring: Implement comprehensive unit and integration test coverage for data pipelines. Establish and maintain monitoring alerting and dashboarding solutions (e.g. Grafana) for data quality pipeline health and Infrastructure Management (OpenShift/Azure): Contribute to the setup configuration and maintenance of data-related infrastructure on OpenShift ensuring deployment readiness and leveraging tools like HELM for application packaging and & Automation: Drive CI/CD best practices using GitHub Actions ensuring automated testing (unit tests) build and deployment processes for data solutions to environments like & Data Modeling: Develop and optimize complex SQL queries for data extraction transformation and loading. Apply strong data modeling principles for efficient data storage and retrieval in SQL Server and other data Ecosystem Leverage: Utilize a broad range of Azure data and analytics services including Azure Data Factory Azure Databricks Azure SQL Server Azure Key Vault and others to build comprehensive data Optimization: Proactively identify and resolve performance bottlenecks in data pipelines and databases through query optimization indexing strategies and efficient data processing & Documentation: Work closely with data scientists analysts and other engineering teams to understand data requirements. Create clear and concise documentation for data pipelines architecture and Core Skills & Qualifications:Programming & Data Processing:Strong proficiency in Python and PySpark for large-scale data processing and ETL Warehousing & SQL:Expertise in SQL for complex querying data manipulation and schema design.(Optional but highly preferred): Proven experience in SQL optimization and performance Development:Demonstrable experience in designing building and maintaining robust ETL/ELT data Data Platform (Azure Focus):Hands-on experience with Azure with core Azure Analytics Services including Azure Data Factory Azure SQL Server and Azure Key & CI/CD:Experience implementing CI/CD pipelines from GitHub (including GitHub Actions) for automated testing (unit tests) build and deployment & Orchestration:Familiarity and practical experience with OpenShift (setup deployment-ready configurations and management).Experience with HELM for deploying applications on Kubernetes/ & Observability:Experience in setting up and configuring Grafana for dashboards to monitor data quality and pipeline Qualifications:Bachelors or Masters degree in Computer Science Engineering Data Science or a related quantitative Azure certifications (e.g. Azure Data Engineer Associate).Experience with real-time data processing frameworks (e.g. Kafka Azure Event Hubs).Understanding of data governance data security and compliance best practices.

Qualifications :

BEMCAM Tech

Additional Information :

6-8

Remote Work :

No

Employment Type :

Full-time