Data Engineer
Data Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
EXPERIENCE AND EDUCATION:
Essential Qualifications/Experience:
3 years of professional experience in software development and/or data engineering.
Advanced knowledge of the Python programming language and its ecosystem especially for data engineering
Proven experience in developing using cloud services (Azure)
Professional experience in processing very large climatology datasets up to several terabyte per dataset with cloud based resources employing horizontal and vertical scalability HPC patterns and Python ecosystem
Experience on the requirements above can be supported with:
Experience with one or more industry standard distributed data systems: Databricks Apache Kafka Azure Event Hub Azure Machine Learning
Experience with developing backend APIs using Python frameworks especially for complex web applications
Experience using containers (Docker) and orchestration technologies (kubernetes)
Experience in developing workflows using tools (Apache Airflow)
Experience in manipulating storing and visualizing geospatial/satellite/climatology data
Support the STOCMRE Research Division and Engineering Division datarelated projects based on NATO approved Cloud environments
Setup and management of data engineering pipelines data processing machine learning frameworks and tools cloud based HPC clusters and data cataloguing
Software Development in Python configuration of pipelines and application of DevOps tools using Microsoft Azure Docker Kubernetes GitLab
Deployment of dockerized application with focus on security
Employment Type
Full Time
