DataOpsCloud Data Engineer
Job Location:
Monthly Salary:
Experience Required:
Posted on:
Vacancies:
Job Summary
DataOps / Cloud Data Engineer
Overview
The DataOps / Cloud Data Engineer is responsible for designing developing testing implementing and supporting enterprise data solutions across cloud and hybrid environments. The role focuses on building scalable data pipelines data lakehouse and data warehouse solutions and automated data integration processes that enable analytics reporting and business intelligence initiatives.
Responsibilities
Data Engineering & Pipeline Development
- Design develop test implement and troubleshoot enterprise data pipelines.
- Build and maintain data ingestion transformation and orchestration processes using cloud-native and ETL tools.
- Develop complex data transformation procedures to support business and operational reporting requirements.
- Process and integrate structured and unstructured data from multiple sources.
- Implement data quality validation monitoring and error-handling mechanisms.
- Optimize data workflows for performance scalability and reliability.
Cloud Data Platform Development
- Develop and support cloud-based data solutions using AWS and/or Azure services.
- Build and maintain data lakehouse environments using Databricks Delta Lake.
- Develop cloud-based data warehouse solutions using platforms such as AWS Redshift.
- Implement automated data processing and orchestration workflows using services such as AWS Glue Step Functions Lambda S3 and Azure Data Factory.
- Support cloud migration and data modernization initiatives.
Data Warehousing & Modeling
- Design and maintain relational and dimensional data models.
- Develop and support enterprise data warehouses data marts and analytical datasets.
- Implement ETL/ELT processes using tools such as Informatica IDMC Azure Data Factory and AWS Glue.
- Support reporting and visualization requirements by providing optimized and governed data assets.
Operations & Support
- Participate in production support troubleshooting incident resolution and root-cause analysis.
- Support change management and deployment activities across development testing and production environments.
- Ensure adherence to security governance and data management standards.
Project Delivery & Stakeholder Collaboration
- Participate in all phases of the Software Development Life Cycle (SDLC) including requirements gathering design development testing deployment and support.
- Collaborate with business stakeholders architects analysts and project teams to deliver data solutions.
- Work within Agile/Scrum and Waterfall project methodologies.
- Provide technical guidance and recommendations related to data architecture and engineering best practices.
Required Skills and Experience
Data Engineering (40%)
- Strong programming and scripting experience with Python SQL Linux Shell and PowerShell.
- Experience using Pandas and PySpark for data processing and analysis.
- Experience working with XLSX CSV JSON files relational databases cloud storage and structured/unstructured data.
- Experience developing and maintaining enterprise data pipelines.
Cloud Technologies (25%)
- Experience with AWS and/or Azure cloud platforms.
- Experience with AWS services including Glue Step Functions Lambda S3 and Redshift.
- Experience with Databricks Delta Lake and lakehouse architectures.
- Experience implementing automated cloud data processing solutions.
Data Warehousing & ETL (25%)
- Experience with ETL/ELT tools such as Informatica IDMC Azure Data Factory and AWS Glue.
- Strong understanding of relational and dimensional data modeling.
- Experience supporting enterprise reporting and analytics environments.
- Knowledge of data architecture and warehouse design principles.
Professional Skills (5%)
- Full SDLC experience.
- Agile/Scrum and Waterfall project delivery experience.
- Change and Incident Management experience.
- Strong communication presentation negotiation problem-solving and decision-making skills.
Public Sector Experience (5%)
- Experience working within the Ontario Public Sector (OPS) and/or broader public sector environments is considered an asset.
Mandatory Requirements
- Data engineering experience using Pandas and PySpark.
- Experience with Databricks Delta Lake and lakehouse solutions.
- Experience working with XLSX CSV JSON files relational databases cloud storage and structured/unstructured data.
- Experience with AWS services including Glue Step Functions Lambda and S3.
- Experience extracting transforming and loading data using Informatica IDMC.
Work Arrangement
- Onsite position (5 days per week).
- Standard working hours: Monday to Friday 8:00 AM 5:00 PM.
- Expected workday: 7.25 hours excluding lunch breaks.
- Resource must work from the designated OPS office location.
Required Skills:
Mandatory Skills & Experience Cloud & Infrastructure Engineering Proven experience designing deploying and maintaining hybrid infrastructure environments across: Microsoft Azure Amazon Web Services (AWS) Linux-based environments SaaS solutions COTS solutions On-premises systems Strong experience with Windows and Linux server administration including: CentOS Ubuntu CoreOS Experience supporting modern web technology stacks including: LAMP MEAN Drupal Elasticsearch Experience with virtualization platforms storage solutions networking and enterprise infrastructure services. Strong understanding of cloud architecture frameworks technology assessments and cloud solution strategies. Identity & Access Management Experience managing and supporting identity and access controls using: Active Directory Microsoft Entra ID Enterprise IAM solutions Experience implementing authentication authorization and access governance controls. DevOps & CI/CD Automation Strong experience designing building and administering CI/CD pipelines for continuous integration and deployment. Experience with development and collaboration tools including: Azure DevOps JIRA Confluence Maven Jenkins Experience with Infrastructure as Code (IaC) and version-controlled deployments. Strong knowledge of DevOps methodologies automation frameworks and release management best practices. Experience with containerization and orchestration technologies including: Docker Kubernetes AKS / EKS Experience automating operational and compliance-related activities using: PowerShell Bash Python Programming & Scripting Hands-on experience with scripting and programming languages including: Python Bash PHP Java JavaScript Experience with build and configuration management tools including: Git Ansible Chef Puppet API & Integration Experience Experience building and consuming APIs and integration endpoints. Experience integrating cloud platforms enterprise systems and deployment pipelines. Understanding of enterprise integration architectures and service-based deployments. Security Monitoring & Compliance Strong understanding of infrastructure security cloud security and DevOps security best practices including: Least privilege access Secure logging Secrets management Vulnerability remediation Experience with security and monitoring tools including: Radware Dynatrace Splunk Azure Monitor / Log Analytics AWS CloudTrail / CloudWatch ELK Stack / SIEM solutions Experience implementing centralized audit logging and monitoring across enterprise platforms. Ability to support internal and external audits through reporting evidence gathering and log analysis. Experience supporting forensic investigations audit remediation and incident reviews. Familiarity with Government of Ontario I&IT Directives Policies and Information Technology Standards. Release Management & Operational Support Experience coordinating infrastructure and application releases across technical and business teams. Knowledge of change management processes deployment approvals rollback planning and release governance. Experience with release readiness reviews deployment validation and post-release monitoring. Ability to maintain release documentation operational records and audit artifacts. Project Management & Communication Strong project planning coordination and resource management skills. Experience with roadmap planning modernization initiatives and infrastructure investment planning. Strong stakeholder communication and cross-functional collaboration skills. Ability to explain complex technical concepts to business users leadership teams and non-technical stakeholders. Strong documentation reporting presentation and technical writing skills. Mobile Device Management Experience supporting mobile device management for cellular devices and tablets across multiple platforms and operating systems. Desirable Skills Experience supporting Government of Ontario enterprise environments. Experience with enterprise modernization and cloud transformation initiatives. Familiarity with enterprise governance compliance frameworks and audit readiness programs. Experience mentoring and coaching junior DevOps and infrastructure resources. Strong understanding of operational resilience disaster recovery and enterprise monitoring strategies. Additional Information Strong hands-on DevOps cloud infrastructure automation CI/CD security and enterprise release management experience is mandatory for this role.
