DataOpsCloud Data Engineer – Senior
Job Location:
Monthly Salary:
Experience Required:
Posted:
Application Deadline:
Vacancies:
Job Summary
- Design develop optimize and support Azure Data Factory and Databricks pipelines integrating Oracle databases with cloud Lakehouse environments.
- Develop and optimize Databricks solutions using Medallion Architecture including Bronze Silver and Gold layers.
- Design and optimize data connections between Databricks/Lakehouse environments and on-premises data sources for downstream consumers.
- Design and implement scalable relational data models dimensional models fact/dimension models star schemas data warehouses data lakes and data lakehouses.
- Translate and modernize existing Informatica ETL processes into Azure Data Factory and Databricks ELT.
- Develop data pipelines and workflows for orchestration deployment automation parallel processing data movement and Delta Lake management/indexing.
- Work extensively with Python SQL T-SQL PL/SQL Informatica Azure Data Factory SSIS Databricks and Microsoft Fabric.
- Implement CI/CD and automated data provisioning processes using Azure DevOps and related technologies.
- Develop and manage cloud data services including Data Lakehouse Delta Lake Azure Storage Accounts Key Vault virtual machines repositories Parquet files and related Azure services.
- Work with structured semi-structured and unstructured data across enterprise data warehouse data lake and lakehouse environments.
- Design and implement high-volume data migration and transformation solutions across OLTP and OLAP environments and cloud SaaS PaaS and IaaS platforms.
- Implement data quality processes covering data validation profiling cleansing monitoring and reconciliation.
- Apply data governance and information architecture standards and develop conceptual logical and physical data models.
- Implement secure data access using Microsoft Entra ID including authentication authorization and role-based access control.
- Implement data anonymization and masking techniques for sensitive and regulated information.
- Develop and maintain data lineage to provide end-to-end visibility into data movement and transformations.
- Perform DataOps performance monitoring troubleshooting tuning and production support.
- Investigate defects and production issues and develop and deploy appropriate fixes.
- Participate in system unit SIT SAT and UAT testing.
- Prepare and maintain technical system business design support release and training documentation.
- Collaborate with business stakeholders project managers architects developers data teams and other technical teams.
- Provide technical leadership guidance and solution design expertise across multiple teams and workstreams.
- Support Agile and Waterfall delivery methodologies and established SDLC processes.
- Informatica ETL
- Azure Data Factory (ADF)
- Azure Databricks
- Databricks Medallion Architecture
- Python
- SQL
- T-SQL
- PL/SQL
- SSIS
- Microsoft Fabric
- Lakeflow
- SQL Optimization
- Delta Lake
- Data pipeline development workflow orchestration deployment and automation
- Dataflow parallelism data movement and Delta Lake indexing
- Cloud data engineering and cloud data platforms
- Data Lake Data Warehouse and Lakehouse solutions
- Relational and dimensional data modelling
- Fact/dimension modelling and star schema
- Enterprise data warehouse and data lake/lakehouse development
- Cloud and on-premises data integration
- CI/CD and automated data provisioning
- Azure DevOps
- Data quality profiling cleansing validation and monitoring
- Data governance and data lineage
- Microsoft Entra ID and RBAC
- Data anonymization and masking
- High-volume/VLD data migration and transformation
- OLTP and OLAP environments
- Strong analytical problem-solving decision-making communication presentation and interpersonal skills.
- Proven experience designing and developing complex data and business intelligence solutions.
- Strong DataOps best practices and Agile development/deployment experience.
- Experience working with large and diverse structured and unstructured datasets.
- Experience with data analysis data profiling KPI development and data visualization.
- Strong production troubleshooting and issue-resolution experience.
- Experience coordinating modernization of large complex multi-platform and multi-tier technology environments.
- Experience creating detailed business system technical and solution design documentation.
- Experience with AODA/WCAG compliance.
- Demonstrated ability to work collaboratively across multiple teams and meet strict deadlines.
- Strong experience with SDLC Agile and Waterfall methodologies.
- Experience conducting unit testing system testing SIT SAT and UAT.
- Experience tracking reporting and resolving project issues and risks.
- Experience gathering and consolidating business and technical requirements.
- Experience leading requirements-gathering sessions and supporting change requests and project artifacts.
- Experience coordinating multiple projects and competing priorities.
- Ability to provide technical leadership and guidance to development and business teams.
- Ability to ensure business requirements are accurately reflected in solution designs and technical specifications.
- Lead and coordinate technical teams and workstreams.
- Provide technical advice and mentorship to team members.
- Work effectively with business users project managers architects developers and IT teams.
- Facilitate solution integration across multiple teams and platforms.
- Communicate technical concepts clearly to both technical and non-technical stakeholders.
- Demonstrate strong ownership accountability and delivery focus.
- Previous experience working in a public-sector organization of comparable size is strongly preferred.
- Experience delivering enterprise-scale data modernization cloud migration or data integration initiatives within government or large regulated environments is an asset.
- All deliverables and supporting documentation.
- Design support release and training documentation.
- Project requirements and solution design documentation.
- Documentation stored in designated repositories such as Git SharePoint HPQC TFS or other approved project repositories.
- Written communication to the Project Manager and designated Ministry staff identifying where documentation has been stored.
- At least one walkthrough of the project documentation.
- One-on-one knowledge-transfer sessions email communication document updates and documentation reviews.
Required Skills:
Mandatory Technical Skills Candidates must demonstrate strong hands-on experience in the following: Informatica ETL Azure Data Factory (ADF) Azure Databricks Databricks Medallion Architecture Python SQL T-SQL PL/SQL SSIS Microsoft Fabric Lakeflow SQL Optimization Delta Lake Data pipeline development workflow orchestration deployment and automation Dataflow parallelism data movement and Delta Lake indexing Cloud data engineering and cloud data platforms Data Lake Data Warehouse and Lakehouse solutions Relational and dimensional data modelling Fact/dimension modelling and star schema Enterprise data warehouse and data lake/lakehouse development Cloud and on-premises data integration CI/CD and automated data provisioning Azure DevOps Data quality profiling cleansing validation and monitoring Data governance and data lineage Microsoft Entra ID and RBAC Data anonymization and masking High-volume/VLD data migration and transformation OLTP and OLAP environments Core Qualifications Strong analytical problem-solving decision-making communication presentation and interpersonal skills. Proven experience designing and developing complex data and business intelligence solutions. Strong DataOps best practices and Agile development/deployment experience. Experience working with large and diverse structured and unstructured datasets. Experience with data analysis data profiling KPI development and data visualization. Strong production troubleshooting and issue-resolution experience. Experience coordinating modernization of large complex multi-platform and multi-tier technology environments. Experience creating detailed business system technical and solution design documentation. Experience with AODA/WCAG compliance. Demonstrated ability to work collaboratively across multiple teams and meet strict deadlines. Project & Delivery Experience Strong experience with SDLC Agile and Waterfall methodologies. Experience conducting unit testing system testing SIT SAT and UAT. Experience tracking reporting and resolving project issues and risks. Experience gathering and consolidating business and technical requirements. Experience leading requirements-gathering sessions and supporting change requests and project artifacts. Experience coordinating multiple projects and competing priorities. Ability to provide technical leadership and guidance to development and business teams. Ability to ensure business requirements are accurately reflected in solution designs and technical specifications. Leadership & Collaboration The successful candidate will: Lead and coordinate technical teams and workstreams. Provide technical advice and mentorship to team members. Work effectively with business users project managers architects developers and IT teams. Facilitate solution integration across multiple teams and platforms. Communicate technical concepts clearly to both technical and non-technical stakeholders. Demonstrate strong ownership accountability and delivery focus. Public Sector Experience Previous experience working in a public-sector organization of comparable size is strongly preferred. Experience delivering enterprise-scale data modernization cloud migration or data integration initiatives within government or large regulated environments is an asset. Certifications Databricks and/or Microsoft Fabric certifications are considered an asset. Knowledge Transfer Requirement The consultant will be responsible for transferring knowledge to the designated Community Services Cluster resource. Knowledge transfer must be completed one week before the project end date or one week before the consultant leaves the Ministry whichever applies. Knowledge transfer will include: All deliverables and supporting documentation. Design support release and training documentation. Project requirements and solution design documentation. Documentation stored in designated repositories such as Git SharePoint HPQC TFS or other approved project repositories. Written communication to the Project Manager and designated Ministry staff identifying where documentation has been stored. At least one walkthrough of the project documentation. One-on-one knowledge-transfer sessions email communication document updates and documentation reviews. Ideal Candidate Profile The ideal candidate is a senior hands-on Cloud Data Engineer/DataOps professional who combines deep expertise in Informatica Azure Data Factory Databricks Python SQL and Medallion Architecture with strong enterprise data modelling cloud migration DataOps CI/CD data governance and production support experience. Important: This is a 100% onsite position requiring the consultant to work from 777 Bay Street Toronto five days per week.
Required Education:
Bachelors degree or higher in Computer Science Information Technology Software Engineering Data Engineering Data Science Computer Engineering or a related technical discipline
