NCDPI - Technical Architect- Senior

Job Title: Data Engineer Azure Platform

Overview:

We are looking for an experienced Data Engineer to join our team. You will use various methods to import raw data into Azure Cloud. You will analyze data and create rules for the ETL process. The import process will also include identifying the business rules and apply them to the files stored in staging area before moving the files to processed area.

The ideal candidate will have a strong background in data engineering along with expertise in cloud computing. This role offers an exciting opportunity to play a key role in modernizing our technology stack and driving our organizations digital transformation journey.

Responsibilities:

Collaborate with crossfunctional business teams to understand requirements analyze and organize raw data import by identifying the ETL rules.

Identify and apply transformation rules for legacy data migration from mainframe systems to the Azure cloud platform.

Develop and implement data pipelines ETL processes and data integration solutions to facilitate the conversion of data.

Develop and maintain documentation best practices and standards for data engineering processes and procedures.

Provide technical leadership and mentorship to team members guiding them in best practices for data engineering and cloud migration projects.

Explore ways to enhance data quality and reliability.

Requirements and Skills

1. **Data Modeling and Database Design:**

Proficiency in designing and implementing relational and nonrelational databases.

Knowledge of data modeling techniques and tools.

2. **Azure Data Services:**

Deep understanding and handson experience with Azure data services such as Azure SQL Database Azure Cosmos DB Azure Data Lake Storage Azure Blob Storage etc.

3. **ETL (Extract Transform Load) Processes:**

Expertise in designing and implementing ETL processes using tools like Azure Data Factory or Azure Databricks.

Knowledge of data integration and transformation techniques.

4. **SQL and TSQL:**

Proficient in writing complex SQL queries and stored procedures.

Familiarity with TransactSQL (TSQL) for managing and querying SQL Server databases.

5. **Data Security and Compliance:**

Knowledge of data security best practices and experience implementing security measures for data storage and processing on Azure.

Awareness of compliance requirements and ability to implement solutions that adhere to NC DPI and DIT regulations.

6. **Data Monitoring and Optimization:**

Ability to set up monitoring and logging for data pipelines and databases.

Skills in optimizing data storage and processing for performance and costeffectiveness.

7. **Scripting and Automation:**

Proficiency in scripting languages like Python or PowerShell for automating data engineering tasks.

Knowledge of Azure Resource Manager (ARM) templates for infrastructure as code.

Additionally obtaining relevant certifications such as the Microsoft Certified: Azure Data Engineer Associate can further validate and enhance these skills.

Qualifications:

Bachelors degree in computer science Engineering or a related field

Proven experience as a data engineer with a strong focus on data migration ETL and data integration.

Handson experience with data warehousing concepts database technologies (SQL on Azure)

Strong problemsolving skills attention to detail and the ability to work effectively in a fastpaced environment.

Excellent communication and interpersonal skills with the ability to collaborate effectively with crossfunctional teams and stakeholders.