Lead Data Engineer R
Share
Job Location:
Monthly Salary:
Posted on:
13 hours ago
Vacancies:
1 Vacancy
Job Summary
Lead Data Engineer
Primary Skills
- Azure Data Factory Azure Data Lake logic Apps and Azure SQLMPP PySpark Databricks
Specialization
- Azure Data Engineering Advanced: Senior Data Engineer
Job requirements
- Job Description - Azure Data Engineer
- At least 4 to 6 years of relevant experience in data engineering team with 2-3 years hands-on experience in supporting Azure cloud technologies.
- Hands-on Expertise in Azure Data Engineering tool stack primarily Azure Data Factory Azure Data Lake logic Apps and Azure SQLMPP (Azure Synapse) .
- Deep knowledge and hands-on on creating Azure Databricks notebooks (Good knowledge on PySpark Databricks SQL Delta Lake concepts) or Synapse Spark.
- Good understanding and exposure to Production Support of data integration and ETL systems with an understanding of SLAs cross team collaboration and expectation setting.
- Good understanding and experience around Data Warehousing ETL/ELT process Relational Databases Slowly changing dimension considerations etc.
- Should be familiar with Azure Synapse Workspace and good understanding of the Best practices for Data Loading to various target systems like Azure Synapse data lake etc from various source systems.
- Understanding and able to build complex data transformation logic using Azure Databricks Pyspark for structured semistructured data with different formats
- understanding of No-SQL databases and experience with the same.
- Good understanding of SQL Can write stored procedures CTE queries functions understanding of normalization.
- Understanding of data loading from different source file formats (parquet xml json csv etc) Experience with data Extraction from different data sources (Relational FTP SAP etc)
- Good understanding of security requirements of different Azure components
- Experience in best practices to resolve issues due to ingestion of varied data volumes by code related changes optimal usage of clusters partitioning and other methods
- Worked on 1-2 projects involving migration of large scale projects from On-Prem to cloud and also having a decent understanding of remediation of performance issues
- Understanding of Devops (CICD processes for automated deployments) and as applicable for data engineering (ADF ADB SQL deployments)
- Very good understanding of Agile to deliver consistent project outcome.
- Very high data affinity and strong analytical skills understanding of data modelling concepts
- Analyzing and translating business needs into long-term solution data models
- Evaluating and analysis (reverse engineering) of existing data systems.
- Creation of Conceptual data models and data flows Physical data models and data strategies
- Nice to have:- Certification in DP-203 (Microsoft Certified: Azure Data Engineer Associate) Databricks Data Engineering Certification
We may use artificial intelligence (AI) tools to support parts of the hiring process such as reviewing applications analyzing resumes or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed please contact us.
Required Experience:
IC
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
Brillio is a global leader in Enterprise Digital Transformation Solutions, providing strategic consulting services and solutions using emerging technologies.
