Senior Data Engineer
Share
Job Location:
Monthly Salary:
Posted on:
6 days ago
Vacancies:
1 Vacancy
Job Summary
About the project:
Data solution to gather and transform data from multiple heterogeneous systems across our clients organization into the Azure Data Lakehouse using Databricks and ADF.
Responsibilities:
Design complex ETL processes of various data sources in the data Lakehouse
Build new and maintain existing data pipelines using Python to improve efficiency and latency
Improve data quality through anomaly detection by building and working with internal tools to measure data and automatically detect changes
Identify design and implement internal process improvements including re-designing infrastructure for greater scalability optimizing data delivery and automating manual processes
Perform data modeling and improve our existing data models for analytics
Collaborate with SMEs architects analysts data scientists and others to build solutions that integrate data from many of our enterprise data sources
Partner with stakeholders including data design product and executive teams and assist them with data-related technical issues
Requirements:
3-5 years of commercial experience in building and maintaining complex data pipelines
Proficiency in Python and SQL
Well-versed in the optimization of ETL processes
Solid skills in Bash scripting
Solid knowledge of relational SQL and NoSQL databases: PostgreSQL/Redshift/MongoDB Snowflake
Knowledge of job scheduling and orchestration tools (Airflow etc.)
Good knowledge of cloud services (Azure)
Solid expertise in data processing tools for Azure: Elasticsearch Azure Data Factory Databricks Azure Synapse
Decent knowledge of CI/CD (Docker Jenkins Terraform Git)
Good understanding of algorithms and data structures
Experience in the schema and dimensional data design
Excellent communication skills both written and verbal
English level: upper intermediate
Nice to have:
Databricks certification
Microsoft Azure certification
We offer*:
- Flexible working format - remote office-based or flexible
- A competitive salary and good compensation package
- Personalized career growth
- Professional development tools (mentorship program tech talks and trainings centers of excellence and more)
- Active tech communities with regular knowledge sharing
- Education reimbursement
- Memorable anniversary presents
- Corporate events and team buildings
- Other location-specific benefits
*not applicable for freelancers
Required Experience:
Senior IC
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
N-iX is a global software development company that helps world’s leading organizations achieve lasting business value using advanced technology.
