Senior Data Engineer
Senior Data Engineer
Posted on :
03-04-2024
Employer Active
1 Vacancy
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Monthly Salary
Not Disclosed
Salary Not Disclosed
Vacancy
1 Vacancy
Posted on :
03-04-2024
Job Description
Req ID : 2687205
Responsibilities:
- Design construct install test and maintain highly scalable data management systems on Azure and AWS utilizing services such as Azure Data Lake Azure Data Factory AWS S3 and AWS Glue.
- Integrate new data management technologies and software engineering tools into existing structures focusing on Azure and AWS ecosystems.
- Utilize Azure Databricks and AWS EMR for big data processing tasks to create data set processes for data modeling mining and production.
- Develop analytics applications using Azure Synapse Analytics and AWS Redshift to leverage the data pipeline providing actionable insights into key business performance metrics.
- Collaborate with stakeholders across the Executive Product Data and Design teams to support their data infrastructure needs using Azure and AWS cloud services.
- Implement data security and compliance procedures using Azure Security Center and AWS Identity and Access Management (IAM).
- Evaluate and improve existing systems incorporating new technologies and software engineering tools available in Azure and AWS ecosystems.
- Monitor analytics and metrics results using tools like Azure Monitor and AWS CloudWatch.
- Implement systems for tracking data quality and consistency using Azure and AWS services ensuring data integrity across the pipeline.
- Automate data workflows and pipeline processes using Azure Logic Apps and AWS Step Functions integrating with Apache Airflow for orchestration.
- Bachelors or Masters degree in Computer Science Engineering or related field.
- Proven experience as a Data Engineer or similar role with a deep understanding of Azure and AWS cloud services relevant to data engineering (e.g. Azure Data Lake Azure Data Factory AWS S3 AWS Glue).
- Expertise in SQL and programming languages especially Python for data manipulation and analysis.
- Strong experience with big data tools such as Apache Spark and Hadoop.
- Proficiency in data pipeline and workflow management tools particularly Apache Airflow.
- Experience with data modeling data warehousing and building ETL pipelines in both Azure (using Azure Data Factory Azure Databricks) and AWS environments (using AWS Glue AWS Data Pipeline).
- Knowledge of streamprocessing systems (e.g. Apache Kafka Azure Stream Analytics Amazon Kinesis) is a plus.
- Familiarity with machine learning algorithms and data science principles is beneficial.
- Strong problemsolving skills and ability to project manage and multitask.
- Excellent communication and teamwork skills with an ability to navigate and negotiate with stakeholders across the organization.
Employment Type
Full Time
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
