Data Engineer
Data Engineer
Posted on :
17-02-2025
Employer Active
1 Vacancy
The job posting is outdated and position may be filled
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Job Description
We are seeking an experienced AWS Data Engineer with expertise in Databricks to design build and optimize largescale data pipelines and data processing workflows on the AWS Cloud. The ideal candidate will have handson experience working with Databricks big data technologies and AWSnative services ensuring efficient data ingestion transformation and analytics to support businesscritical decisions.
Key Responsibilities:
- Data Pipeline Development: Design implement and manage scalable ETL/ELT pipelines using AWS services and Databricks.
- Data Integration: Ingest and process structured semistructured and unstructured data from multiple sources into AWS Data Lake or Databricks.
- Data Transformation: Develop advanced data processing workflows using PySpark Databricks SQL or Scala to enable analytics and reporting.
- Databricks Management: Configure and optimize Databricks clusters notebooks and jobs for performance and cost efficiency.
- AWS Architecture: Design and implement solutions leveraging AWSnative services like S3 Glue Redshift EMR Lambda Kinesis and Athena.
- Collaboration: Work closely with Data Analysts Data Scientists and other Engineers to understand business requirements and deliver datadriven solutions.
- Performance Tuning: Optimize data pipelines storage and queries for performance scalability and reliability.
- Monitoring and Security: Ensure data pipelines are secure robust and monitored using CloudWatch Datadog or equivalent tools.
- Documentation: Maintain clear and concise documentation for data pipelines workflows and architecture.
Required Skills & Qualifications:
- Data Engineering Expertise: 4 years of experience in data engineering with at least 2 years working on Databricks.
- AWS Cloud Services: Handson experience with AWS ecosystem including S3 Glue Redshift DynamoDB Lambda and other AWS data services.
- Programming Languages: Proficiency in Python (PySpark) Scala or SQL for data processing and transformation.
- Databricks: Extensive experience with Databricks Workspace Delta Lake and managing databricks jobs and pipelines.
- Big Data Frameworks: Strong knowledge of Apache Spark for distributed data processing.
- Data Warehousing: Experience with modern data warehouse solutions including Redshift Snowflake or Databricks SQL.
- Version Control & CI/CD: Familiarity with Git Terraform and CI/CD pipelines for deploying data solutions.
- Monitoring & Debugging: Experience with tools like CloudWatch Datadog or equivalent for pipeline monitoring and troubleshooting.
Preferred Qualifications:
- Certification in AWS Data Analytics or Databricks.
- Experience with realtime data streaming tools like Kafka Kinesis or AWS MSK.
- Knowledge of data governance and data security best practices.
- Exposure to machine learning workflows and integration with Databricks
Requirements
Employment Type
Full Time
Key Skills
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
