drjobs Senior Data Engineer with Databricks Remote

Senior Data Engineer with Databricks Remote

Employer Active

1 Vacancy
drjobs

Job Alert

You will be updated with latest job alerts via email
Valid email field required
Send jobs
Send me jobs like this
drjobs

Job Alert

You will be updated with latest job alerts via email

Valid email field required
Send jobs
Job Location drjobs

India

Monthly Salary drjobs

Not Disclosed

drjobs

Salary Not Disclosed

Vacancy

1 Vacancy

Job Description

Role: Senior Data Engineer with Databricks.

Experience: 5 Years

Job Type: Contract

Contract Duration: 6 Months

Budget: 1.0 lakh per month

Location : Remote

JOB DESCRIPTION:

We are looking for a dynamic and experienced Senior Data Engineer Databricks to design build and optimize robust data pipelines using the Databricks Lakehouse platform. The ideal candidate should have strong hands-on skills in Apache Spark PySpark cloud data services and a good grasp of Python and Java. This role involves close collaboration with architects analysts and developers to deliver scalable and high-performing data solutions across AWS Azure and GCP.

ESSENTIAL JOB FUNCTIONS

1. Data Pipeline Development

Build scalable and efficient ETL/ELT workflows using Databricks and Spark for both batch and streaming data.

Leverage Delta Lake and Unity Catalog for structured data management and governance.

Optimize Spark jobs by tuning configurations caching partitioning and serialization techniques.

2. Cloud-Based Implementation

Develop and deploy data workflows onAWS (S3 EMRGlue) Azure (ADLS ADF Synapse) and/orGCP (GCS Dataflow BigQuery).

Manage and optimize data storage access control and pipeline orchestration using native cloud tools.

Use tools like Databricks Auto Loader and SQL Warehousing for efficient data ingestion and querying.

3. Programming & Automation

Write clean reusable and production-grade code in Python and Java.

Automate workflows using orchestration tools(e.g. Airflow ADF or Cloud Composer).

Implement robust testing logging and monitoring mechanisms for data pipelines.

4. Collaboration & Support

Collaborate with data analysts data scientists and business users to meet evolving data needs.

Support production workflows troubleshoot failures and resolve performance bottlenecks.

Document solutions maintain version control and follow Agile/Scrum processes

Required Skills

Technical Skills:

Databricks: Hands-on experience with notebooks cluster management Delta Lake Unity Catalog and job orchestration.

Spark: Expertise in Spark transformations joins window functions and performance tuning.

Programming: Strong in PySpark and Java with experience in data validation and error handling.

Cloud Services: Good understanding of AWS Azure or GCP data services and security models.

DevOps/Tools: Familiarity with Git CI/CD Docker (preferred) and data monitoring tools.

Experience:

5 8 years of data engineering or backend development experience.

Minimum 1 2 years of hands-on work in Databricks with Spark.

Exposure to large-scale data migration processing or analytics projects.

Certifications (nice to have): Databricks Certified Data Engineer Associate

Working Conditions

Hours of work - Full-time hours; Flexibility for remote work with ensuring availability during US Timings.

Overtime expectations - Overtime may not be required as long as the commitment is accomplished

Work environment - Primarily remote; occasional on-site work may be needed only during client visit.

Travel requirements - No travel required.

On-call responsibilities - On-call duties during deployment phases.

Special conditions or requirements - Not Applicable.

Workplace Policies and Agreements

  • Confidentiality Agreement: Required to safeguard client sensitive data.
  • Non-Compete Agreement: Must be signed to ensure proprietary model security.
  • Non-Disclosure Agreement: Must be signed to ensure client confidentiality and security.

APACHE SPARK , GIT , JAVA , DEVOPS , CI/CD , AUTOMATION , PYTHON , AZURE , BIGQUERY , DOCKER , SPARK , AWS

Employment Type

Remote

Company Industry

Key Skills

  • Apache Hive
  • S3
  • Hadoop
  • Redshift
  • Spark
  • AWS
  • Apache Pig
  • NoSQL
  • Big Data
  • Data Warehouse
  • Kafka
  • Scala

About Company

Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.