Data Support Engineer (Databricks experience)

Chicago, IL - USA

Monthly Salary: Not Disclosed

Posted on: 5 hours ago

Vacancies: 1 Vacancy

Job Summary

Position: Data Support Engineer (Databricks experience)

Location: Chicago IL 6060

Duration: 6 months

Job Description:

A Data Application Support role specializing in Databricks focuses on maintaining troubleshooting and optimizing Spark-based data pipelines and Lakehouse architectures. Key responsibilities include resolving L2/L3 production incidents performance tuning SQL/Python (PySpark) jobs managing Delta Lake assets and collaborating with data engineers to ensure data reliability.

Key Responsibilities

Incident & Problem Management: Provide L2/L3 support for data applications resolving production issues and troubleshooting Databricks jobs notebooks and workflows.
Performance Tuning: Optimize Spark applications SQL queries and Delta Lake tables to improve efficiency and reduce costs.
Pipeline Maintenance: Monitor and troubleshoot ETL/ELT pipelines in Databricks (including Data Factory/Delta Live Tables) ensuring data quality and lineage (Unity Catalog).
Collaboration: Act as a liaison between users data engineering teams and platform engineering providing technical expertise and contributing to documentation.
Automation: Create tools to automate routine support tasks and enhance support team productivity. Databricks 5

Required Qualifications & Skills

Technical Expertise: Strong hands-on experience with Apache Spark Python/PySpark and SQL.
Databricks Ecosystem: Proficiency with Databricks Unified Data Analytics Platform Delta Lake and ideally Azure Databricks.
Cloud Data Storage: Experience with Azure Data Lake Storage (ADLS Gen2) or similar data lake technologies.
Version Control & CI/CD: Experience with Git/Azure DevOps for code management.
Problem-Solving: Strong analytical skills with the ability to diagnose complex data processing bottlenecks.

Nice-to-Have Skills

Data orchestration tools (e.g. Apache Airflow Azure Data Factory).
Data governance tools (e.g. Unity Catalog Collibra).
Streaming data knowledge (e.g. Spark Structured Streaming).

Position: Data Support Engineer (Databricks experience) Location: Chicago IL 6060 Duration: 6 months Job Description: A Data Application Support role specializing in Databricks focuses on maintaining troubleshooting and optimizing Spark-based data pipelines and Lakehouse architectures. Key responsi...