Data Engineer

Plantation, FL - USA

Monthly Salary: Not Disclosed

Posted on: 3 hours ago

Vacancies: 1 Vacancy

Job Summary

Location: Plantation: 4 days on site

1. Data Platform Infrastructure & DevOps

Administer optimize and scale our Databricks Lakehouse environment ensuring high performance cost efficiency and operational excellence.
Develop maintain and enhance our data platform infrastructure and security configurations using Terraform. This includes provisioning Databricks workspaces SQL Endpoints Unity Catalog objects and network components.
Manage and enforce Unity Catalog for data governance access control and metadata management.
Implement and manage CI/CD pipelines for data pipelines dbt projects and infrastructure deployments using GitHub Actions.
Automate operational tasks monitoring and alerting for the data platform.
Implement and enforce DevSecOps principles working closely with security teams to ensure compliance and manage/rotate credentials securely.

2. Data Engineering & Pipeline Development

Design and implement data ingestion patterns into Databricks using Delta Lake optimizing for large-scale data processing and storage.
Develop optimize and troubleshoot complex Spark jobs (PySpark/Scala) for data processing and transformation within Databricks.
Manage and extend data ingestion pipelines using Airbyte (or similar modern tools like Fivetran Stitch) including configuring connectors monitoring syncs and ensuring data quality and reliability from diverse source systems (e.g. ERP CRM marketing supply chain).
Orchestrate and automate data pipelines and dbt models using Databricks Workflows and potentially integrating with other orchestration tools.

3. Data Modeling & Collaboration

Collaborate with Analytics Engineers to translate business requirements into efficient and scalable data models using dbt (Data Build Tool).
Implement dbt best practices for modularity testing documentation and version control ensuring seamless integration with Databricks.
Partner effectively with Analytics Engineers Data Scientists and business stakeholders to deliver high-quality data solutions.
Provide technical guidance and mentorship to junior team members and champion data engineering best practices code quality and documentation standards.

Qualifications & Skills

Education: Bachelors degree in Computer Science Data Engineering or a related technical field required.
Experience: 5 years of progressive experience as a Data Engineer with a strong focus on cloud-based data platforms.
Deep Databricks Expertise: Extensive experience with Spark (PySpark/Scala) Delta Lake Unity Catalog Databricks SQL and platform administration.
Data Modeling: Proven experience with dbt for data modeling transformation and testing.
Infrastructure as Code (IaC): Strong proficiency with Terraform for defining provisioning and managing cloud infrastructure and Databricks resources as code.
DevOps & CI/CD: Expertise in Git and GitHub Actions for version control and implementing robust CI/CD pipelines.
Programming: Proficiency in SQL and at least one programming language (Python strongly preferred Scala is a plus).
Data Architecture: Solid understanding of data warehousing data lake and lakehouse architectures.

Location: Plantation: 4 days on site 1. Data Platform Infrastructure & DevOps Administer optimize and scale our Databricks Lakehouse environment ensuring high performance cost efficiency and operational excellence. Develop maintain and enhance our data platform infrastructure and security confi...

Location: Plantation: 4 days on site

1. Data Platform Infrastructure & DevOps

Administer optimize and scale our Databricks Lakehouse environment ensuring high performance cost efficiency and operational excellence.
Develop maintain and enhance our data platform infrastructure and security configurations using Terraform. This includes provisioning Databricks workspaces SQL Endpoints Unity Catalog objects and network components.
Manage and enforce Unity Catalog for data governance access control and metadata management.
Implement and manage CI/CD pipelines for data pipelines dbt projects and infrastructure deployments using GitHub Actions.
Automate operational tasks monitoring and alerting for the data platform.
Implement and enforce DevSecOps principles working closely with security teams to ensure compliance and manage/rotate credentials securely.

2. Data Engineering & Pipeline Development

Design and implement data ingestion patterns into Databricks using Delta Lake optimizing for large-scale data processing and storage.
Develop optimize and troubleshoot complex Spark jobs (PySpark/Scala) for data processing and transformation within Databricks.
Manage and extend data ingestion pipelines using Airbyte (or similar modern tools like Fivetran Stitch) including configuring connectors monitoring syncs and ensuring data quality and reliability from diverse source systems (e.g. ERP CRM marketing supply chain).
Orchestrate and automate data pipelines and dbt models using Databricks Workflows and potentially integrating with other orchestration tools.

3. Data Modeling & Collaboration

Collaborate with Analytics Engineers to translate business requirements into efficient and scalable data models using dbt (Data Build Tool).
Implement dbt best practices for modularity testing documentation and version control ensuring seamless integration with Databricks.
Partner effectively with Analytics Engineers Data Scientists and business stakeholders to deliver high-quality data solutions.
Provide technical guidance and mentorship to junior team members and champion data engineering best practices code quality and documentation standards.

Qualifications & Skills

Education: Bachelors degree in Computer Science Data Engineering or a related technical field required.
Experience: 5 years of progressive experience as a Data Engineer with a strong focus on cloud-based data platforms.
Deep Databricks Expertise: Extensive experience with Spark (PySpark/Scala) Delta Lake Unity Catalog Databricks SQL and platform administration.
Data Modeling: Proven experience with dbt for data modeling transformation and testing.
Infrastructure as Code (IaC): Strong proficiency with Terraform for defining provisioning and managing cloud infrastructure and Databricks resources as code.
DevOps & CI/CD: Expertise in Git and GitHub Actions for version control and implementing robust CI/CD pipelines.
Programming: Proficiency in SQL and at least one programming language (Python strongly preferred Scala is a plus).
Data Architecture: Solid understanding of data warehousing data lake and lakehouse architectures.