Senior Data Engineer Brazil

Role Overview

We are looking for a Senior Data Engineer / Databricks Engineer with 8 years of experience to design develop optimize and support enterprise data pipelines and analytical datasets.

The candidate will work with Azure Databricks Azure Data Lake Azure Data Factory PostgreSQL and Azure DevOps to deliver reliable data ingestion transformation reconciliation and reporting-ready data solutions.

Key Responsibilities

• Design develop and maintain ETL/ELT pipelines for data ingestion transformation and integration.
• Develop Databricks notebooks and reusable data engineering components using PySpark and SQL.
• Implement Medallion Architecture (Bronze Silver and Gold layers).
• Integrate operational financial accounting portfolio risk and enterprise data sources.
• Develop curated analytical datasets and data models for reporting and analytics.
• Implement data-quality checks validation rules reconciliation controls and exception handling.
• Optimize Databricks workloads SQL queries and PostgreSQL-related data processing.
• Monitor scheduled data loads and troubleshoot pipeline failures and production incidents.
• Support data lineage metadata source-to-target mapping and technical documentation.
• Implement CI/CD source control deployment validation and release-management processes.
• Collaborate with architects BI developers QA analysts business analysts and DevOps teams.
• Maintain pipeline documentation operational runbooks and monitoring reports.

Mandatory Skills & Experience

• 8 years of overall IT experience with strong relevant Data Engineering experience.
• Azure Databricks notebooks transformations and pipeline development.
• Azure Data Lake Storage (ADLS).
• PySpark Python and SQL.
• ETL/ELT pipeline development and data integration.
• Medallion Architecture Bronze Silver and Gold layers.
• Strong SQL and PostgreSQL skills.
• Data quality reconciliation validation and exception handling.
• Performance tuning and production troubleshooting.
• Data pipeline monitoring and scheduled job support.

Preferred Skills

• Azure Data Factory (ADF).
• Azure DevOps Git CI/CD pipelines and automated deployments.
• Data lineage metadata management and source-to-target mapping.
• Delta Lake and lakehouse data modeling.
• Batch processing job recovery and operational automation.
• Financial services accounting portfolio risk or reinsurance data.
• Release management root-cause analysis and production support.