Data Architect Databricks

Job Title: Databricks - Lead Data Engineer

Work Location: Hyderabad (Hybrid)

Experience: 8 - 12 years

Role summary

We are looking for a hands-on Databricks Lead to design build and take to production governed financial data products on an Azure Databricks Lakehouse. You will own the Databricks solution end to end - file-based ingestion medallion pipelines hierarchy modelling row-level security zero-outage publishing and performance - and lead a small engineering team while working closely with source-system owners finance stakeholders and security and platform teams.

What you will build

A typical engagement publishes multi-dimensional financial data (for example a management P&L with entity department project product version period and account dimensions) from an enterprise planning system as a single row-secured consumption table in Databricks refreshed several times a day. The quality bar:

Exact reconciliation to source control totals; no data loss.

Full fidelity of hierarchies alternate hierarchies aliases and attributes.

Zero consumer outage during refresh; a failed run never replaces the last good data.

Row-level security that matches the source systems access rules.

Sub-second response for agreed benchmark queries.

Additive schema changes absorbed without code changes; full rebuild from Git.

Key responsibilities

Architecture and technical leadership

Own the Databricks solution design and drive open decisions with stakeholders: landing zone publish mechanism orchestration pattern compute retention and recovery targets.

Translate integration specifications into buildable designs data contracts and a delivery plan; lead design reviews with integration architects and platform teams.

Lead mentor and review the work of 2-4 data engineers; set coding testing and documentation standards.

Raise risks early (data volume security parity BI connectivity) and propose options with trade-offs.

Platform setup and governance (Unity Catalog)

Set up catalogs schemas storage credentials external locations and volumes for landing zones and medallion layers across dev test and prod.

Define the grant model with account-level groups synced from Microsoft Entra ID; configure service principals for jobs and CI/CD.

Work with Azure network teams on private endpoints for ADLS and Network Connectivity Configurations for serverless compute.

Ingestion and orchestration

Build manifest-gated file-arrival-triggered Databricks Workflows with task dependencies task values condition-task publish gates queueing retries timeouts and notifications.

Implement Bronze ingestion from delimited files with checksum and row-count verification against a JSON manifest.

Guarantee idempotent reruns and safe handling of overlapping or late runs.

Data modelling and transformation (Silver / Gold)

Build PySpark logic to validate and flatten ragged parent-child hierarchies detecting cycles orphans multi-parent members and non-unit weights.

Handle alternate hierarchies through configuration as flattened tables or bridge tables.

Pivot long-form attributes and aliases into columns; build a contract-driven Gold generator that absorbs additive schema changes automatically.

Design wide denormalized consumption tables including leaf/consolidated grain flags to prevent double counting.

Data quality and reconciliation

Implement blocking quality checks (contract referential integrity hierarchy typing enrichment security) and exact reconciliation to source control totals.

Build control tables (run log check results reconciliation schema contract) and an operations dashboard.

Security

Implement source-system security parity: map source users to Entra identities resolve group and member-level rights into a security lookup table and enforce them with a Unity Catalog row-filter function.

Keep data and security consistent during publish; define break-glass access ownership and audit monitoring.

Advise BI teams on per-user connectivity (SSO / Direct Query) so row-level security holds end to end.

Publishing and performance

Implement zero-outage publishing with atomic Delta overwrites or blue/green swaps plus rollback via time travel.

Tune for sub-second queries: liquid clustering data-skipping statistics predictive optimization Photon SQL Warehouse sizing and row-filter cost; run benchmark tests with real access profiles.

DevOps testing and operations

Package everything as Databricks Asset Bundles with CI/CD (GitHub Actions or Azure DevOps) across dev test and prod.

Own the test strategy: unit tests contract and fixture tests integration SIT reconciliation security UAT and performance tests.

Write the runbook (rerun rollback full rebuild schema change) and hand over to production support.

Collaboration

Agree file contracts manifests and control totals with source-system administrators.

Work with finance users to define benchmark queries and validate numbers during UAT and parallel runs.

Report progress risks and decisions to stakeholders and delivery leadership.

Required skills

Required Skills:

SnowflakeDBT