Data Engineer (Private Lending)
Share
Job Location:
Johannesburg - South Africa
Monthly Salary:
Not Disclosed
Posted on:
22 hours ago
Vacancies:
1 Vacancy
Job Summary
Were hiring an experienced Azure Data Engineer to design build and scale our nextgeneration data platform for private lending. Youll own robust ETL/ELT pipelines using Azure Data Factory Databricks (Delta Lake) Synapse Analytics and ADLS Gen2integrating structured and semistructured data from core lending systems CRM payment gateways credit bureaus and thirdparty REST APIs. Youll collaborate with data scientists BI developers and risk/compliance teams to deliver analytics for risk modeling portfolio performance collections and regulatory reporting.
What Youll Do (Key Responsibilities)
- Design & Build Pipelines: Develop scalable secure ADF pipelines and Databricks notebooks for batch and nearrealtime ingestion and transformations; implement medallion architecture (Bronze/Silver/Gold) on ADLS Gen2/Delta.
- REST API Integrations: Implement robust API ingestion with OAuth2 pagination (cursor/offset) retry/backoff error handling idempotent upserts and incremental watermarks.
- Data Modeling & ELT: Build curated fact/dimension models (loan repayment collateral delinquency) optimized for Synapse/Power BI; implement SCD Type 2 partitioning and schema evolution.
- SQL Excellence: Author highperformance TSQL (stored procedures views indexing strategies MERGE/upserts) for transformations reconciliations and regulatory extracts.
- OnPrem to Cloud: Modernize/replace SSIS packages with ADF/Data Flows or Databricks; orchestrate dependencies and schedules.
- CDC & Streaming: Implement CDC from SQL Server or file drops (Auto Loader) handling latearriving data and deduplication.
- Security & Governance: Apply Managed Identity Key Vault RBAC/ACLs encryption at rest/in transit and PII controls; integrate with Purview for lineage and catalog.
- Observability: Configure Log Analytics dashboards and alerting for ADF/Databricks/Synapse; drive reliability and cost optimization.
- DevOps: Use Azure DevOps (Git YAML pipelines) for CI/CD across ADF/Databricks/Synapse; implement environment promotion and configuration as code.
- Stakeholder Collaboration: Partner with product risk finance and compliance teams to translate requirements into robust data solutions and SLAs.
Requirements
- 68 years in Data Engineering with at least 3 years on Azure (ADF Databricks Synapse ADLS Gen2).
- Strong SQL/TSQL: complex joins window functions dynamic SQL stored procedures performance tuning and MERGE/upserts.
- Handson with SQL Server and SSIS (migration/modernization to ADF preferred).
- Proven REST API integration experience: authentication (OAuth2 client credentials) pagination throttling/rate limits retries and incremental ingestion.
- Expertise with Databricks (PySpark) and Delta Lake (ACID time travel schema evolution optimize/vacuum).
- Experience building data models and curated layers for BI/analytics; familiarity with Synapse Serverless/Dedicated.
- Solid grasp of data quality (validations reconciliations) error handling and observability.
- Strong understanding of data security (PII masking encryption access controls) and complianceoriented design.
- CI/CD with Azure DevOps (repos branching pull requests pipelines).
- Excellent communication and ownership mindset in crossfunctional environments.
- Domain knowledge in lending/fintech e.g. loan origination/servicing delinquency collections vintage/roll rate analytics regulatory reporting.
- Python for orchestration/helpers Power BI exposure or Event Hub/Functions for event ingestion.
- Purview (glossary lineage) Great Expectations/Deequlike data quality frameworks.
- Cost optimization (cluster sizing partitioning/file size best practices serverless vs dedicated tradeoffs).
- Experience with infrastructureascode (Bicep/Terraform) is a plus.
Required Skills:
68 years in Data Engineering with at least 3 years on Azure (ADF Databricks Synapse ADLS Gen2). Strong SQL/TSQL: complex joins window functions dynamic SQL stored procedures performance tuning and MERGE/upserts. Handson with SQL Server and SSIS (migration/modernization to ADF preferred). Proven REST API integration experience: authentication (OAuth2 client credentials) pagination throttling/rate limits retries and incremental ingestion. Expertise with Databricks (PySpark) and Delta Lake (ACID time travel schema evolution optimize/vacuum). Experience building data models and curated layers for BI/analytics; familiarity with Synapse Serverless/Dedicated. Solid grasp of data quality (validations reconciliations) error handling and observability. Strong understanding of data security (PII masking encryption access controls) and complianceoriented design. CI/CD with Azure DevOps (repos branching pull requests pipelines). Excellent communication and ownership mindset in crossfunctional environments. NicetoHave (Good to Bring) Domain knowledge in lending/fintech e.g. loan origination/servicing delinquency collections vintage/roll rate analytics regulatory reporting. Python for orchestration/helpers Power BI exposure or Event Hub/Functions for event ingestion. Purview (glossary lineage) Great Expectations/Deequlike data quality frameworks. Cost optimization (cluster sizing partitioning/file size best practices serverless vs dedicated tradeoffs). Experience with infrastructureascode (Bicep/Terraform) is a plus.
Company Industry
IT Services and IT Consulting
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
