Enterprise Data Lake Lead

Data Lake Tech. Lead
12 Months
Remote

Primary skills:
Apache Kafka
Databricks platform
DataHub or Collibra/Alation/Apache Atlas
RDBMS and Modern Data lake
Kubernetes - Functional knowledge

Required Qualifications
Technical Expertise around 12-15 years of experience.
Data Streaming & Processing
5 years with Apache Kafka (streaming architecture Kafka Connect Schema Registry stream processing)
3 years with Databricks (Delta Lake Apache Spark optimization Unity Catalog cluster management)
Deep understanding of data lake architecture patterns (Bronze/Silver/Gold medallion architecture)
Data Governance & Cataloging
3 years withDataHubor similar metadata management platforms (Alation Collibra Apache Atlas)
Deep experience building and operating enterprise data catalog systems
Expertisein automated metadata extraction lineage tracking and impact analysis
Experience with data quality frameworks and metadata-driven data operations
Knowledge of data governance policies data classification and compliance automation
Understanding of data discovery access control and self-service analytics enablement
Relational Database Systems
5 years with enterprise RDBMS platforms including:
SQL Server(T-SQL SSIS SSRS replication AlwaysOnAvailability Groups)
PostgreSQL(advanced query optimization partitioning extensions streaming replication)
MySQL(replication clustering performance tuning)
Strong SQL skills (complex queries stored procedures window functions query optimization)
Database design principles (normalization indexing strategies schema design partitioning)
Change Data Capture (CDC) patterns and implementation (Debezium Azure Data Factory AWS DMS custom solutions)
Database migration experience (schema migration data migration zero-downtime migrations)
Cloud & Infrastructure
5 years with Azure or AWS Cloud including:
Azure: Data Lake Storage (Gen2) Event Hubs AKS Azure SQL Key Vault Azure AD Monitor
AWS: S3 MSK/Kinesis EKS RDS/Aurora Secrets Manager IAM CloudWatch
Cloud-native data services networking security and IAM
3 years with Kubernetes (deployment strategies scaling monitoring service mesh Helm)
3 years with Terraform (modules state management multi-environment deployments multi-cloud)
Programming & Development
Strongproficiencyin any programming languages like C# Go
Expert-levelSQLacross multiple database platforms
Experience with Python for data engineering tasks (preferred)
Familiarity with Shell scripting (Bash)
Leadership & Experience
7 yearsin data engineering platform engineering or database engineering roles
3 yearsin technical leadership capacity (Tech Lead Principal Engineer)
Proventrack recordof delivering large-scale data infrastructure projects
Experience leading teams of 5-10 engineers
Strong architectural design and system thinking capabilities
Experience migrating legacy RDBMS workloads to modern data lakearchitectures
Demonstrated ability to balance technical excellence with business needs

Regards

Ram

Celersoft LLC

Required Skills:

KAFKA CONNECTAPACHE KAFKAPOSTGRESQLAZURE ADAPACHEENTERPRISE DATA LAKEAWSDATA GOVERNANCEMYSQLT-SQLNETWORKINGSSISRDBMSSQLAPACHE SPARKKUBERNETESSTREAM PROCESSINGDATA STREAMINGDATA CLASSIFICATIONSQL SKILLSC#ARCHITECTURE PATTERNSAZURE SQLMETADATA MANAGEMENT