Lakshminarayanan selvam

Designed and delivered end-to-end Azure data engineering solutions using Azure Data Factory, Azure Databricks, ADLS Gen2, and Delta Lake to support large-scale batch and analytical workloads.Developed automated and parameterized ingestion pipelines in Azure Data Factory, utilizing linked services, datasets, data flows, and dynamic pipeline configurations to ingest data from multiple source systems.Implemented data transformation logic using PySpark (DataFrames API) and Spark SQL, handling complex joins, aggregations, window functions, and data quality validations.Built and maintained Delta Lake tables with ACID transactions, schema enforcement, schema evolution, and time travel, ensuring data reliability and consistency across processing layers.Applied Spark and Delta Lake performance optimization techniques, including optimal partitioning, file compaction, Z-Ordering, broadcast joins, caching strategies, and shuffle tuning.Architected the data platform following the Medallion Architecture (Bronze, Silver, Gold) to enable separation of raw, curated, and business-ready datasets.Implemented data governance and access control using Databricks Unity Catalog, managing catalogs, schemas, table-level permissions, and auditing for compliance requirements.Created and scheduled Databricks Jobs and Lakeflow declarative pipelines to orchestrate end-to-end data processing workflows with monitoring, retries, and failure handling.Configured ADF triggers including Schedule, Tumbling Window, and Storage Event triggers to support both time-based and event-driven ingestion patterns.Worked with Kafka-based streaming architectures, gaining exposure to real-time ingestion concepts such as topics, partitions, offsets, and micro-batch processing.Followed industry best practices for logging, monitoring, alerting, and pipeline orchestration, leveraging ADF pipeline monitoring and Databricks job logs for operational support.

Designed and delivered end-to-end Azure data engineering solutions using Azure Data Factory, Azure Databricks, ADLS Gen2, and Delta Lake to support large-scale batch and analytical workloads.
Developed automated and parameterized ingestion pipelines in Azure Data Factory, utilizing linked services, datasets, data flows, and dynamic pipeline configurations to ingest data from multiple source systems.
Implemented data transformation logic using PySpark (DataFrames API) and Spark SQL, handling complex joins, aggregations, window functions, and data quality validations.
Built and maintained Delta Lake tables with ACID transactions, schema enforcement, schema evolution, and time travel, ensuring data reliability and consistency across processing layers.
Applied Spark and Delta Lake performance optimization techniques, including optimal partitioning, file compaction, Z-Ordering, broadcast joins, caching strategies, and shuffle tuning.
Architected the data platform following the Medallion Architecture (Bronze, Silver, Gold) to enable separation of raw, curated, and business-ready datasets.
Implemented data governance and access control using Databricks Unity Catalog, managing catalogs, schemas, table-level permissions, and auditing for compliance requirements.
Created and scheduled Databricks Jobs and Lakeflow declarative pipelines to orchestrate end-to-end data processing workflows with monitoring, retries, and failure handling.
Configured ADF triggers including Schedule, Tumbling Window, and Storage Event triggers to support both time-based and event-driven ingestion patterns.
Worked with Kafka-based streaming architectures, gaining exposure to real-time ingestion concepts such as topics, partitions, offsets, and micro-batch processing.
Followed industry best practices for logging, monitoring, alerting, and pipeline orchestration, leveraging ADF pipeline monitoring and Databricks job logs for operational support.

Designed and supported end-to-end Azure data engineering solutions using Azure Data Factory, Azure Databricks, and ADLS for batch and analytical workloads.
Built and maintained Azure Data Factory ingestion pipelines using parameterized datasets, linked services, and dynamic pipeline configurations to load data from relational sources.
Developed data transformation logic using Spark APIs-Python PySpark DataFrames and Spark SQL.
Implemented and managed Delta Lake tables with ACID transactions, schema enforcement, and schema evolution to support reliable incremental data loads.
Performed basic Spark performance tuning by applying appropriate partitioning and efficient query logic to improve pipeline execution.
Supported pipeline orchestration, monitoring, and troubleshooting, following best practices for logging and handling failures in ADF and Databricks jobs.

Supported Azure Data Factory (ADF) batch ingestion pipelines by configuring datasets, linked services, and basic triggers for SQL Server and MySQL sources.
Assisted in writing and maintaining Spark SQL queries.
Performed basic PySpark optimizations such as partition tuning, caching, and filtering to improve query performance.
Helped manage Delta Lake tables by validating data loads and supporting schema updates during ingestion.
Followed Git-based version control and CI/CD practices while supporting pipeline execution, monitoring, and issue resolution.

Evaluated and hired engineers across SQL, Python, Spark, and cloud technologies.
Managed ATS platforms, LinkedIn sourcing, and data-driven screening workflows.
Developed strong understanding of real-world data engineering skill requirements.