Shoeb Khan

Shoeb Khan

Data Engineer
India
Marathi, English, Hindi, Urdu
Full-time Hybrid

نبذة عني

Data Engineer with 4+ years of experience designing, developing, and maintaining scalable data pipelines and ETL/ELT workflows using AWS, Databricks, PySpark, Spark SQL, and SQL. Experienced in building cloud-based data …

الخبرة

Senior Engineer

Nagarro · Pune , Maharashtra , India
Aug 2025 - حتى الآن · 1 سنة 2 أشهر

<p>Designed and developed a metadata-driven data pipeline framework supporting data processing for 11 downstream data sources.</p><p> Developed AWS Lambda-based event-driven orchestration using Amazon S3, SQS and event triggers to initiate data processing workflows.</p><p> Built and maintained audit, checkpoint, logging and metadata tables to provide end-to-end tracking of pipeline execution, status and failure reasons.</p><p> Developed a Databricks Submit Job to read checkpoint and compute configuration tables, identify eligible data sources and trigger respective processing workflows.</p><p> Implemented configuration-driven processing controls to enable or disable data source processing dynamically.</p><p> Developed a reusable utility logging framework using Python, PySpark and SQL for tracking execution details across pipeline stages.</p><p> Implemented checkpoint-based status tracking and failure handling, ensuring subsequent processing steps were stopped when a preceding step failed.</p><p> Integrated AWS Lambda with Databricks APIs and used AWS Secrets Manager for secure credential management.</p><p> Worked with Databricks Jobs, notebooks and Delta Lake tables as part of the overall data processing framework.</p>

Data Engineer

ZS Associates · Pune , Maharashtra , India
Jul 2024 - حتى الآن · 2 سنوات 4 أشهر

<p>Developed and enhanced Call Plan data processes across Informatica, Hive and Azure environments.</p><p> Performed data lineage analysis and created Source-to-Target Mapping (STTM) for multi-layer data processing.</p><p> Developed Python-based Metric Engine workflows with JSON configurations for data loading and processing.</p><p> Implemented Data Quality Management (DQM) checks and SCD Type 2 for data validation and historical tracking.</p><p> Used AWS EC2, EMR and Hive for large-scale data processing and integrated data into Snowflake.</p><p> Performed unit testing, SIT, RCA and Spark optimization to ensure data accuracy and improve processing performance.</p>

Software Engineer

Zielotech Software Pvt Ltd · Pune , Maharashtra , India
Dec 2021 - Jun 2024 · 2 سنوات 6 أشهر

<p>Developed an AWS-based data processing and bookkeeping system for financial data across multiple source systems.</p><p> Designed S3-based data ingestion with Lambda triggers to automatically initiate AWS Glue processing workflows.</p><p> Developed PySpark validation and transformation logic in AWS Glue to ensure data quality and reliable loading into target tables.</p><p> Implemented error handling, data correction and AWS SNS notifications for operational monitoring and failure management.</p><p> Integrated Amazon Redshift with Power BI for automated calculations, reporting and business visualization.</p>

المشاريع

marvel

Nagarro
المدة : 05-Feb-2026 - 23-Oct-2026

<p>MARVEL — Data Engineering Platform</p><p>Designed and developed a metadata-driven data pipeline framework supporting 11 downstream data sources. Implemented AWS Lambda-based event-driven orchestration using Amazon S3 and SQS to initiate data processing workflows. Built and maintained metadata, audit, checkpoint, and logging tables to track pipeline execution, processing status, and failure reasons.</p><p>Developed a Databricks Submit Job to read checkpoint and configuration tables, identify eligible data sources, and trigger the corresponding processing workflows. Implemented configuration-driven controls to dynamically enable or disable source processing and developed reusable logging utilities using Python, PySpark, and SQL.</p><p>Integrated AWS Lambda with Databricks APIs and used AWS Secrets Manager for secure credential management. Implemented checkpoint-based status tracking and failure handling to prevent downstream processing when preceding steps failed.</p><p>Technologies: AWS S3, AWS Lambda, Amazon SQS, Amazon SNS, Databricks Jobs, Databricks Notebooks, Delta Lake, Python, PySpark, SQL, AWS Secrets Manager.</p>

الشهادات

computer science degree

Dr. Babasaheb Ambedkar Technological University, India · Aurangabad , Maharashtra , India · 2021

المهارات

Python SQL Spark SQL ETL ELT Data Pipelines Data Transformation Data Integration Data Warehousing Snowflake ChatGPT GitHub Copilot Kiro Apache Spark PySpark Databricks Informatica Data Validation Data Quality Checks Error Handling Pipeline Monitoring AWS Azure Amazon S3 AWS Glue AWS Lambda Amazon RDS Amazon Redshift Amazon SNS Amazon CloudWatch Git Jupyter Notebook VS Code MS Excel SQS AWS Secrets Manager Delta Lake Azure Data Factory AWS EC2 AWS EMR Hive Trino Power BI
الإبلاغ عن هذا الملف الشخصي؟