About Me
• Highly skilled Data Engineer and Data Analyst with over 8 years of expertise in AWS and Azure cloud platforms. Proficient in data modelling, architecting, implementing, and optimizing ETL pipelines, data warehouses, an…
• Highly skilled Data Engineer and Data Analyst with over 8 years of expertise in AWS and Azure cloud platforms. Proficient in data modelling, architecting, implementing, and optimizing ETL pipelines, data warehouses, and analytics solutions.
• Spearheaded AWS-based development projects, utilized Databricks for delta lake solution and built data solutions using NoSQL databases such as DynamoDB, collaborating with stakeholders and solution architects.
• Experience in building data pipelines using Azure Data factory, Azure Databricks and loading data to Azure Data Lake, Azure SQL Database, Azure SQL Data Warehouse and controlling and granting database access.
• Experience with Amazon EC2, Amazon S3, Amazon RDS, VPC, IAM, Amazon Elastic Load Balancing, Auto Scaling, Cloud Watch, SNS, SES, SQS, Lambda, EMR and other services of the AWS family.
Experience
Data Engineer
Spearheaded AWS-based development projects, collaborating with stakeholders for scalable solutions, resulting in a 20% reduction in data processing time.
Utilized AWS services like ECS batch, Glue, IAM roles, Athena, and S3 to construct robust data pipelines, achieving a 15% increase in data integration efficiency.
Employed extensive SQL queries within Databricks to orchestrate complex data processing tasks, leading to a 25% improvement in data analysis accuracy.
Senior Data Engineer
Spearheaded AWS-based development projects, driving collaboration with stakeholders and solution architects for scalable solutions, resulting in a 20% reduction in data processing time.
Implemented infrastructure as code using Terraform to manage and automate AWS resources, resulting in a 25% improvement in deployment speed and consistency.
Utilized AWS services like EC2, Kinesis, Redshift, Glue, IAM roles, Athena, and S3 to construct robust data pipelines, achieving a 15% increase in data integration efficiency and observability.
Employed extensive SQL queries within Databricks to orchestrate complex data processing tasks, leading to a 25% improvement in data analysis accuracy.
Led the customer churn analysis project using Snowflake, applying advanced analytics techniques to identify churn predictors and segment customers based on their behavior, contributing to improved customer retention strategies.
Leveraged Snowflake's SQL capabilities to optimize data retrieval and manipulation, resulting in a 30% enhancement in query performance and resource utilization.
Demonstrated expertise in data modeling, translating business requirements into efficient data structures, contributing to a 15% increase in data governance efficiency.
Engineered complex ETL pipelines in Glue for real-time streaming and batch processing, integrating metadata services and UI dashboards, facilitating a 25% improvement in data monitoring and prediction accuracy.
Led initiatives to monitor and optimize AWS-related traffic and services, streamlining deployment pipelines, and enhancing efficiency in the software delivery lifecycle, resulting in a 15% increase in system reliability.
Implemented data quality best practices and optimized data workflows, aligning closely with data governance initiatives, resulting in a 20% improvement in data quality and lineage management.
Data Engineer
Orchestrated Azure Data Factory to seamlessly integrate both on-premises (MySQL, Cassandra) and cloud-based (Blob storage, Azure SQL DB) data, implementing transformations to load data efficiently back into Azure Synapse, resulting in a 20% reduction in data integration time.
Managed, Configured and scheduled resources across the cluster using Azure Kubernetes Service.
Utilized Terraform to define and provision infrastructure for Azure resources, streamlining the deployment process and reducing manual configuration errors by 30%.
Leveraged SQL queries within Spark SQL in Databricks to streamline data transformations and automate RDD case classes to schema RDD conversion, enhancing overall efficiency in data processing workflows, leading to a 25% increase in data processing speed.
Developed data ingestion pipelines on Azure HDInsight Spark cluster using Azure Data Factory and Spark SQL, alongside working with Cosmos DB (SQL API and Mongo API), resulting in a 30% improvement in data ingestion performance.
Responsible for error rate by minimizing the disruptions to the users during deployment and by lowering the errors or failures occurs during the deployment.
Performed the migration of large data sets to Databricks (Spark), created and administered cluster, load data, configured data pipelines, loaded data from ADLS Gen2 to Databricks using ADF pipelines.
Implemented Databrick notebooks to streamline and curate data for various business use cases, achieving a 20% increase in data accessibility and usability.
Extensively utilized Azure Data Factory for data transformations, Integration Runtimes, Azure Key Vaults, and Triggers, and migrated data factory pipelines to higher environments using ARM Templates, reducing manual efforts by 30%.
Ingested data in mini-batches and performs RDD transformations on those mini-batches of data by using Spark Streaming to perform streaming analytics in Databricks.
Big Data Engineer
Automated creation and metadata update of external Hive tables on HDFS through Spark batch jobs, reducing manual intervention.
Developed Data Serialization spark common module for converting Complex objects into sequence bits using AVRO, PARQUET, JSON, CSV formats, resulting in a 30% reduction in data storage costs.
Applied ERModeling and Dimensional Data Modeling techniques, including Star Schema and Snowflake Schema, optimizing data warehouse structures and improving query performance.
Populated HDFS and PostgreSQL with massive data volumes using Apache Kafka, enabling real-time analytics and reporting, resulting in a 40% reduction in data ingestion time.
Ensured high availability and performance of Elastic MapReduce (EMR) clusters through continuous monitoring and management via AWS console, resulting in a 15% reduction in downtime incidents.
Automated data ingestion pipeline for multiple sources in AWS using Python scripts, streamlining data acquisition process and reducing manual effort.
Designed and developed Tableau visualizations, enhancing data insights and decision-making capabilities, resulting in an improvement in reporting accuracy and efficiency.
Data Analyst
Generated comprehensive reports and performed data analysis on key performance and operational metrics using Power BI, resulting in a 15% increase in efficiency and cost savings of $50,000.
Conducted data analysis and forecasting using Python, resulting in accurate predictions and identification of trends that led to a 10% increase in sales revenue and improved resource allocation.
Provided critical supporting information and insights that facilitated informed decision-making and resulted in a 30% increase in customer satisfaction and retention.
Assisted in extracting and evaluating data, using complex SQL queries from multiple data sources, enabling data-driven optimization and achieving a 12% increase in effectiveness.
Synthesized business intelligence and trend data to provide strategic recommendations, leading to a 10% improvement in operational efficiency and cost savings of $20,000 annually.
Successfully maintained and updated business intelligence tools, databases, and dashboards, ensuring accurate and up-to-date information for decision-makers, resulting in a 30% reduction in reporting time and enhanced data accessibility.