Experience in Query optimization, Improving Performance and maintain data integrity. Over 4.5 Years' experience as data engineer, Design solutions of database, ETL and Data warehousing. Optimizing the ETL (AWS Glue) perf…
Experience in Query optimization, Improving Performance and maintain data integrity. Over 4.5 Years' experience as data engineer, Design solutions of database, ETL and Data warehousing. Optimizing the ETL (AWS Glue) performance by 10X using PySpark. Performed the complete code refactoring of PySpark by using OOPs principle to increase the code reusability in AWS Glue. Created a Sanky diagram for AWS Glue pipeline for data linage. Experience in writing SQL, Stored Procedures and Database Designing, Views. ETL Development using AWS Glue with PySpark. Expertise in defining end-to-end BI architecture like data source, data warehouse to store data, ETL or data pipeline, data lake for analytics, and different visualization. Experience in database activities like Data Modeling, Database Design, Development and Maintenance, Performance Monitoring and Tuning, Troubleshooting, Normalization, and Documentation.
Designed and executed ETL pipelines in alignment with Data Lakehouse principles, leveraging AWS S3, AWS Glue, Apache HUDI, and AWS Kinesis. • Achieved a remarkable 10X improvement in Spark Jobs performance, resulting in significant reductions in both running time and operational costs. • Successfully managed daily data loads of 12GB, overseeing 10 million daily transactions. • Developed a user-friendly API and Dashboard complete with lineage diagrams to facilitate monitoring of Glue Pipelines and real-time tracking of table data freshness. • Seamlessly integrated diverse data sources, including MongoDB, MySQL, Kinesis, Static files, and Glue Catalog, for streamlined data processing. • Employed Apache HUDI to enhance the Data Lake Platform and harnessed Redshift Spectrum for efficient data querying and reporting. • Conducted code refactoring to optimize system performance and maintainability. • Deployed Data Lakehouse Schema Change management.
نظام التوظيف التشغيلي المعتمد على الذكاء الاصطناعي أولاً، الذي يوحّد ويؤتمت دورة التوظيف بالكامل للمؤسسات الحديثة حول العالم.
د.جوب منطقة حرة ذ.م.م. 2026 © جميع الحقوق محفوظة
