Senior Data Engineer

Devsinc is looking for a highly skilled Senior Data Engineer with 46 years of professional experience to design build and maintain scalable data pipelines and processing systems that support analytics data products and AI/ML capabilities.

The ideal candidate will have strong hands-on experience with ETL/ELT pipelines Python SQL Apache Airflow Apache Spark and Redis along with the ability to develop reliable and maintainable workflows for large complex and continuously growing datasets. You will collaborate with Product Business Intelligence Data Science and Engineering teams to deliver production-ready datasets and high-performance data solutions.

Responsibilities

• Design develop and maintain scalable ETL/ELT pipelines for data ingestion transformation validation and delivery.
• Build schedule and orchestrate production-grade data workflows using Apache Airflow.
• Develop distributed data-processing jobs using Apache Spark.
• Write efficient reusable and maintainable Python code for data processing automation and pipeline development.
• Develop and optimize complex SQL queries for data transformation analysis and quality validation.
• Design pipelines capable of processing large-scale structured semi-structured and unstructured datasets.
• Implement Redis for caching high-performance data access and data-intensive application requirements.
• Integrate data from APIs relational databases files third-party providers and other internal and external sources.
• Implement data validation monitoring logging error handling alerting and pipeline observability.
• Optimize pipeline performance data storage processing time and infrastructure costs.
• Develop reusable data-ingestion and transformation frameworks instead of one-off scripts.
• Troubleshoot pipeline failures performance bottlenecks and data-quality issues to ensure timely resolution.
• Collaborate with BI Product Data Science and Engineering teams to deliver reliable production-ready datasets.
• Establish and maintain data-engineering standards technical documentation and development best practices.

• Bachelors degree in Computer Science Software Engineering Data Science or a related field.
• 46 years of professional experience in Data Engineering or a closely related role.
• Strong hands-on experience designing and implementing production-grade ETL/ELT pipelines.
• Strong proficiency in Python for data processing automation and data-engineering workflows.
• Advanced SQL skills and a strong understanding of relational databases.
• Practical production experience with Apache Airflow for workflow orchestration.
• Hands-on experience with Apache Spark and distributed data processing.
• Strong understanding of data modelling transformation patterns and data-pipeline architecture.
• Experience with Redis caching strategies and high-performance data-access patterns.
• Experience processing large datasets and optimizing pipeline and query performance.
• Strong understanding of data quality validation monitoring observability and pipeline reliability.
• Familiarity with Linux Git Docker/containers and modern software-engineering practices.
• Strong analytical troubleshooting communication and cross-functional collaboration skills.

Preferred Qualification

• Experience with cloud data platforms and object storage services such as AWS Azure or GCP.
• Experience working with PostgreSQL data warehouses or analytical databases.
• Experience processing geospatial data or large-scale location-based datasets.
• Familiarity with DuckDB Apache Sedona Trino Presto or similar analytical technologies.
• Experience processing high-volume event mobility transactional or geospatial data.
• Familiarity with CI/CD pipelines and infrastructure-as-code practices.
• Experience supporting data products analytics platforms or AI/ML pipelines.

Required Experience:

Senior IC