Data Engineer

Pune - India

Monthly Salary: Not Disclosed

Posted on: 12 hours ago

Vacancies: 1 Vacancy

Job Summary

Job Description:

Stream Processing Development: Design develop and optimize stream processing pipelines using
Apache Flink and Kafka to process real-time data streams efficiently.
Data Ingestion: Implement robust data ingestion pipelines to collect process and distribute
streaming data from various sources into the Flink and Kafka ecosystem.
Data Transformation: Perform data transformation and enrichment operations on streaming data
using Spark Streaming and other relevant technologies to derive actionable insights.
Performance Optimization: Continuously optimize stream processing pipelines for performance
scalability and reliability ensuring low-latency and high-throughput data processing.
Monitoring and Troubleshooting: Monitor stream processing jobs troubleshoot issues and
implement necessary optimizations to ensure smooth operation and minimal downtime.
Integration with AWS Services: Leverage AWS technologies such as Amazon Kinesis AWS Lambda
Amazon EMR and others to build end-to-end stream processing solutions in the cloud environment.
Data Governance and Security: Implement data governance and security measures to ensure
compliance with regulatory requirements and protect sensitive data in streaming pipelines.
Collaboration: Collaborate closely with cross-functional teams including data scientists software
engineers and business stakeholders to understand requirements and deliver impactful solutions.
Documentation: Create and maintain comprehensive documentation for stream processing
pipelines including design specifications deployment instructions and operational procedures.
Continuous Learning: Stay updated with the latest advancements in stream processing technologies
tools and best practices and incorporate them into the development process as appropriate.

Job Description: Stream Processing Development: Design develop and optimize stream processing pipelines using Apache Flink and Kafka to process real-time data streams efficiently. Data Ingestion: Implement robust data ingestion pipelines to collect process and distribute streaming data f...