Data Engineer - Assistant Manager
Data Engineer - Assistant Manager
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
Key Responsibilities:
Data Pipeline Development: Design and develop efficient big data pipelines (batch as well as streaming) using Apache Spark Trino ensuring timely and accurate data delivery.
Collaboration and Communication: Work closely with data scientists analysts and stakeholders to understand data requirements and perform exploratory data analysis to recommend best feature attributes data models for AI model training and deliver high-quality data solutions.
Data Exploration: Analyse customer data and patterns and suggest use cases in BFSI like Instights AI & GenAI use cases.
Data Quality and Security: Ensure data quality integrity and security across all data platforms maintaining robust data governance practices.
Documentation and Troubleshooting: Own and document data pipelines and data lineage monitoring and troubleshooting data pipeline issues to ensure timely and accurate data delivery.
Technical Skills:
- Big Data Processing: Hands-on experience with Apache Spark Trino Minio/S3 Clickhouse and SQL for batch and streaming data pipelines.
Design and develop scalable data pipelines using Apache Spark (PySpark) and Python.
Implement batch and real-time data processing solutions on large datasets.
Work with various SQL and NoSQL databases (e.g. PostgreSQL MySQL MongoDB Cassandra DynamoDB).
- Data Migration & Warehousing: Expertise in metadata management ETL processes and cloud-based data solutions.
- Data Analysis & Exploration: Strong skills in dataset analysis pattern recognition and data visualization using Tableau or Power BI.
- Collaborate with data architects analysts and other engineers to understand business requirements and translate them into technical solutions.
- Optimize and troubleshoot Spark jobs for performance scalability and reliability.
- Exposure to containerization technologies like Docker and orchestration tools like Kubernetes.
Required Experience:
Manager
Employment Type
Full-Time
