Intermediate Data Engineer
Intermediate Data Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
The Intermediae Data Engineer is accountable for creating and maintaining the technological infrastructure of Spot Money and implementing methods to improve data reliability and quality. Ensure the consistent delivery of data architecture throughout ongoing projects by creating data models building data pipelines and coordinating ETL (extract transform load).
Responsibilities :
Data Ingestion & Processing
- Assist in building ETL/ELT pipelines using AWS services like AWS Glue Lambda and Step Functions.
- Work with Amazon S3 for data storage and ingestion.
- Support realtime and batch data processing jobs.
Data Storage & Management
- Load and manage data in AWS
- Work with Parquet Avro JSON and CSV file formats in data lakes.
Data Transformation &Integration
- Support data transformations using PySpark SQL or AWS Glue ETL.
- Assist in writing SQL queries for data aggregation cleansing and structuring.
Automation & Orchestration
- Monitor scheduled workflows
- Write basic Python or Shell scripts for automating tasks.
Monitoring & Debugging
- Assist in monitoring and troubleshooting failed data pipelines.
- Perform unit testing and basic debugging of data workflows.
Requirements :
Knowledge skills and attributes:
- Programming Python SQL.
- AWS Services S3 Glue Lambda Redshift DynamoDB RDS CloudWatch IAM.
- Data Processing Spark (PySpark) SQLbased transformations.
- Orchestration Step Functions Airflow or Lambda.
- Version Control Git.
- Flexibility to perform adhoc tasks as per business needs.
Education and training:
- Diploma in Computer Science Information Technology Engineering or equivalent.
- Bachelors degree in computer science Information Technology Engineering or equivalent advantageous.
- Relevant certification (Certified Data Engineer Certificate Professional etc).
Experience:
- 2 years in a similar role.
- Experience with architecture and design of secure highly available and scalable systems.
- Experience with writing scripts using scripting language.
- Experience working with data sets and ordering data through various functions.
- Experience working with internet technologies.
- Experience dealing with internal and external stakeholders.
- Experience working with and/or on Power BI Snowflake and AWS system advantageous.
Required Experience:
Manager
Employment Type
Full-Time
