Data Engineering

Job Title: Middleware Engineer
Position: Data Engineer
Experience: 5-6yrs
Category: IT Infrastructure
Main location: India Karnataka Bangalore
Employment Type: Full Time
Qualification: Bachelors degree in Computer Science or related field or higher.
Roles and Responsibilities


Data Engineer - 5-6 years experience.
Responsibilities

Design develop and maintain data architectures pipelines and workflows for the collection processing storage and retrieval of large volumes of structured and unstructured data from multiple sources.
Collaborate with cross-functional teams to identify and prioritize data engineering requirements and to develop and deploy data-driven solutions to address business challenges.
Build and maintain scalable data storage and retrieval systems (e.g. data lakes data warehouses databases) fault-tolerant and high-performance data platforms on cloud infrastructure such as AWS Azure or Google Cloud Platform.
Develop and maintain ETL workflows data pipelines and data transformation processes to prepare data for machine learning and AI applications.
Implement and optimize distributed computing frameworks such as Hadoop Spark or Flink to support high-performance and scalable processing of large data sets.
Build and maintain monitoring alerting and logging systems to ensure the availability reliability and performance of data pipelines and data platforms.
Collaborate with Data Scientists and Machine Learning Engineers to deploy models on production environments and ensure their scalability reliability and accuracy.
Requirements:

Bachelor s or master s degree in computer science engineering or related field.
At least 5-6 years of experience in data engineering with a strong background in machine learning cloud computing and big data technologies.
Experience with at least one major cloud platform (AWS Azure GCP).
Proficiency in programming languages like Python Java and SQL.
Experience with distributed computing technologies such as Hadoop Spark and Kafka.
Familiarity with database technologies such as SQL NoSQL NewSQL.
Experience with data warehousing and ETL tools such as Redshift Snowflake or Airflow.
Strong problem-solving and analytical skills.
Excellent communication and teamwork skills.
Preferred qualification:

Experience with DevOps practices and tools such as Docker Kubernetes or Ansible Terraform.
Experience with data visualization tools such as Tableau Superset Power BI or Plotly .
Experience with stream processing frameworks such as Kafka Pulsar or Kinesis.
Experience with data governance data security and compliance.
Experience with software engineering best practices and methodologies such as Agile or Scrum.
Must Have Skills

data engineer with expertise in machine learning cloud computing and big data technologies.
Data Engineering Experince on multiple clouds one of them preferably GCP
data lakes data warehouses databases
ETL workflows data pipelinesdata platforms
Hadoop Spark or Flink
Hadoop Spark and Kafka
SQL NoSQL NewSQL
Redshift Snowflake or Airflow