Cloudera Data Engineer
Share
Job Location:
Lincoln, NE - USA
Monthly Salary:
Not Disclosed
Posted on:
15 days ago
Vacancies:
1 Vacancy
Job Summary
Job Title: Cloudera Data Engineer - 65050
Duration: 12 Months
Location: Lincoln Nebraska
Cloudera Data Engineer (Cloud DevOps Administrator Expert)
Job Summary
We are seeking a Cloudera Data Engineer to support the migration of a
Medicaid Data Warehouse Implementation in AWS environment. The resource will
support the migration and continued operations of an existing
Cloudera/Hive/Scala-based data pipeline environment from one AWS account to
another.
This position is responsible for ensuring a seamless transition validating
data integrity and job performance and maintaining reliable daily
operations post-migration.
The role will work closely with the existing project team for the
underlying AWS infrastructure (VPC IAM S3 EC2 networking). The resource
will focus on Cloudera cluster migration data pipeline reconfiguration and
operational stability.
Key Responsibilities
* Replicate and configure existing Cloudera cluster (HDFS YARN Hive
Spark) in the new AWS account.
* Coordinate with project team to ensure proper infrastructure
provisioning (EC2 security groups IAM roles and networking).
* Reconfigure cluster connectivity and job dependencies for the new
environment.
* Migrate and validate metadata stores (Hive Metastore job configs
dependencies).
* Validate job execution and data outputs for parity with existing
environment.
* Deploy test and operate existing Hive Spark (Scala) jobs
post-migration.
* Maintain job schedules dependencies and runtime configurations.
* Monitor job performance identify bottlenecks and apply tuning or
code-level optimizations.
* Troubleshoot failures and implement automated recovery or alerting
where applicable.
* Monitor Cloudera Manager dashboards cluster health and resource
utilization.
* Manage user roles and access within Cloudera environment.
* Implement periodic data cleanup archiving and housekeeping
processes.
* Document configurations migration steps and operational runbooks.
Required Skills and Experience:
Bachelors degree in computer science Information Systems or a
related field.
7 years of experience in data engineering or big data
development
4 years experience with Cloudera platform (HDFS YARN Hive
Spark Oozie)
Experience deploying and operating Cloudera workloads on AWS
(EC2 S3 IAM CloudWatch)
Strong proficiency in Scala Java and HiveQL; Python or Bash
scripting experience preferred
Strong proficiency in Apache Spark & Scala programming for data
processing and transformation.
Hands on experience with Cloudera distribution of Hadoop.
Hands-on experience implementing business-rules processing using
Drools.
Able to work with infrastructure DevOps and data governance
teams in a multi-disciplinary environment.
Preferred Qualifications:
Candidates with Cloudera certification (e.g. CDP Data Engineer
or Cloudera Administrator)
Experience with Cloudera version upgrades or AWS-to-AWS
environment migrations.
Experience in public-sector or large enterprise data
environments.
Duration: 12 Months
Location: Lincoln Nebraska
Cloudera Data Engineer (Cloud DevOps Administrator Expert)
Job Summary
We are seeking a Cloudera Data Engineer to support the migration of a
Medicaid Data Warehouse Implementation in AWS environment. The resource will
support the migration and continued operations of an existing
Cloudera/Hive/Scala-based data pipeline environment from one AWS account to
another.
This position is responsible for ensuring a seamless transition validating
data integrity and job performance and maintaining reliable daily
operations post-migration.
The role will work closely with the existing project team for the
underlying AWS infrastructure (VPC IAM S3 EC2 networking). The resource
will focus on Cloudera cluster migration data pipeline reconfiguration and
operational stability.
Key Responsibilities
* Replicate and configure existing Cloudera cluster (HDFS YARN Hive
Spark) in the new AWS account.
* Coordinate with project team to ensure proper infrastructure
provisioning (EC2 security groups IAM roles and networking).
* Reconfigure cluster connectivity and job dependencies for the new
environment.
* Migrate and validate metadata stores (Hive Metastore job configs
dependencies).
* Validate job execution and data outputs for parity with existing
environment.
* Deploy test and operate existing Hive Spark (Scala) jobs
post-migration.
* Maintain job schedules dependencies and runtime configurations.
* Monitor job performance identify bottlenecks and apply tuning or
code-level optimizations.
* Troubleshoot failures and implement automated recovery or alerting
where applicable.
* Monitor Cloudera Manager dashboards cluster health and resource
utilization.
* Manage user roles and access within Cloudera environment.
* Implement periodic data cleanup archiving and housekeeping
processes.
* Document configurations migration steps and operational runbooks.
Required Skills and Experience:
Bachelors degree in computer science Information Systems or a
related field.
7 years of experience in data engineering or big data
development
4 years experience with Cloudera platform (HDFS YARN Hive
Spark Oozie)
Experience deploying and operating Cloudera workloads on AWS
(EC2 S3 IAM CloudWatch)
Strong proficiency in Scala Java and HiveQL; Python or Bash
scripting experience preferred
Strong proficiency in Apache Spark & Scala programming for data
processing and transformation.
Hands on experience with Cloudera distribution of Hadoop.
Hands-on experience implementing business-rules processing using
Drools.
Able to work with infrastructure DevOps and data governance
teams in a multi-disciplinary environment.
Preferred Qualifications:
Candidates with Cloudera certification (e.g. CDP Data Engineer
or Cloudera Administrator)
Experience with Cloudera version upgrades or AWS-to-AWS
environment migrations.
Experience in public-sector or large enterprise data
environments.
Required Skills:
VPCNETWORKINGDEVOPSHADOOPSPARKAWSAPACHE SPARKJAVAHDFSHIVEQLPYTHONSCALA
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
