Data migration (Spark)
Share
Job Location:
San Francisco, CA - USA
Monthly Salary:
Not Disclosed
Posted on:
12 hours ago
Vacancies:
1 Vacancy
Job Summary
Greetings from Inherent Technologies! Hope this email finds you in the best of your health
My name is Bhanu and Im an IT recruiter at Inherent Technologies.
Position: Data migration (Spark)
Location: San Francisco CA***REMOTE***
Duration: 1 Year
Job Description:
Mandatory Skills: Sparks jobs migration & Kubernetes skills
A Spark job migration specialist migrates data pipelines JAR tasks and analytics workloads from legacy systems (like Hadoop/CDH or AWS EMR) to ACOS modern platforms This involves refactoring code (e.g. Hive to PySpark) performance testing and updating Spark 2.x to 3.x.
Key Job Responsibilities
Workload Migration: Migrate JVM workloads and Spark-Submit tasks to Databricks JAR tasks or Notebook tasks.
Pipeline Re-engineering: Convert existing HiveQL scripts and Oozie workflows into optimized Spark SQL or PySpark applications.
Refactoring: Adapt data pipelines from Azure Synapse to any cloud platform including updating library dependencies and notebook references.
Performance Optimization: Implement Adaptive Query Execution (AQE) in Spark 3 to improve shuffle performance and fix skew joins.
Testing & Validation: Perform regression testing to ensure output consistency between old and new systems using validation scripts.
Job Customization: Use () to label monitor and troubleshoot specific Spark tasks in the UI.
Job Description/Profile
Role: Big Data Migration Engineer (Spark)
Experience: 5 years experience with Apache Spark (PySpark/Scala) and Cloud platforms (Azure/AWS).
Requirements:
Strong experience with HDFS Hadoop ecosystem (Hive Spark HBase MapReduce).
Experience in data migration to cloud/enterprise data platforms.
Knowledge of:
Data ingestion tools (Sqoop Kafka NiFi etc.)
Cloud storage (ADLS S3 Blob Storage)
Distributed processing frameworks
SQL and performance tuning expertise.
Experience in scripting (Python Shell Scala).
Key Migration Focus Areas
Data Pipelines: Ensuring schema evolution data correctness and testing with golden datasets.
Job Definitions: Reconfiguring job properties cluster settings and Spark configurations.
