Spark Engineer

Req number:

Employment type:

Worksite flexibility:

Who we are

CAI is a global technology services firm with over 8500 associates worldwide and a yearly revenue of $1 billion. We have over 40 years of excellence in uniting talent and technology to power the possible for our clients colleagues and communities. As a privately held company we have the freedom and focus to do what is rightwhatever it takes. Our tailor-made solutions create lasting results across the public and commercial sectors and we are trailblazers in bringing neurodiversity to the enterprise.

Job Summary

Job Description

We are looking for a Spark Engineerwith deep expertise in distributed data processing ETL pipelines and performance tuning for high-volume data environments.This position will befull-timeandremote.

What Youll Do:

• Design develop and maintain big data solutions using Apache Spark (Batch and Streaming).

• Build data pipelines for processing structured semi-structured and unstructured data from multiple sources.

• Optimize Spark jobs for performance and scalability across large datasets.

• Integrate Spark with various data storage systems (HDFS S3 Hive Cassandra etc.).

• Collaborate with data scientists and analysts to deliver robust data solutions for analytics and machine learning.

• Implement data quality checks monitoring and alerting for Spark-based workflows.

• Ensure security and compliance of data processing systems.

• Troubleshoot and resolve data pipeline and Spar k job issues in production environments

What Youll Need

Required:

• Bachelors degree in Computer Science Engineering or related field (Masters preferred).

• 3 years of hands-on experience with Apache Spark (Core SQL Streaming).

• Strong programming skills in Scala Java or Python (PySpark).

• Solid understanding of distributed computing concepts and big data ecosystems (Hadoop YARN HDFS).

• Experience with data serialization formats (Parquet ORC Avro).

• Familiarity with data lake and cloud environments (AWS EMR Databricks GCP DataProc or Azure Synapse).

• Knowledge of SQL and experience with data warehouses (Snowflake Redshift BigQuery is a plus).

• Strong background in performance tuning and Spark job optimization.

• Experience with CI/CD pipelines and version control (Git).

• Familiarity with containerization (Docker Kubernetes) is an advantage.

Preferred:

• Experience with stream processing frameworks (Kafka Flink).

• Exposure to machine learning workflows with Spark MLlib.

• Knowledge of workflow orchestration tools (Airflow Luigi).

Physical Demands

• Ability to safely and successfully perform the essential job functions

• Sedentary work that involves sitting or remaining stationary most of the time with occasional need to move around the office to attend meetings etc.

• Ability to conduct repetitive tasks on a computer utilizing a mouse keyboard and monitor

Reasonable accommodation statement

If you require a reasonable accommodation in completing this application interviewing completing any pre-employment testing or otherwise participating in the employment selection process please direct your inquiries to or (888).