Senior Data Engineer

American IT Systems

Job Location:

Sunnyvale, CA - USA

Monthly Salary: Not Disclosed

Posted on: 16 days ago

Vacancies: 1 Vacancy

Job Summary

Only W2 Candidates required Initial screening/ Glider is required for this role

Senior Data Engineer

Location Sunnyvale CA
Onsite Requirement - Y/N- Y
Number of days onsite 5 Days
If Onsite Office Address

Rate :: 60-65/hr on w2 Max

Mandatory Areas
Must Have Skills Data Engineer
Skill 1 big data spark pyspark apache airflow etl process
Skill 2 4 years in GCP

Description:

We are seeking a Data Engineer with Spark & Streaming skills builds real-time scalable data pipelines using tools like Spark Kafka and cloud services (GCP) to ingest transform and deliver data for analytics and ML.

Responsibilities:

As a Senior Data Engineer you will Design develop and maintain ETL/ELT data pipelines for batch and real-time data ingestion transformation and loading using Spark (PySpark/Scala) and streaming technologies (Kafka Flink).
Build and optimize scalable data architectures including data lakes data warehouses (Big Query) and streaming platforms.
Performance Tuning: Optimize Spark jobs SQL queries and data processing workflows for speed efficiency and cost-effectiveness Data Quality: Implement data quality checks monitoring and alerting systems to ensure data accuracy and consistency. Required Skills & Qualifications: Programming: Strong proficiency in Python SQL and potentially Scala/Java.
Big Data: Expertise in Apache Spark (Spark SQL DataFrames Streaming).
Streaming: Experience with messaging queues like Apache Kafka or Pub/Sub.
Cloud: Familiarity with GCP Azure data services. Databases: Knowledge of data warehousing (Snowflake Redshift) and NoSQL databases.
Tools: Experience with Airflow Databricks Docker Kubernetes is a plus.
Experience Level Total IT Experience - Minimum 8 years GCP - 4 years of recent GCP experience

Only W2 Candidates required Initial screening/ Glider is required for this role Senior Data Engineer Location Sunnyvale CA Onsite Requirement - Y/N- Y Number of days onsite 5 Days If Onsite Office Address Rate :: 60-65/hr on w2 Max Mandatory Areas Must Have Skills Data Engineer Sk...