DataOps Engineer

We are looking for a midlevel engineer to build and operate a data platform that uses Apache Iceberg as the lakehouse table format and Dockerbased microservices (Spark Flink Presto etc.). you will own the endtoend delivery pipeline monitoring security and incident response ensuring the platform runs reliably at scale.

Key Responsibilities

• Iceberg operations: support tables manage schema changes partitions snapshot retention and keep the catalog (Hive Metastore AWS Glue Nessie ) synchronized.
• Docker image creation & testing: write multistage Dockerfiles for Spark/Flink/Presto run local test environments with DockerCompose and conduct vulnerability scans (Trivy Snyk ).
• Data pipeline development: build ETL/ELT jobs that ingest raw data and write to Iceberg tables; add simple streaming components using Kafka Pulsar or Kinesis when needed.
• CI/CD automation: configure pipelines (GitHub Actions GitLab CI Azure DevOps ) to lint Dockerfiles scan images version Iceberg metadata and deploy pipelines without downtime.
• Automation with Ansible/Python: script cluster provisioning catalog configuration vacuum/compaction and other routine housekeeping tasks.
• Observability: instrument services with OpenTelemetry Prometheus Grafana and Loki; create dashboards showing pipeline latency resource usage table health and error rates; set up basic alerts.
• SLA monitoring: measure data freshness job success rates and query response times against agreedupon targets and report deviations.
• Incident response: join the oncall rotation perform firstline diagnosis and resolution of pipeline failures Iceberg metadata issues or container crashes; write concise rootcause analyses and suggest improvements.
• Security & compliance support: help enforce image signing mTLS IAM roles and bucket policies; collaborate with the security team to meet GDPR HIPAA or ISO 27001 requirements.
• Knowledge sharing: keep internal documentation up to date and run short tech demos or brownbag sessions on Iceberg Docker best practices and automation techniques.

   

Qualifications :

• Bachelors degree in Computer Science IT Data Engineering or a related field (Masters a plus).
• 5 years of handson experience building and operating largescale data platforms (lakehouse datawarehouse or bigdata ecosystems).
• Proven production experience with Apache Iceberg (table creation partition management schema evolution catalog integration).
• Strong Docker skills: multistage builds DockerCompose testing routine image security scanning.
• Experience with at least one major dataprocessing engine (Spark Flink or Presto/Trino) and its connection to Iceberg tables.
• Proficiency in Python and/or Ansible for automating infrastructure and platform tasks.
• Experience building CI/CD pipelines that include Docker linting vulnerability scanning and automated deployment of datapipeline code.
• Familiarity with observability tooling (Prometheus Grafana OpenTelemetry Loki) and ability to create useful alerts and dashboards.
• Ability to respond to incidents write clear rootcause analysis reports and contribute to postmortem actions.
• Willingness to participate in an oncall rotation as a firstline responder.
• Availability to work onsite in New Jersey for the initial assignment and relocate to Dallas by October 2026.

Preferred Qualifications

• Experience with cloudnative data services on AWS Azure or GCP (EMR Dataproc Synapse etc.).
• Familiarity with other lakehouse formats such as Delta Lake or Apache Hudi and ability to evaluate tradeoffs against Iceberg.
• Knowledge of streaming platforms (Kafka Pulsar Kinesis) and realtime processing patterns.
• Relevant certifications (Databricks Lakehouse Associate Google Professional Data Engineer AWS Certified Data Analytics Specialty etc.).
• Background supporting data platforms in regulated industries (pharma finance healthcare) and understanding of associated compliance frameworks.

Additional Information :

All your information will be kept confidential according to EEO guidelines.

 *** NO C2C ***

Remote Work :

No

Employment Type :

Contract