Data Engineer – IAM Modernization
Share
Job Location:
Charlotte, VT - USA
Monthly Salary:
Not Disclosed
Posted on:
Yesterday
Vacancies:
1 Vacancy
Job Summary
Role Summary
We are looking for a Data Engineer with strong expertise in the Hadoop ecosystem ETL development and data transformation logic focused on modernizing IAM data flows. This role involves terminating legacy batch SQL jobs re-pointing feeds via NDM and pushing IAM data into a Cyber Data Lake built on Hadoop. The engineer will design and implement push-based near real-time ingestion pipelines with transformation logic applied during ingestion enabling scalable secure and audit-ready IAM datasets.
Key Responsibilities
- Modernization & Migration
- Decommission existing batch SQL jobs and migrate to modern ingestion architecture.
- Re-point upstream and downstream feeds using NDM for secure data transfers.
- Onboard IAM datasets into a Cyber Data Lake (Hadoop) with optimized storage formats (Parquet/ORC) and partitioning.
- Pipeline Development & Transformation
- Build ETL/ELT pipelines using Spark/Hive to perform transformations during ingestion (schema mapping normalization deduplication).
- Implement push-based near real-time ingestion (event-driven or micro-batch) instead of scheduled pulls.
- Apply complex IAM-specific transformation logic for identities accounts (human & non-human) roles entitlements and policies.
- Data Quality & Observability
- Define and automate data quality checks (completeness accuracy referential integrity).
- Implement monitoring logging and alerting for ingestion pipelines and NDM transfers.
- Performance & Optimization
- Tune Spark jobs Hive queries and storage strategies for scale and cost efficiency.
- Optimize resource allocation and implement backpressure controls for streaming ingestion.
- Security & Compliance
- Enforce least privilege and secure handling of sensitive IAM attributes (PII).
- Maintain metadata lineage and data dictionaries; ensure compliance with audit requirements.
- Client Collaboration
- Work onsite with client IAM teams application owners and auditors to clarify requirements and deliver modernization milestones.
- Maintain detailed documentation (ERDs flow diagrams runbooks).
Required Qualifications
- 5 8 years of experience in Data Engineering with exposure to IAM data and modernization projects.
- Strong hands-on experience with Hadoop ecosystem: HDFS Hive Spark (SQL/Scala/PySpark).
- Proven experience in ETL/ELT design data transformation logic and pipeline optimization.
- Experience terminating legacy batch SQL jobs and migrating to modern ingestion patterns.
- Practical knowledge of NDM for secure data transfers.
- Expertise in push-based ingestion and near real-time data processing.
- Understanding of IAM concepts: identities service/non-human accounts roles entitlements policies.
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
