Sr. Data Engineer (Ab Initio+ Python)
Sr. Data Engineer (Ab Initio+ Python)
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
Key Responsibilities
Lead Agile Development: Guide and support multiple Agile teams focused on data Extract Ingestion and transformation.
Modernize Legacy Systems: Migrate data pipelines from Ab Initio and Filesystem to modern technologies such as PySpark S3 Airflow Parquet and Iceberg.
Full-Stack Engineering: Design and develop scalable backend services using PySpark and Python.
Data Platform Enablement: Support ingestion of 300 data feeds into the platform to ensure timely nightly batch processing.
Cross-Functional Collaboration: Partner with business stakeholders and product owners to understand requirements and deliver effective solutions.
Agile Execution: Working with both Kanban and scrum teams and should be familiar with both and check-ins and managing tasks via Jira.
Mentorship and Communication: Provide technical guidance foster collaboration and proactively seek help when encountering blockers.
Platform Transition Support: Contribute to the migration from legacy systems to a new data platform over the next two years.
BAU and Strategic Support: Balance business-as-usual responsibilities while contributing to long-term platform modernization.
Documentation and Data Modeling: Maintain clear technical documentation and demonstrate a strong understanding of columnar data structures.
Experience on the different file format systems (Parquet ORC AVRO).
Experience on the code containerization deployments using Docker / Kubernetes.
Java background would be a plus.
Good Knowledge on large scale ETL Based frameworks
Experience on ETL tool (AbInitio).
Required Skills & Experience
Top Technical Skills
10 Years as Data Engineer
3 years of experience with PySpark S3 Iceberg Git Python Airflow and Parquet
5 years of experience with SQL
Experience with Agile methodologies and tools like Jira
Familiarity with Kafka
Experience with GitHub Copilot Web Services Visual studio IntelliJ and Gradle
Experience in monitoring tools like Grafana or Prometheus
Preferred Qualifications
Proven experience leading Agile teams and mentoring junior developers
Strong communication skills and the ability to collaborate with business stakeholders
Comfortable working in both Scrum and Kanban model with frequent scrum check-ins
Ability to identify blockers and proactively seek help when needed
Experience working in a regulated environment with a focus on compliance and data governance.
2 years of working with Ab Initio graphs and plans
Employment Type
Full-time
