PythonSparkAI Developer

Our client is seeking an experienced Python and Spark Data Engineer to develop production-grade data pipelines and support the replatforming of legacy workloads into a modern lakehouse environment. This role is suited to an autonomous engineer who values clean tested code and can work confidently from technical specifications and architecture decision records.

Requirements:

• 4 years professional Python development experience.
• 2 years production experience building Spark/PySpark or comparable distributed data pipelines.
• Strong Python 3.12 skills including type hints Pydantic ABC-based patterns and modern packaging.
• Strong Apache Spark knowledge including DataFrame API Spark SQL partitioning tuning and driver/executor architecture.
• Production experience with Delta Lake or equivalent technologies such as Iceberg or Hudi.
• Strong SQL skills and the ability to translate complex legacy T-SQL logic into Spark SQL.
• Experience with pytest fixtures test markers and tiered testing strategies.
• Practical FastAPI or equivalent REST API development experience.
• Experience with Docker and Compose-based development environments.
• Strong Git and CI/CD discipline including GitHub flow Ruff static type checking and automated test gates.
• Bachelors degree in Computer Science Engineering or equivalent experience.

Should you meet the requirements for this position please email your updated CV attached to alternatively contact or visit our website . Correspondence will only be conducted with short listed candidates. Should you not hear from us within 3 days please consider your application unsuccessful.