In this role youll work cross-functionally across product and data science teams to build large-scale stream and batch processing data pipelines that power Analytics Experimentation and Machine Learning. You will design a unified and groundbreaking data processing framework using Flink and/or Spark. Your work will focus on optimizing performance ensuring data quality and contributing to a long-term vision that extends the frameworks capabilities to new user scenarios and groundbreaking machine learning applications. You will collaborate closely with Siri Search and other teams to design solutions that transform raw data into datasets that drive innovation. Youll automate dataset lifecycles with strong quality standards and help partners confidently use the data for product insights.
7 years of experience designing building and maintaining distributed data processing systems at scale.
5 years of hands-on experience with stream and/or batch processing technologies such as Flink Spark Kafka Airflow Iceberg and Trino.
Strong in algorithms data structures data modeling and SQL with experience working on large-scale complex and high-dimensional datasets.
Proficient in at least one modern programming language (e.g. Java Scala and Python).
MS or BS in Computer Science Engineering Math Statistics or a related field or equivalent practical experience in data engineering.
Experience with machine learning algorithms or pipelines particularly in the context of data engineering.
Experience supporting ML engineers or data scientists with feature engineering or model data pipelines is a plus.
Familiarity with testing tools and methodologies for validating large-scale distributed data systems (e.g. data quality checks pipeline testing frameworks fault tolerance testing).
Proven software engineering fundamentals including experience with design testing version control and CI/CD best practices.
Comfortable working independently in a fast-paced ambiguous environment.
Excellent communication and problem-solving skills.
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.