Were looking for a seasoned Big Data Engineering Lead with expertise in Scala Python and PySpark to lead our client data engineering team. Youll be responsible for designing and implementing scalable efficient and fault-tolerant data pipelines as well as mentoring team members and driving technical innovation.
Key Responsibilities:
- Design and develop large-scale data pipelines using Scala Python and PySpark
- Lead and mentor a team of data engineers to build and maintain data architectures
- Collaborate with cross-functional teams to identify data requirements and implement data-driven solutions
- Ensure data quality integrity and security across all data pipelines
- Develop and maintain technical documentation for data pipelines and architectures
- Stay up-to-date with industry trends and emerging technologies in big data cloud computing and data engineering
- Drive technical innovation and recommend new tools and technologies to improve data engineering capabilities
Requirements:
- 5 years of experience in big data engineering with expertise in Scala Python and PySpark
- Strong experience with big data technologies such as Apache Spark Hadoop and Kafka
- Experience with cloud-based data platforms such as AWS GCP or Azure
- Strong understanding of data architecture data governance and data security
- Excellent leadership and mentoring skills with experience leading high-performing teams
- Strong communication and collaboration skills with ability to work with cross-functional teams
- Bachelors or Masters degree in Computer Science Engineering or related field