THIS IS A CONTRACT TO POSSIBLE HIRE OPPORTUNITY!
As a Senior Data Engineer you will be responsible for designing building and optimizing scalable data pipelines that power AI and analytics solutions for our clients. You will lead the architecture and deployment of data infrastructure ensuring efficiency reliability and security across large-scale datasets as well as being hands in the development process.
What youll do:
- Design develop and maintain scalable ETL/ELT data pipelines for AI and analytics applications.
- Optimize data architectures and storage solutions using Snowflake.
- Develop big data processing jobs using PySpark Spark SQL and distributed computing frameworks.
- Ensure data quality governance and security best practices across all environments.
- Implement CI/CD workflows for automated deployment and infrastructure as code (IaC).
- Collaborate with cross-functional teams (data scientists software engineers analysts) to build end-to-end data solutions.
- Lead troubleshooting and performance tuning efforts for high-volume data processing systems.
- Develop and maintain Python-based backend services to support data infrastructure.
- Implement Apache Airflow Dataplane or similar orchestration tools to automate and monitor workflows.
Qualifications :
- Strong proficiency in SQL for data processing and transformation.
- Strong knowledge of object-oriented programming in at least one language (Python Scala or Java).
- Hands-on experience deploying and managing large-scale data pipelines in production environments.
- Expertise in workflow orchestration tools like Apache Airflow Dataplane or equivalent.
- Deep understanding of cloud-based data platforms such as Databricks and Snowflake.
- Knowledge of CI/CD pipelines and infrastructure as code for data workflows.
- Familiarity with cloud environments (AWS preferred Azure or GCP) and cloud-native data processing.
- Expertise in Spark PySpark and Spark SQL with a solid understanding of distributed computing frameworks.
- Proven ability to lead projects and mentor junior engineers in a fast-paced collaborative environment.
- Experience working with Cortex is an added bonus.
Remote Work :
No
Employment Type :
Full-time