Senior Data Engineer (Azure Data FactoryDatabricksPySpark)
Job Location:
Monthly Salary:
Posted:
Application Deadline:
Vacancies:
Job Summary
Publicis Sapient is a digital transformation partner helping established organizations get to their future digitally enabled state both in the way they work and the way they serve their customers. We help unlock value through a start-up mindset and modern methods fusing strategy consulting and customer experience with agile engineering and problem-solving creativity. United by our core values and our purpose of helping people thrive in the brave pursuit of next our 20000 people in 53 offices around the world combine experience across technology data sciences consulting and customer obsession to accelerate our clients businesses through designing the products and services their customers truly value.
- Design and build scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks (ADB)
- Develop PySpark-based transformation logic for large-scale data processing including joins aggregations and window functions
- Architect and implement hybrid data integrations between cloud and on-premises systems
- Enable secure connectivity from Databricks to on-prem databases using enterprise-grade patterns
- Build and optimize data models across Cosmos DB and Snowflake for different workloads
- Implement monitoring logging and error-handling mechanisms to ensure reliability and performance
- Collaborate with cross-functional teams to define standards patterns and best practices for data engineering solutions
- Strong experience building data pipelines with Azure Data Factory and Azure Databricks
- Hands-on expertise in PySpark for distributed data processing
- Advanced knowledge of SQL including complex joins and performance optimization
- Experience working with Cosmos DB and Snowflake
- Solid programming skills in Python Scala or Java
- Experience with Kafka or event-driven architectures
- Strong understanding of data architecture and distributed systems
- Proven ability to deliver production-grade data solutions in complex environments
- DataFrame transformations:
selectfiltergroupByaggwithColumn - Window functions:
rankrownumberlaglead - Joins: inner left broadcast joins and usage scenarios
- Reading/writing data: Parquet Delta CSV
- UDFs: syntax registration and performance tradeoffs
- Ability to produce production-ready code without pseudo-code
- Cluster types auto-scaling and cluster policies
- Notebook orchestration workflows and job scheduling
- Delta Lake: ACID schema evolution time travel OPTIMIZE VACUUM
- Unity Catalog: governance lineage and access control
- Integration with Azure Data Factory
- JDBC/ODBC connectivity from Databricks
- Secure credential management with Azure Key Vault and secret scopes
- Network architecture: VNet injection private endpoints Self-Hosted IR
- End-to-end pipeline design from on-prem to cloud
- Performance optimization and error handling in hybrid environments
- Experience designing end-to-end data architectures in Azure ecosystems
- Knowledge of Delta Lake features (ACID transactions schema evolution time travel)
- Experience with Databricks Workflows and orchestration
- Understanding of data governance (Unity Catalog)
- Experience with hybrid cloud/on-prem integration patterns
- Exposure to performance optimization at scale
- An inclusive workplace that promotes diversity and collaboration.
- Access to ongoing learning and development opportunities.
- Competitive compensation and benefits package.
- Flexibility to support work-life balance.
- Comprehensive health benefits for you and your family.
- Generous paid leave and holidays.
- Wellness program and employee assistance.
As part of our dedication to an inclusive and diverse workforce Publicis Sapient is committed to Equal Employment Opportunity without regard for race color national origin ethnicity gender protected veteran status disability sexual orientation gender identity or religion. We are also committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or an accommodation due to a disability you may contact us at
Required Experience:
Senior IC
About Company
Publicis Media is one of the four solutions hubs of Publicis Groupe ([Euronext Paris FR0000130577, CAC 40], alongside Publicis Communications, Publicis.Sapient and Publicis Healthcare. Led by Steve King, CEO, Publicis Media is powered by its five global brands, Starcom, Zenith, Spark ... View more
