Senior Data Engineer
Share
Job Location:
Monthly Salary:
Posted on:
20 hours ago
Vacancies:
1 Vacancy
Job Summary
We are looking for Senior Data Engineer to join Sopra Steria Polska a team for a client in the banking sector. You will be working in a stable environment in an international team working for one of the most popular Dutch banks.
Note that we can only offer cooperation to people who are located in Poland and are willing to commute to our office in Katowice or Gdańsk Poland.
Responsibilities:
Build and maintain data pipelines and platforms in Databricks and Azure using Apache Spark.
Ensure data security governance and compliance using Unity Catalog and best practices.
Develop reusable Python libraries for integration across Databricks and Azure Functions.
Designing and implementing batch data pipelines in Databricks using Apache Spark (PySpark) and Delta Lake.
Applying Spark performance tuning techniques (e.g. partitioning caching broadcast joins shuffle optimization) for large-scale data processing.
Setting up and managing Unity Catalog for data governance and security.
Building Python libraries for reusable business logic and utilities used in Databricks Spark jobs and Azure Functions to expose APIs.
Developing CI/CD pipelines in Azure DevOps for data workflows and infrastructure deployment.
Collaborating with stakeholders to understand data requirements and deliver solutions that create customer value.
Tech stack on the project:
.Net/Databricks
Python API
Apache Spark
Azure
SQL Server
Qualifications :
Must have requirements:
5 years of experience in data engineering roles.
Deep hands-on experience with Databricks including Unity Catalog Delta Lake and Lakehouse architecture.
Strong proficiency in Apache Spark including PySpark Spark SQL and Spark performance tuning.
Experience creating Python libraries for reusable components and API integration.
Experience with Azure Cloud services (Data Lake Key Vault Active Directory App Services).
Familiarity with SQL Server and relational database concepts.
Understanding of data security principles and compliance requirements.
Experience building CI/CD pipelines in Azure DevOps for data workflows.
Infrastructure as Code (Bicep or Terraform).
Good communication skills and a team player.
Experience with Agile ways of working.
English B2/C1.
Being open to occasional business trips abroad and visits in our office in Katowice or Gdańsk.
EU Citizenship.
Nice to have requirements:
Experience with finance or banking systems.
Knowledge of streaming architectures (though primary focus is batch).
Familiarity with Spark Structured Streaming and MLlib.
Advanced cost optimization strategies for Spark workloads in cloud environments.
Additional Information :
What we offer:
BENEFITS (UoP): Luxmed Medicover Sport Worksmile educational platforms languages learning platform referral bonus copyrights life insurance workation
DEVELOPMENT OPPORTUNITIES (UoP and B2B): certifications (paid by the company) conferences Tech Lunches possibility to join our Communities (Project Management Architecture Security Process Management Leadership AI and Cloud)
The recruitment process for this position consists of three stages:
a short phone call with a recruiter (30 min max)
one-hour long interview on Teams (with both general and technical questions)
one-hour long interview on Teams (with project representatives)
Salary range:
Senior: UOP: 12 000 -17 000 PLN gross/month.
At our organization we are committed to fighting against all forms of discrimination. We foster a work environment that is inclusive and respectful of all differences.
Remote Work :
Yes
Employment Type :
Full-time
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
Sopra Steria, a major Tech player in Europe with 52,000* employees in nearly 30 countries, is recognised for its consulting, digital services and solutions. It helps its clients drive their digital transformation and obtain tangible and sustainable benefits. The Group provides end-to- ... View more
