Pyspark Data Engineer
Share
Job Location:
Monthly Salary:
Posted on:
18 hours ago
Vacancies:
1 Vacancy
Job Summary
Thales is a global technology leader trusted by governments institutions and enterprises to tackle their most demanding challenges. From quantum applications and artificial intelligence to cybersecurity and 6G innovation our solutions empower critical decisions rooted in human intelligence. Operating at the forefront of aerospace and space cybersecurity and digital identity were driven by a mission to build a future we can all trust.
Present in India since 1953 Thales is headquartered in Noida and has other operational offices and sites spread across Delhi Gurugram Bengaluru and Mumbai among others. Over 2200 employees are working with Thales and its joint ventures in India. Since the beginning Thales has been playing an essential role in Indias growth story by sharing its technologies and expertise in Defence Aerospace and Cyber & Digital sectors. Thales has two engineering competence centres in India - one in Noida focused on Cyber & Digital business while the one in Bengaluru focuses on hardware software and systems engineering capabilities for both the civil and defence sectors serving global needs. The Group has also established an MRO (Maintenance Repair & Overhaul) facility in Gurugram to provide comprehensive avionics maintenance and repair services to Indian airlines and support the growth of the local aviation industry.Job Responsibilities:
- Expert in Python with knowledge of PySpark
- Writing distributed data processing code with PySpark
- Optimizing Spark jobs using DataFrames API and RDDs
- Working with Spark SQL for querying large datasets (structured sql/mysql database)
- Implementing UDFs (User Defined Functions) for custom transformations
- Performance tuning (caching partitioning broadcast joins threading multiprocessing Predicate Pushdown)
- ETL development using Databricks workflows and Delta Live Tables
- Data pipelines with Databricks Jobs
- Streaming data processing with Structured Streaming ( event hub or AWS SQS)
- Non structured file processing like parquet columnar data format
- Responsible for writing reusable testable and efficient code ensuring the applications performance security and scalability.
Minimum Skills/Experience:
- Bachelors degree in Engineering Computer Science or related study
- Proven experience as a Python language with PySpark
- Minimum 5 years of experience
- Understanding of data structures data modeling and software architecture
- Ability to write robust code in Python
- Outstanding analytical and problem-solving skills
- Experience with Git JIRA Confluence
- Experience in working on Linux OS.
At Thales were committed to fostering a workplace where respect trust collaboration and passion drive everything we do. Here youll feel empowered to bring your best self thrive in a supportive culture and love the work you do. Join us and be part of a team reimagining technology to create solutions that truly make a difference for a safer greener and more inclusive world.
Required Experience:
IC
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
In all critical environments - air, land, sea, space and cyberspace - decision-makers, operators, crews and members of our armed services and security forces are faced with millions of important decisions every day. It is in supporting these people that Thales in the United States ha ... View more
