Staff Platform Engineer - Big Data Platform Management
Share
Job Location:
Monthly Salary:
Posted on:
30+ days ago
Vacancies:
1 Vacancy
Job Summary
CSIT develops products to advance the national security interests of Singapore. We use our products in a wide range of operations including but not limited to Counter-terrorism and Computer NetworkDefence. We are looking for talented Senior Big Data Platform Staff Engineer. Join the awesome CSIT family and use cutting-edge technologies to protect the nation.
Role
- We are seeking an experienced Big Data Platform Staff Engineer to join our team responsible for managing and optimizing our Big Data infrastructure platform. The successful candidate will have a deep understanding of Big Data technologies architectures and systems as well as experience with managing large-scale data processing and analytics environments. The role will involve designing implementing and maintaining the Big Data infrastructure platform ensuring high availability scalability and performance.
Responsibilities
- Design implement and manage the Big Data infrastructure platform including Hadoop Spark NoSQL databases and data lakes.
- Manage Big Data clusters including monitoring troubleshooting and optimizing cluster performance as well as managing job scheduling and resource allocation.
- Design and implement storage solutions for Big Data workloads including HDFS S3 and other object storage systems.
- Ensure the security and governance of the Big Data infrastructure platform including managing access controls implementing data encryption and conducting regular security audits.
- Collaborate with data engineers and data scientists to optimize Big Data application performance and provide technical support and guidance on using the Big Data infrastructure platform.
- Plan and manage Big Data resource capacity including forecasting procurement and deployment of new hardware and software.
Requirements (Minimum Qualifications)
- Bachelors degree in Computer Science Engineering or a related field.
- At least 8 years of experience in managing Big Data Platforms. Proven experience with technologies like Spark Kafka Hadoop Elastic Stack Kudu NiFi and Kafka.
- Strong knowledge of Big Data architectures including data lakes data warehouses and data pipelines.
- Experience in troubleshooting and resolving issues of the Big Data stack ranging from hardware networking operating system to software
- Strong understanding of data storage systems including HDFS S3 and other object storage systems.
- Experience with scripting languages such as Python Scala or Java.
- Experience in managing physical and virtual infrastructures.
- Experience in supporting search workloads in multi-terabytes and billions of records.
- Experience in distributed systems.
- Proven experience in using DevOps tools and CI/CD pipelines effectively.
- Experience in leading engineering teams.
Nice to Have
- Certifications in Big Data such as Hadoop Certified Developer or Spark Certified Developer.
- Experience with cloud-based Big Data platforms such as AWS EMR Google Cloud Dataproc or Azure HDInsight.
- Experience with machine learning and deep learning frameworks such as TensorFlow PyTorch or Scikit-learn.
- Familiarity with containerization such as Docker or Kubernetes.
- Experience with agile development methodologies and version control systems such as Git.
- Experience with cloud computing platforms such as AWS Azure or Google Cloud.
- Experience with containerization such as Docker or Kubernetes.
Why join us
- The work is purposeful and meaningful
- You will work with the best engineers
- We work with modern technologies and tech stacks
- We have excellent engineering culture and work-life balance
- We aspire to engineering and operational excellence
- We empower to innovate
- We grow together as a family
As CSIT is an agency under the Ministry of Defence (Singapore) only Singapore Citizens will be considered.
Required Experience:
Staff IC
Key Skills
- Computer Science
- Docker
- Kubernetes
- Python
- VMware
- C/C++
- Go
- System Architecture
- gRPC
- OS Kernels
- Perl
- Distributed Systems
About Company
Learn how Forward SG initiatives can benefit you at our interactive roadshows
