Senior Dev Engineer- Data Platform
Senior Dev Engineer- Data Platform
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
Responsibilities include but are not limited to the following:
Optimize Spark clusters for cost efficiency and performance by implementing robust monitoring systems to identify bottlenecks using data and metrics. Provide actionable recommendations for continuous improvement
Optimize the infrastructure required for extraction transformation and loading of data from a wide variety of data sources using SQL and AWS big data technologies
Work with data and analytics experts to strive for greater cost efficiencies in our data systems
Requirements
Who you are:
Experience with processing large workloads and complex code on Spark clusters
Proven experience in setting up monitoring for Spark clusters and driving optimization based on insights and findings
Experience in designing and implementing scalable Data Warehouse solutions to support analytical and reporting needs
Strong analytic skills related to working with unstructured datasets
Build processes supporting data transformation data structures metadata dependency and workload management
Working knowledge of message queuing stream processing and highly scalable big data data stores
Experience using the following software/tools:
Expertise with Python and Jupyter notebooks is a MUST.
Experience with big data tools: Spark Kafka etc.
Experience with relational SQL and NoSQL databases including Postgres and Cassandra
Experience with data pipeline and workflow management tools: Azkaban Luigi Airflow etc.
Experience with AWS cloud services: EC2 EMR RDS Redshift
Working knowledge of streamprocessing systems: Storm SparkStreaming etc. is a plus
We are seeking a Senior Network Operations Engineer who: Holds a degree in Computer Science or equivalent, with 10+ years of professional experience, including experience in an enterprise SaaS organization. Possesses expert-level knowledge in Firewall, IDS/IPS, DNS, BGP, OSPF, AWS, and Azure network constructs. Has experience deploying physical network and security devices, as well as cloud networking solutions like web application firewalls. Demonstrates proficiency in coding with Python, Java, Ruby, or equivalent languages. Familiarity with deploying cloud network infrastructure using Terraform and CloudFormation, or experience with Chef & Ansible. Exhibits confidence in project ownership and issue resolution. Has a passion for excellence, natural problem-solving skills, and critical thinking abilities. Possesses a solid understanding of networking systems and protocols, particularly in a public cloud environment. Thrives in a collaborative and supportive work environment. Desirable qualifications (not essential) include: Experience with monitoring technologies like Prometheus, Influx, Grafana, Elasticsearch, New Relic, Dynatrace or Datadog. Familiarity with container technologies such as Kubernetes, ECS, and EKS. Knowledge of databases like MySQL, MS-SQL, and NoSQL. Understanding of security and certificate management. Relevant operating system, networking, and cloud certifications.
Employment Type
Full Time
