Senior Lead Data Engineer R
Share
Job Location:
San Francisco, CA - USA
Monthly Salary:
Not Disclosed
Posted on:
30+ days ago
Vacancies:
1 Vacancy
Job Summary
Senior Lead Data Engineer
Primary Skills
- Architect construct test and maintain ELT pipelines using Hadoop and Spark SQL
Specialization
- Hadoop and Spark SQL
Job requirements
- Key Responsibilities:
- Architect construct test and maintain ELT pipelines using Hadoop and Spark SQL
- - Manage and optimize the infrastructure ensuring reliability and performance
- -Design and implement data models and schemas for enterprise data warehouse and data marts
- -Write and maintain scripts in Python for various applications
- -Develop optimize troubleshoot complex SQL queries and database structures for relational and non-relational databases
- - Work closely with analysts and product managers to understand data requirements and deliver high-quality data
- -Provide technical guidance and mentorship to other team members fostering best practices in data engineering
- -Monitor system performance identify bottlenecks implement solutions to ensure high data availability
- -High proficiency in the Hadoop ecosystem (HDFS MapReduce Yarn) and Apache Spark Scala for large scale data processing
- -Good knowledge in Python for data manipulation and scripting
- -Mastery of advanced SQL and deep experience working with relational databases and datawarehouses
- -Hands-on experience with workflow management tools like Apache Airflow
- -Familiarity with Unix shell scripting for automation and system management
Required Skills:
$93100 - $133000 a year
We may use artificial intelligence (AI) tools to support parts of the hiring process such as reviewing applications analyzing resumes or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed please contact us.
Required Experience:
Senior IC
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
Brillio is a global leader in Enterprise Digital Transformation Solutions, providing strategic consulting services and solutions using emerging technologies.
