Lead Data Engineer
Share
Job Location:
Monthly Salary:
Posted on:
11 hours ago
Vacancies:
1 Vacancy
Job Summary
TransUnions Job Applicant Privacy Notice
What Well Bring:
What Youll Bring:We are looking for a Lead Data Engineer to join our growing Data Engineering and Analytics Practice who will drive building next generation suite of products and platform by designing coding building and deploying highly scalable and robust solutions. You will be based both from our offices in Pune and working remotely as part of our flex together this fast-paced role you will work with Business Stakeholders to achieve business goals. This exciting role will offer a host of development opportunities as part of a growing global business.
What Youll Bring:
Key Responsibilities:
- Design build test and deploy innovative Big Data solutions at scale including data lakes data warehouses and real-time analytics.
- Extract clean transform and analyze vast amounts of raw datafrom various data sources.
- Build robust data pipelines and API integrationswith various internal systems.
- Work across all stages of the data lifecycle including data ingestion storage processing and visualization.
- Implement best practices in data governance security and complianceacross all data analytics processes.
- Estimate effort identify risks and plan execution effectively.
- Proactively monitor identify and escalate issues or root causesof systemic issues.
- Enable data scientists business and product partnersto fully leverage our platform.
- Engage with business stakeholders to understand client requirementsand build technical solutions and delivery plans.
- Evaluate and communicate technical risks effectivelyand ensure assignments are delivered on schedule with desired quality.
- Provide end-to-end big data solutions and design detailsto data engineering teams.
- Demonstrate excellent analytical and problem-solving skills.
- Exhibit excellent communication skills with experience communicating with senior business stakeholders.
- Lead technical delivery on use cases plan and delegate tasks to junior team members and oversee work from inception to final product.
Skills & Experience:
Essential:
- Bachelors degree in Computer Science Engineering Statistics or a related field
- 8 years of data engineering experience with at least 3 years in senior roles.
- 5 years of experience in Big Data technologies(e.g. Spark Hive Hadoop Databricks).
- Strong experience designing and implementing data pipelines.
- Excellent knowledge of data engineering concepts and best practices.
- Proven ability to lead mentor inspire and support junior team members.
- Ability to lead technical deliverables autonomouslyand guide junior data engineers.
- Strong attention to detail and adherence to best practices.
- Experience in designing solutions using batch data processing methods real-time streams ETL processes and business intelligence tools.
- Experience designing logical data models and physical data models including data warehouse and data mart designs.
- Strong SQL knowledge and experience(T-SQL working with SQL Server SSMS).
- Advanced proficiency with Apache Spark including PySpark and SparkSQL for distributed data processing.
- Working knowledge of Apache Hive.
- Proficiency in Python Pandas PySpark(Scala/Java knowledge is desirable).
- Knowledge of Delta Lake concepts and common data formats Lakehouse architecture.
- Source control with Git.
- Expertise in designing and implementing scalable data pipelines and ETL processesusing the GCP data stack including BigQuery Dataflow Pub/Sub Cloud Storage Cloud Composer Cloud Functions Dataproc (Spark).
- Expertise in building and managing ETL workflows using Apache Airflow including DAG creation scheduling and error handling.
- Knowledge of CI/CD conceptsand experience designing CI/CD for data pipelines.
- Software engineering principles including:
- Object-oriented programming (OOP) principles.
- Design patterns and their application in data engineering.
- Software development lifecycle (SDLC).
- Agile methodologies and practices.
- Unit testing integration testing and test-driven development (TDD).
- Performance optimization and scalability considerations.
Desirable:
- Experience with streaming services such as Kafkais a plus.
- R & Sparklyr experienceis a plus.
- Knowledge of MLOps concepts AI/ML lifecycle management and MLflow.
- Expertise in writing complex highly optimized queries across large data setsto write data pipelines and data processing layers.
- Jenkins experienceis a plus.
Relevant certifications(e.g. Google Cloud Professional Data Engineer).
Impact Youll Make:
TransUnion a place to grow:
We know that it is unrealistic to expect candidates to have each and every aspect of the essential and/or desirable skills listed above if there is something you cant tick off right now good you can learn here!
Impact you will make:
Enable Decision Making across the organization using data driven culture.
This is a hybrid position and involves regular performance of job responsibilities virtually as well as in-person at an assigned TU office location for a minimum of two days a week.TransUnion Job Title
Specialist IV Data Science and AnalyticsRequired Experience:
IC
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
Company Overview TransUnion is a leading global risk and information solutions provider. Our mission is to help people worldwide access opportunities that lead to a higher quality of life. We accomplish this by delivering solutions that help organizations optimize their risk-based dec ... View more
