EY GDS Consulting AI And DATA AWS-Azure Databricks Senior E

At EY youll have the chance to build a career as unique as you are with the global scale support inclusive culture and technology to become the best version of you. And were counting on your unique voice and perspective to help EY become even better too. Join us and build an exceptional experience for yourself and a better working world for all.

Job Description for Senior Data Engineer

Rank Senior

• Objectives and Purpose

  • The Senior Data Engineer ingests builds and supports large-scale data architectures that serve multiple downstream systems and business users. This individual supports the Data Engineer Leads and partners with Visualization on data quality and troubleshooting needs.
  • The Senior Data Engineer will:
    • Clean aggregate and organize data from disparate sources and transfer it to data warehouses.
    • Support development testing and maintenance of data pipelines and platforms to enable data quality to be utilized within business dashboards and tools.
    • Create maintain and support the data platform and infrastructure that enables the analytics front-end; this includes the testing maintenance construction and development of architectures such as high-volume large-scale data processing and databases with proper verification and validation processes.

  Your key responsibilities

  • Develop and maintain scalable data pipelines in line with ETL principles and build out new integrations using AWS native technologies to support continuing increases in data source volume and complexity.
  • Define data requirements gather and mine data while validating the efficiency of data tools in the Big Data Environment.
  • Lead the evaluation implementation and deployment of emerging tools and processes to improve productivity.
  • Implement processes and systems to provide accurate and available data to key stakeholders downstream systems and business processes.
  • Partner with Business Analysts and Solution Architects to develop technical architectures for strategic enterprise projects and initiatives.
  • Coordinate with Data Scientists to understand data requirements and design solutions that enable advanced analytics machine learning and predictive modelling.
  • Mentor and coach junior Data Engineers on data standards and practices promoting the values of learning and growth.
  • Foster a culture of sharing re-use design for scale stability and operational efficiency of data and analytical solutions.

  To qualify for the role you must have the following:

  Essential skillsets

  • Bachelors degree in engineering Computer Science Data Science or related field
  • 6 years of experience in software development data engineering ETL and analytics reporting development
  • Experience in building and maintaining data and system integrations using dimensional data modelling and optimized ETL pipelines.
  • Proven track record of designing and implementing complex data solutions
  • Demonstrated understanding and experience using:
    • Python & PySpark: Ability to write efficient scripts for data transformation and pipeline orchestration knowledge of Spark or any distributing frameworks
    • Knowledge of data modelling and data architecture principles
    • Familiarity with event-driven architectures API -based data sources data quality validation archival strategies and incremental loading.
    • Distributed Data Framework (e.g. Spark). Good understanding of Spark architecture
    • Cloud platform services (AWS / Azure). AWS services (AWS Glue Lambda S3 Stepfunctions)
    • Proficiency in SQL and relational database management systems
    • Optimize and tune database performance including indexing query optimization and storage management.
    • CICD - Terraform or CloudFormation Jenkins Bitbucket : For infrastructure-as-code and deployment automation.
    • Experience with database design concepts: schema definition normalization relationships indexing constraints.
    • Knowledge of Data lakes Data warehouses
    • Databricks/Delta Lakehouse architecture
    • Code management platforms like Github/ Gitlab/ etc.
  • Understanding of database architecture Data modelling concepts and administration.
  • Handson experience of Spark Structured Streaming for building real-time ETL pipelines.
  • Utilizes the principles of continuous integration and delivery to automate the deployment of code changes to elevate environments fostering enhanced code quality test coverage and automation of resilient test cases.
  • Proficient in programming languages (e.g. SQL Python Pyspark) to design develop maintain and optimize data architecture/pipelines that fit business goals.
  • Strong organizational skills with the ability to work multiple projects simultaneously and operate as a leading member across globally distributed teams to deliver high-quality services and solutions.
  • Excellent written and verbal communication skills including storytelling and interacting effectively with multifunctional teams and other strategic partners
  • Strong problem solving and troubleshooting skills
  • Ability to work in a fast-paced environment and adapt to changing business priorities
  
  

  Desired skillsets

  • Masters degree in engineering specialized in Computer Science Data Science or related field
  • Experience working with pharma clients preferably in commercial pharma marketing and medical affairs
  • Demonstrated understanding and experience using:
    • Job orchestration tools like Tidal/Airflow/ or similar
    • Knowledge on No SQL
  • Databricks Certified Data Engineer Associate
  • AWS Certified Data Engineer - Associate

EY Building a better working world

EY exists to build a better working world helping to create long-term value for clients people and society and build trust in the capital markets.

Enabled by data and technology diverse EY teams in over 150 countries provide trust through assurance and help clients grow transform and operate.

Working across assurance consulting law strategy tax and transactions EY teams ask better questions to find new answers for the complex issues facing our world today.