AWS Data Bricks Engineer

Skills

Total of 6 years of experience. 23 yrs. experience designing developing deploying and/or supporting data pipelines using Databricks

Expertise in designing and deploying data applications on cloud solutions AWS or Teradata SaaS

Hands on experience in performance tuning and optimizing code running in Databricks environment

Proficient in programming languages like Pyspark and Python

Good understanding of SQL TSQL and/or PL/SQL

Demonstrated analytical and problemsolving skills particularly those that apply to a big data environment

Knowledge of Teradata is an advantage

Responsibilities

Designing and implementing data ingestion pipelines from multiple sources

Developing scalable and reusable frameworks for ingesting of data sets

Integrating the end to end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times

Working with event based / streaming technologies to ingest and process data

Working with other members of the project team to support delivery of additional project components (API interfaces Search)

Evaluating the performance and applicability of multiple tools against customer requirements