As a Data Engineer you will be working in one of the worlds largest and most complex data warehouse environments. You will design implement and support scalable data infrastructure solutions to integrate with multi heterogeneous data sources aggregate and retrieve data in a fast and safe mode curate data that can be used in reporting analysis machine learning models and ad-hoc data requests. You will be managing multiple Redshift clusters that support reporting needs for the transportation org. You will be exposed to cutting edge AWS big data technologies. You should have excellent business and communication skills to be able to work with business owners and Tech leaders to gather infrastructure requirements design data infrastructure build up data pipelines and data-sets to meet business needs. You stay abreast of emerging technologies investigating and implementing where appropriate.
Key job responsibilities - Design and develop the pipelines required for optimal extraction transformation and loading of data from a wide variety of data sources using SQL Python and AWS big data technologies. - Oversee and continually improve production operations including optimizing data delivery re-designing infrastructure for greater scalability code deployments bug fixes and overall release management and coordination. - Establish and maintain best practices for the design development and support of data integration solutions including documentation. - Work closely with Product teams Data Scientists Software developers and Business Intelligence Engineer to explore new data sources and deliver the data. - Able to read write and debug data processing and orchestration code written Python/Scala etc following best coding standards (e.g. version controlled code reviewed etc.)
- Experience as a Data Engineer or in a similar role - Knowledge of distributed systems as it pertains to data storage and computing - Bachelors degree
- Experience with AWS technologies like Redshift S3 AWS Glue EMR Kinesis FireHose Lambda and IAM roles and permissions - Experience with non-relational databases / data stores (object storage document or key-value stores graph databases column-family databases)
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status disability or other legally protected status.
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.