Senior Data Engineer

The Senior Data Engineer must be able to meet the key criteria below:

1. Location: 100 telework
2. Years Experience: 10 years
3. Education: Bachelors in IT related field
4. Security Clearance: IBR is a federal contractor. Applicants must be able to obtain and maintain a Public Trust security clearance.
   • NOTE: United States Citizenship work authorization is required as part of the eligibility criteria to be able to obtain this type of security clearance.
5. Employment Type: Exempt
6. Key Skills:
   • 10 years of IT experience focusing on enterprise data architecture and management
   • Experience with Databricks Structured Streaming Delta Lake concepts and Delta Live Tables required
   • Experience with ETL and ELT tools such as SSIS Pentaho and/or Data Migration Services
   • Advanced level SQL experience (Joins Aggregation Windowing functions Common Table Expressions RDBMS schema design Postgres performance optimization)

Overview

Do you want to help build a portfolio of nextgeneration mobileenabled data collection systems and enterprise portals As a Data Engineer at IBR you will support the Agile based engineering of robust secure and scalable enterprise web portal solutions hosted in AWS. This position will work closely with the solutions delivery team to support the operations team performing Deployment Systems Integration Testing and Operations & Maintenance activities.

Responsibilities

• Plan create and maintain data architectures ensuring alignment with business requirements
• Obtain data formulate dataset processes and store optimized data
• Identify problems and inefficiencies and apply solutions
• Determine tasks where manual participation can be eliminated with automation.
• Identify and optimize data bottlenecks leveraging automation where possible
• Create and manage data lifecycle policies (retention backups/restore etc)
• Indepth knowledge for creating maintaining and managing ETL/ELT pipelines
• Create maintain and manage data transformations
• Maintain/update documentation
• Create maintain and manage data pipeline schedules
• Monitor data pipelines
• Create maintain and manage data quality gates (Great Expectations) to ensure high data quality
• Support AI/ML teams with optimizing feature engineering code
• Expertise in Spark/Python/Databricks Data Lake and SQL
• Create maintain and manage Spark Structured Steaming jobs including using the newer Delta Live Tables and/or DBT
• Research existing data in the data lake to determine best sources for data
• Create manage and maintain ksqlDB and Kafka Streams queries/code
• Data driven testing for data quality
• Maintain and update Pythonbased data processing scripts executed on AWS Lambdas
• Unit tests for all the Spark Python data processing and Lambda codes
• Maintain PCIS Reporting Database data lake with optimizations and maintenance (performance tuning etc)
• Streamlining data processing experience including formalizing concepts of how to handle lake data defining windows and how window definitions impact data freshness.

Qualifications

• 10 years of IT experience focusing on enterprise data architecture and management
• Must be able to obtain a Public Trust security clearance
• Bachelor degree required
• Experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling
• Experience with Databricks Structured Streaming Delta Lake concepts and Delta Live Tables required
  • Additional experience with Spark Spark SQL Spark DataFrames and DataSets and PySpark
  • Data Lake concepts such as time travel and schema evolution and optimization
  • Structured Streaming and Delta Live Tables with Databricks a bonus
• Experience leading and architecting enterprisewide initiatives specifically system integration data migration transformation data warehouse build data mart build and data lakes implementation / support
  • Advanced level understanding of streaming data pipelines and how they differ from batch systems
  • Formalize concepts of how to handle late data defining windows and data freshness
  • Advanced understanding of ETL and ELT and ETL/ELT tools such as SSIS Pentaho Data Migration Service etc
  • Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window sliding window high watermark etc.
  • Familiarity and/or expertise with Great Expectations or other data quality/data validation frameworks a bonus
  • Understanding of streaming data pipelines and batch systems
  • Familiarity with concepts such as late data defining windows and how window definitions impact data freshness
• Advanced level SQL experience (Joins Aggregation Windowing functions Common Table Expressions RDBMS schema design Postgres performance optimization)
  • Indexing and partitioning strategy experience
• Debug troubleshoot design and implement solutions to complex technical issues
• Experience with largescale highperformance enterprise big data application deployment and solution
• Understanding how to create DAGs to define workflows
• Familiarity with CI/CD pipelines containerization and pipeline orchestration tools such as Airflow Prefect etc a bonus but not required
• Architecture experience in AWS environment a bonus
  • Familiarity working with Kinesis and/or Lambda specifically with how to push and pull data how to use AWS tools to view data in Kinesis streams and for processing massive data at scale a bonus
  • Experience with Docker Jenkins and CloudWatch
  • Ability to write and maintain Jenkinsfiles for supporting CI/CD pipelines
  • Experience working with AWS Lambdas for configuration and optimization
  • Experience working with DynamoDB to query and write data
  • Experience with S3
• Knowledge of Python (Python 3 desired) for CI/CD pipelines a bonus
  • Familiarity with Pytest and Unittest a bonus
• Experience working with JSON and defining JSON Schemas a bonus
• Experience setting up and management Confluent/Kafka topics and ensuring performance using Kafka a bonus
  • Familiarity with Schema Registry message formats such as Avro ORC etc.
  • Understanding how to manage ksqlDB SQL files and migrations and Kafka Streams
• Ability to thrive in a teambased environment
• Experience briefing the benefits and constraints of technology solutions to technology partners stakeholders team members and senior level of management

Physical Demands and Work Environment

While this position primarily involves sedentary work in an office environment and/or homeoffice environment there are some physical demands associated with the role including:

• Extended periods of sitting and working at a computer workstation.
• Occasional lifting and carrying of equipment or materials weighing up to 20 pounds.
• Occasional travel to client sites or meetings may be required.
• Work may need to be performed in a fastpaced environment requiring quick thinking and rapid judgements.
• Additionally flexible work arrangements may be available based on business needs.
• Employee will be exposed to a wide variety of clients in differing functions personalities and abilities.

About IBR
Imagine Believe Realize LLC (IBR) is an emerging small business focused on delivering software and systems engineering solutions to government and commercial clients. Our talent acquisition strategy is tailored to career seeking candidates who embrace continuous learning and desire to grow as a professional in the software/systems engineering industry. We strive to enhance our team members ability to thrive in the workplace by creating a proper work/life balance and firstclass benefits package that includes:

• Nationwide medical dental and vision insurance
• 3 weeks of Paid Time Off and 11 Paid Federal Holidays
• 401k matching
• Life Insurance ShortTerm Disability and LongTerm Disability at no cost to our employees
• Flexible spending accounts and Dependent Care spending accounts
• Wellness incentives
• Reimbursement for professional development and certifications
• Training assistance opportunities

Upon hire and in compliance with federal law all persons hired are required to verify identity and eligibility to work in the United States and to complete the required employment eligibility verification and background check. IBR is a Federal Contractor.

Imagine Believe Realize LLC is proud to be an Equal Opportunity and Affirmative Action Employer. We do not discriminate based upon race age religion color national origin gender (including pregnancy childbirth or related medical conditions) sexual orientation gender identity gender expression status as a protected veteran status as an individual with a disability or other applicable legally protected characteristics.
Learn more at

If alternative methods of assistance are needed with the application process additional contact information has been provided below:


407.459.1830