Data Engineer AWSDatabricks
Share
Job Location:
Reston, VA - USA
Monthly Salary:
Not Disclosed
Posted on:
30+ days ago
Vacancies:
1 Vacancy
Job Summary
Overview
Acuity Inc. is seeking a highly skilled Data Engineer to join our Engineering Team helping drive the design and delivery of AWS cloud-scale data platforms for federal clients. This role requires knowledge and/or experience with Spark Delta Lake and distributed data pipelines on Databricks. The ideal candidate brings both engineering and strategic insight into enterprise data modernization.
Responsibilities
- Build and maintain scalable PySpark-based data pipelines in Databricks notebooks to support ingestion transformation and enrichment of structured and semi-structured data.
Design and implement Delta Lake tables optimized for ACID compliance partition pruning schema enforcement and query performance across large datasets.
Develop ETL and ELT workflows that integrate multiple source systems into a centralized query-optimized data warehouse architecture.
Leverage Spark SQL and DataFrame APIs to implement business rules dimensional joins and aggregation logic aligned to warehouse modeling best practices.
Collaborate with data architects and engineers to implement cloud-native data solutions on AWS using S3 Glue RDS and IAM for secure scalable storage and access control.
Optimize pipeline performance through intelligent partitioning caching broadcast joins and adaptive query tuning.
Deploy and version data engineering assets using Git-integrated development workflows and automate deployment with CI/CD tools such as GitLab or Jenkins.
Monitor pipeline health job execution and cluster utilization using native Databricks tools and AWS CloudWatch identifying bottlenecks and optimizing cost-performance tradeoffs.
Conduct technical discovery and mapping of legacy source systems identifying required transformations and designing end-to-end data flows.
Implement governance practices including metadata tagging data quality validation audit logging and lineage tracking using platform-native features and custom logic.
Support ad hoc data access requests develop reusable data assets and maintain shared notebooks that meet operational reporting and analytics needs across teams.
Qualifications
- 2 years of experience in data engineering and Agile analytics
- 2 years of experience creating software for retrieving parsing and processing structured and unstructured data
- 1 to 2 years of experience building scalable ETL and ELT workflows for reporting and analytics
- 1 or more years experience building enterprise data engineering solutions in the cloud with preferred experience with cloud native technologies from AWS and Databricks
- Experience with data quality validation frameworks and storage optimization strategies
- BA or BS degree
Clearance Requirement:
- Must be US Citizen with an ability to obtain and maintain US Suitability
About Acuity
At Acuity your work mattersand so does your experience. Were a management and technology consulting firm supporting critical federal missions where youll have the opportunity to solve meaningful challenges work alongside high-performing teams and make a real impact from day one.
Why Youll Love Working Here:
Grow Your Career Your Way
We invest in you with personalized development plans mentorship and up to $6000 annually for training and certificationsso you can keep building the career you want.
Be Part of Something Innovative
Youll work on cutting-edge solutions that support important government missions in an environment that encourages new ideas and continuous improvement.
Thrive in a People-First Culture
Collaboration respect and support arent just valuestheyre how we operate. Your voice is heard your contributions are recognized and your success is shared.
Feel Valued and Rewarded
We offer competitive compensation comprehensive benefits and a strong focus on work-life balance so you can perform at your bestat work and at home.
Join an Award-Winning Team
Our employees consistently rank us among the bestearning honors like Best Places to Work (Washington Business Journal 9 years) and Top Workplaces (The Washington Post ).
Bring Your Whole Self to Work
Were committed to building a diverse inclusive environment where everyone feels respected supported and empowered to succeed.
Make Your Impact
Join Acuity and be part of a team where your ideas are valued your growth is supported and your work drives meaningful outcomes.
Acuity is an Equal Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.
Recruiting Scams and Fraud
Acuity Inc. has been made aware of fraudulent job postings and individuals impersonating company recruiters. These scams may include fake job offers requests for sensitive personal information or demands for payment.
Please note:
- Acuity never asks candidates to pay for job applications equipment or training.
- All official communications will come from an @ email address.
- If you are unsure about a job posting or recruiter please verify the opportunity on our Careers page.
If you believe you have been targeted by a scam or have concerns about the authenticity of a job listing claiming to represent Acuity Inc. please contact us at . Protecting your security and trust is important to us.
#LI-PA1
#LI-Remote
Required Experience:
IC
Key Skills
About Company
Explore home, auto, and business insurance options from a top-rated company. Find personalized service for you, your family, and your business. Get a quote or find an agent today.
