Sr. Data Engineer with Insurance Domain(W2)
Sr. Data Engineer with Insurance Domain(W2)
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Job : Sr. Data Engineer with Insurance Domain(W2)
Location : Virginia Reston
Skills : Python Jupyter Notebooks Spark PySpark Spark SQL Delta Lake
Sr. Data Engineer with Insurance Domain exp with 15 years
Remote
GC/USC
Looking for VERY senior resources up to hands-on lead level
Experienced with Assertion based Architecture
Engineers vs coders
Coding is done in Jupyter Notebooks on Delta Lakes
Need resources who can articulate design and build highly scalable solutions before jumping into coding
Do NOT want resources who need to be told what to do
Need critical thinkers who can troubleshoot and debug
Independent workers self starters who speak up and raise impediments and offer solutions
Ability to create technical documentation/designs/blueprints for repeatable development patterns
Required skills:
Python
Jupyter Notebooks
Delta Lake
Spark PySpark Spark SQL
Serverless data infrastructure
Data Vault 2.0 methodology experience
Great Expectations data quality validation
Automated Testing
Bonus skills:
Kakfa streaming HUGE plus if they have solid background here
Insurance background
Experience leading mid-to sr. level engineers
Apache Airflow
Agentic AI implementations
Key Responsibilities:
Design develop and maintain data pipelines using Python PySpark and Spark SQL to process and transform large-scale datasets.
Implement Delta Lake architecture to ensure data reliability consistency and integrity for large distributed datasets.
Utilize serverless data infrastructure (e.g. AWS Lambda Azure Functions Databricks) to build scalable and cost-efficient data solutions.
Collaborate with Data Scientists and Analysts by creating reusable Jupyter Notebooks for data exploration analysis and visualization.
Optimize and manage data storage and retrieval processes ensuring high performance and low latency.
Implement best practices for data security governance and compliance within the data infrastructure.
Work closely with cross-functional teams to understand data requirements and deliver solutions aligned with business objectives.
Continuously monitor troubleshoot and improve the performance of data processing pipelines and infrastructure.
Qualifications:
15 years of experience in data engineering or related fields.
Strong programming skills in Python with experience in data processing frameworks like PySpark.
Extensive hands-on experience with Apache Spark and Spark SQL for processing and querying large datasets.
Expertise with Delta Lakes for implementing scalable data lakehouse architectures.
Experience with Jupyter Notebooks for prototyping and collaboration with data teams.
Familiarity with serverless data technologies such as AWS Lambda Azure Functions or similar platforms.
Proficient in working with cloud platforms such as AWS Azure or Google Cloud.
Experience with data pipeline orchestration tools (e.g. Apache Airflow Prefect or similar).
Solid understanding of data warehousing ETL/ELT pipelines and modern data architectures.
Strong problem-solving skills and ability to work in a collaborative environment.
Experience with CI/CD pipelines and DevOps practices is a plus.
Preferred Qualifications:
Experience with Databricks for data engineering workflows.
Familiarity with modern data governance practices and tools like Apache Atlas or AWS Glue.
Knowledge of machine learning workflows and how data engineering supports AI/ML models
Employment Type
Full-time
