AI Engineer – LLM Evaluation & Prompt Engineering

Who is Blueprint

We are a technology solutions firm headquartered in Bellevue Washington with a strong presence across the United States. Unified by a shared passion for solving complicated problems our people are our greatest asset. We use technology as a tool to bridge the gap between strategy and execution powered by the knowledge skills and theexpertiseof our teams who all have unique perspectives and years of experience across multiple smart agile and fun.

What does Blueprint do

Blueprint helps organizations unlock value from existing assets byleveragingcutting-edgetechnology to createadditionalrevenue streams and new lines of business. We connect strategy business solutions products and services to transform and grow companies.

Why Blueprint

At Blueprint we believe in the power of possibility and are passionate about bringing it to you join our bustling product division our multifaceted services team or you want to grow your career in human resources your ability to make an impact is amplified when you join one of our on solving unique business problems while gaining hands-on experience with the worlds best technology. We believe in unique perspectives and build teams of people with diverse skillsets and Blueprint youll have the opportunity to work with multiple clients and teams such as data science and product development all while learning growing and developing new guarantee youwontfind a better place to work and thrive than at Blueprint.

In This Role

In this role you will contribute to building and operating AI-powered middle-tier services that support conversational experiences within widely used productivity applications. You will focus on prompt evaluation testing and automation ensuring that AI responses are accurate reliable and aligned with business and user expectations.

You will work closely with engineering product and data partners to evaluate LLM behavior design test strategies implement supporting code and continuously improve prompt quality and system performance.

Key Responsibilities

• Design evaluate and refine conversational prompts used in AI-driven applications
• Perform manual and automated testing of LLM outputs to validate accuracy relevance and consistency
• Develop and maintain prompt evaluation frameworks and supporting tooling
• Set up and manage test environments for AI and prompt validation workflows
• Write and maintain code (Python or C#) to support evaluations automation and analysis
• Analyze evaluation results and provide data-driven recommendations for prompt improvements
• Review enhancement requests and translate requirements into technical solutions
• Prepare detailed software specifications test plans and test data
• Modify and enhance existing systems to meet new standards or requirements
• Conduct unit testing quality assurance reviews and post-implementation validation
• Support deployment migration and implementation activities
• Troubleshoot issues in both new and legacy systems and resolve defects identified during testing

Required Qualifications

• Bachelors degree in Computer Science Computer Engineering or a related technical field
• 24 years of professional software engineering or AI-related experience
• Strong foundation in computer science fundamentals including data structures algorithms and software design
• Experience developing or supporting large-scale software systems
• Hands-on programming experience in Python or C#
• Experience with unit testing debugging and troubleshooting in production or pre-production systems
• Ability to analyze requirements and translate them into effective technical solutions
• Strong problem-solving skills and attention to detail

Preferred Qualifications

• Prior experience with LLM evaluation prompt engineering or AI experimentation
• Background in data science experimentation or model evaluation
• Experience building automated test frameworks or evaluation pipelines
• Familiarity with conversational AI chatbots or virtual assistant systems
• Experience working with AI/ML-powered applications in production environments
• Strong analytical mindset with the ability to interpret evaluation results and metrics

Salary Range

Pay ranges vary based on multiple factors including without limitation skill sets education responsibilities experience and geographical market. The pay range for this position reflects geographic based ranges for Washington state: $120000- $135000 USD annually. The salary/wage and job title for this opening will be based on the selected candidates qualifications and experience and may be outside this range.

Equal Opportunity Employer

Blueprint Technologies LLC is an equal employment opportunity employer. Qualified applicants are considered without regard to race color age disability sex gender identity or expression orientation veteran/military status religion national origin ancestry marital or familial status genetic information citizenship or any other status protected by law.

If you needassistanceor a reasonable accommodation to complete the application process please reach out to:

Blueprint believe in the importance of a healthy and happy team which is why our comprehensive benefits package includes:

• Medical dental and vision coverage
• Flexible Spending Account
• 401k program
• Competitive PTO offerings
• Parental Leave
• Opportunities for professional growth and development

Location: Redmond WA