Summary
The Agentic AI Developer designs builds and operates agentic AI process automations and reusable
LLMpowered components on our Azure-first stack. Partnering with Business Technology Partners (BTPs)
and teams across the company you will convert business requirements and process maps into
production-grade automations that are observable governed and reusable. A core responsibility is
rigorous quality control: you will develop evaluation workflows (test prompts regression suites)
implement hallucination detection and mitigation measure and reduce bias enforce guardrails and
schema validation and ensure outputs are grounded reliable compliant and auditableconsistent
with SEC governance and U.S. legal requirements for registered financial advisors.
Who you are
Youre a hands-on builder fluent in modern AI tooling who can translate business intent into robust
agentic workflows. You communicate clearly with technical and non-technical stakeholders execute
with precision and bring a strong quality mindset to everything you release. You balance speed with
governance instrument your solutions for observability and cost control and document decisions for
traceability. You are adaptable eager to learn new technologies and able to manage multiple projects
under time constraints.
What youll do
Design agentic AI systems that automate multi-step workflows; orchestrate LLM tools/skills and
APIs to deliver reliable outcomes.
Build reusable components (prompt libraries tools/functions skills RAG pipelines evaluation
harnesses) for enterprise reuse across teams.
Integrate LLMs (e.g. ChatGPT Claude) with Azure services (Azure OpenAI Azure AI
Search/Vector Functions Logic Apps) and Microsoft 365/Copilot Studio.
Use n8n (and/or Power Automate) for orchestration eventing and integrations across line-of-
business systems.
Implement RAG (retrievalaugmented generation) with vector search and grounding/citation
policies; enforce structured outputs (JSON schemas).
Build evaluation & QC: red-team test prompts regression suites groundedness checks bias
assessments output validation cost/latency monitoring.
Instrument observability (logging trace metrics) set SLOs (e.g. firstpass accuracy
hallucination rate bias metrics latency costpertask) and implement guardrails.
Follow intake and prioritization through BTPs/supervisor; collaborate directly with business
stakeholders on requirements and acceptance tests.
Ensure security & compliance (PII handling RBAC data minimization audit trails) aligned to SEC
governance and company policies (e.g. prompt/response archiving).
Create documentation and runbooks; contribute to code reviews and best practices; support
endusers and BTPs with adoption and training.
Troubleshoot and resolve issues related to AI automations and integrations.
Stay current on agentic AI frameworks and Azure/M365 capabilities; propose improvements to
platform tooling and standards.
Other duties responsibilities and special projects as assigned.
What youll bring
Experience
5 years professional programming experience; 2 years building AI tool based solutions (LLM
APIs Copilot ChatGPT/Claude orchestration).
Proven delivery of production automations on Azure and Microsoft 365 ecosystems.
At least 2 years of experience developing solutions with AI tools (e.g. Copilot ChatGPT Claude
n8n Foundry).
Technical skills (core)
Languages: Python C# R (plus familiarity with JavaScript/TypeScript helpful).
Frameworks/tools: LangChain LangGraph LlamaIndex AutoGen (or similar agent frameworks);
Copilot Studio GitHub n8n.
Azure: Azure OpenAI Azure AI Search/Vector Functions Logic Apps Key Vault Storage
DevOps/GitHub Actions.
RAG pipelines vector databases (e.g. Azure AI Search; familiarity with Pinecone/Weaviate a
plus).
API design & integration structured outputs (JSON schema) function/tool calling.
Observability & cost control: logging tracing metrics rate/usage monitoring.
Quality control & evaluation (explicit)
Ability to design and run test prompts and regression suites covering normal edge and
adversarial scenarios.
Skill in hallucination detection/mitigation (groundedness checks citation requirements retrieval
confidence thresholds fallback policies).
Experience measuring and reducing bias (fairness metrics scenario/corpus checks cohort A/B
testing safeprompting).
Output validation using schema enforcement tool/function results verification and
businessrule assertions.
Building evaluation harnesses with clear SLOs/KPIs (firstpass accuracy hallucination rate bias
indicators latency costpertask) and reporting dashboards.
Implementing guardrails (content filters allow/deny lists sensitive-topic controls) and redteam
testing before deployment.
Documenting test plans evaluation results and traceability from requirements to acceptance
criteria.
Collaboration & compliance
Strong communication with BTPs/endusers; aptitude for translating requirements into technical
designs and acceptance tests.
Working knowledge of SEC governance and U.S. regulatory expectations for AI usage in financial
advisory contexts; commitment to Responsible AI practices.
Familiarity with enterprise archiving/monitoring of prompts and outputs per policy (e.g.
Smarsh Theta Lake) and changecontrol practices.
Nice to have
Experience with realestate AI platforms (e.g. V7 Kolena Cadastral) or domain workflows.
Knowledge of GenAIOps/MLOps concepts for lifecycle and evaluation (not model training).
Containers (Docker) and CI/CD; Power Platform experience.
Additional Information for Remote Role
Work Hours: Must have at least 4 hours overlap with Mountain Time zone (Denver CO).
Communication: Strong internet connection and reliable communication tools are essential.
Remote Work Environment: Must have a dedicated workspace free from distractions to ensure
productivity.
Cultural Sensitivity: Ability to work effectively with a diverse team and understand cultural
differences.
5 years professional programming experience; 2 years building AI tool based solutions (LLM APIs Copilot ChatGPT/Claude orchestration). Proven delivery of production automations on Azure and Microsoft 365 ecosystems. At least 2 years of experience developing solutions with AI tools (e.g. Copilot ChatGPT Claude n8n Foundry).