AI Data Analyst

OVERVIEW

The Research Data Science team is responsible for pushing the boundaries of how Spring Venture Group utilizes unstructured data. While traditional analytics focuses on what happened our team focuses on why it happened using Large Language Models (LLMs) to extract structured insights from conversation transcripts. You will play a pivotal role in adapting the organization to the rapidly evolving future of AI helping us stay ahead of the curve in a landscape that changes daily. 

As a Research Data Analyst you will act as the bridge between human intuition and machine precision. You will not simply be writing prompts; you will be engineering structured data extraction pipelines and performing rigorous statistical analysis on the results. You will translate abstract business conceptslike customer sentiment or negotiation frictioninto rigorous testable prompt definitions and then analyze how those new metrics correlate with downstream business outcomes like revenue and retention. 

This is a role for a scientist at heart. You will own the lifecycle of conversational metrics: from Discovery (finding the signal) to Definition (codifying the prompt) to Validation (proving it works via statistics) and finally to Experimentation (using the metric to judge AB tests). You should be passionate about finding the truth in data and have the technical discipline to verify that truth.

3 months into the job you will: 

• Understand the Shared Ideas of the business: Learn the nuances of our sales conversations and how different departments define success.  

• Master our existing Prompt Library: specific syntax and schema enforcement

• Audit the Machine: Execute stratified sampling strategies to manually validate model outputs calculating precision and recall to ensure our metrics are trustworthy.

• Analyze the Impact: Visualize LLM-derived data to explain trends in conversation quality and identify the statistical drivers of performance to stakeholders.  

6 months into the job you will: 

• Lead Metric Discovery: Use LLMs to cluster topics and propose new schemas for unstructured data features.  

• Drive Experimentation: Design and measure AB experiments. You will be responsible for verifying that the metrics used to judge winners are semantically aligned with the business goals.

• Strategic Analysis: Suggest new variants for treatments based on deep-dive analysis of conversation logs connecting qualitative insights to quantitative financial results.

• Guard the Truth: Act as the gatekeeper for Prompt Definitions ensuring that we do not deploy metrics that rely on hallucination or weak signals. 

REPORTS TO

The Research Data Analyst reports to the Director of Research Data Science.  

ESSENTIAL DUTIES

The essential duties for this role include but are not limited to: 

• Designing or writing prompts to support specific AI outcomes
• Analyzing AI performance and making process and algorithm updates for performance improvement
• Continued research and education on the ever-changing AI field to ensure updated best practices
• Go beyond assumptions to scientifically validate prompt performance
• Understand assumptions and their pitfalls; challenge them when appropriate 
• Apply sound logic and mathematics to database queries and reports 
• Create simple and elegant solutions to seemingly complex problems  
• Maintain and update Streamlit/SQL reports as necessary 
• Perform other duties and responsibilities as assigned

Qualifications :

Education/Experience:

• Bachelors degree in a technical field or 2 years of relevant work experience

Skills/Specialties:

• Understanding of how Large Language Models work (context windows temperature hallucination risks) and AI.
• Fluency in at least one program language preferably Python.
• Understanding of A/B testing frameworks statistical significance and metric definition.
• Strong knowledge of SQL and Relational Database systems.
• Experience with Snowflake
• A strong understanding of programming and a willingness to learn even more.  

Additional Information :

Benefits:

• Competitive compensation Medical Dental and Vision benefits after a short waiting period
• 401k matching program
• Generous paid time off program with an additional company break during the holidays
• Annual Volunteer Time Off (VTO) and a donation matching program
• Life Insurance Employee Assistance Program (health & well-being on and off the job) Rewards and Recognition
• Maternity and Parental Leave
• Training program and ongoing support throughout your entire Spring Venture Group career
• Diverse inclusive & welcoming culture
• Optional enrollment options include HSA/FSA AD&D Spousal/Dependent life insurance  Short Term/Long Term Disability Travel Assist and Legal plans

Security Responsibilities:

• Operating in alignment with policies and standards
• Reporting Security Incidents Completing assigned training
• Protecting assigned organizational assets

All your information will be kept confidential according to EEO guidelines.

Spring Venture Group is an Equal Opportunity Employer

Spring Venture Group is an Equal Opportunity Employer

Remote Work :

No

Employment Type :

Full-time