Data Scientist, EngThrive Insights CoreAI

Imagine making AI coding tools more effective for tens of thousands of Microsoft engineersthats what youll do on the Engineering Thrive Insights team. Youll join the team behind the data platform metrics and insights that measure and improve engineering at Microsoft spanning developer satisfaction change velocity build quality and AI tool impact. Your primary focus will be the AI Data Flywheel our platform for collecting signals from engineers using AI coding tools and transforming them into actionable insights and experiments that drive continuous product improvement.

In this role youll define success metrics design and analyze experiments develop statistical and machine-learning models and uncover opportunities to improve AI-assisted software development. Youll build rigorous measurement frameworks evaluate product changes and translate complex behavioral and product data into recommendations for engineering and product teams. Youll deepen your expertise in experimentation causal inference developer productivity frameworks such as DORA and SPACE AI product evaluation and privacy-conscious analyticswhile using the latest AI tools to help shape the next generation of developer experiences at Microsoft scale.

• Leverages understanding of data science and business to examine a project and consider factors that can influence final outcomes within a technical area. Evaluates project plan for resources risks contingencies requirements assumptions and constraints. Documents key business objectives. Effectively communicates business goals in analytical and technical terms. Consistently shares insights with stakeholders.

• Collaborates with immediate team members on defining metrics and frameworks for evaluating somewhat complex AI/large language model (LLM) systems and agentic product performance. Experiments with model before selection (e.g. LLM prompting approach) for tasks within a product or solution Diagnose basic issues and identifies opportunities to drive improvements within AI/LLM systems within domain area. Prototypes and develops AI models using containerized environments (e.g. AI Azure Foundry LangChain) to operationalize build and validate scalable tools/applications/solutions for use by the immediate team. Develops and deploys systems for basic retrieval-augmented generation workflows assisting in the selection of data sources retrieval systems and generative models to ensure quality output. Assists senior members in assessing and refining LLM systems and models.

• Understands where to acquire data necessary for successful completion of the project plan. Utilizes querying visualization and reporting techniques to describe acquired data including format quantity identities and other surface properties. Explores data for key attributes and contributes to the development of data quality report describing results of the task initial findings and impact on the project by leveraging appropriate AI tools (e.g. Codex GitHub Copilot). Collaborates with others to perform data-science experiments using established methodologies statistics optimization and probability theory for general purpose software and statistical packages. Assesses different tools and techniques and selects the appropriate one. Serves as an effective partner in data preparation efforts to Solution Architects Consultants and Data Engineers. Adheres to Microsofts privacy policy related to collecting data preparing data and using AI tools (e.g. Codex GitHub Copilot). Identifies data integrity problems.

• Leverages knowledge of modeling and machine learning solutions (e.g. state-of-the-art statistical analysis causal modeling reinforcement learning) to identify the best approach to complete objectives. Understands modeling techniques (e.g. validation hypothesis testing optimal use of data) and machine learning in order to select the correct approach to prepare data train and optimize the model and evaluate the output for statistical and business significance. Understands the risks of data leakage the bias/variance tradeoff methodological limitations etc. Writes all necessary scripts in the appropriate language: T-SQL U-SQL KQL Python R etc. Constructs hypotheses designs controlled experiments analyzes results using statistical tests and communicates findings to business stakeholders. Effectively communicates with diverse audiences on data-quality issues and initiatives. Understands operational considerations of model deployment such as performance scalability monitoring maintenance integration into engineering production system stability. Develops operational models that run at scale through partnership with data engineering teams. Understands and keeps up to date on embedding models and vector databases.

• Learns and understands the current state of the industry including knowledge of tools techniques strategies and processes that can be utilized to improve process efficiency and performance. Maintains knowledge of current trends within the discipline. Attends internal research conferences and participates in on-hands training when appropriate. Actively contributes to the body of thought leadership and intellectual property (IP) best practices.

Required Qualifications:

• Doctorate in Data Science Mathematics Statistics Econometrics Economics Operations Research Computer Science or related field

• OR Masters Degree in Data Science Mathematics Statistics Econometrics Economics Operations Research Computer Science or related field AND 1 year(s) data-science experience (e.g. managing structured and unstructured data applying statistical techniques and reporting results) or consulting experience

• OR Bachelors Degree in Data Science Mathematics Statistics Econometrics Economics Operations Research Computer Science or related field AND 2 years data-science experience (e.g. managing structured and unstructured data applying statistical techniques and reporting results)

• OR equivalent experience.

• Ability to meet Microsoft customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings:
• Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Preferred Qualifications:

• Doctorate in Data Science Mathematics Statistics Econometrics Economics Operations Research Computer Science or related field AND 1 year(s) data-science experience (e.g. managing structured and unstructured data applying statistical techniques and reporting results)OR Masters Degree in Data Science Mathematics Statistics Econometrics Economics Operations Research Computer Science or related field AND 3 years data-science experience (e.g. managing structured and unstructured data applying statistical techniques and reporting results)OR Bachelors Degree in Data Science Mathematics Statistics Econometrics Economics Operations Research Computer Science or related field AND 5 years data-science experience (e.g. managing structured and unstructured data applying statistical techniques and reporting results)OR equivalent experience.

• Experience with observational causal inference methods design of controlled experiments regression modeling and hypothesis testing.

• Experience evaluating LLMs and/or AI agents designing offline/online evaluations defining quality metrics and measuring model behavior.

• Experience designing surveys and analyzing response data

• Proficiency in Python programming

• Proficiency in analytical query languages like SQL KQL (Kusto Query Language) or similar analytical query languages.

• Experience developing analytics on Spark-based big data platform technologies such as Databricks.

• Data storytelling crafting and presenting compelling data stories to business decisionmakers

• Familiarity with developer productivity metrics and frameworks (e.g. DORA SPACE or similar).

• Demonstrated effective use of AI coding assistants in daily data science development workflow