1. FLASH

• This is a write-up of engineering work I did at Microsoft Research India on FLASH, the M365 Research workflow-automation agent for diagnosing recurring incidents. It is not a paper — there is no publication associated with it, and nothing here should be read as a peer-reviewed result.
• My contribution: AutoToolStub — a parallelised test-case generator using 100-way concurrent BFS with provenance tracking, which auto-generates pre-verified tool stubs from troubleshooting guides (TSGs) with UI-editable placeholders — and the AutoEval pipeline that benchmarked FLASH's incident-triaging agent across 60,000+ tickets.
• FLASH itself is a Microsoft Research project; see the public FLASH project page and the internal Microsoft blog.
• The sections below describe FLASH's design for context, then the evaluation and tool-stub work I built on top of it.

Introduction

FLASH is a multi-agent solution that can complete complex workflows with high reliability and efficiency for routine tasks. Routine tasks are recurring activities carried out regularly following established procedures, often demanding significant human effort in daily life. For instance, diagnosing incidents triggered by monitoring alerts using predefined Troubleshooting Documents (TSG).

The FLASH agent framework introduces several components designed to improve the reliability, efficiency, and affordability of workflow automation.

Main Features

• Status Supervision continuously monitors the execution status of each step within a workflow. It explicitly reasons about the current step’s outcome—whether it's complete, stalled, or requires further action—and determines the appropriate next agent and subsequent step based on this evaluation. This dynamic decision-making ensures robust and adaptive workflow execution, even in complex or non-linear scenarios.


• Flowchart Compilation transforms text-based workflows into structured, executable flowcharts, significantly reducing runtime parsing overhead and minimizing execution errors. FLASH also supports a specialized grammar within the workflow definition, allowing users to define custom steps and transitions. This grammar-based approach enables consistent parsing aligned with user knowledge, ensuring more reliable and interpretable workflow execution.


• Domain Knowledge Integration allows users to enhance agent performance by incorporating relevant expertise at both the team and workflow levels. With a customizable prompt configuration feature, users can inject team-specific knowledge into built-in agents to reflect their unique operational context. Additionally, users can define workflow-specific knowledge to guide agent behavior for particular tasks. This flexible design ensures the agents operate with relevant expertise, improving the accuracy and efficiency of automated workflows.


• Hindsight Integration processes insights from past failures by collecting and analyzing previously failed cases within a comprehensive offline repository of similar incidents. FLASH reflects on these failures to identify root causes and effective resolutions, and incorporates this knowledge during runtime to guide decision-making. This reflection mechanism helps improve accuracy, prevent repeated errors, and enhance overall system performance in future executions.
• User-controllable Execution (Coming Soon) enables user to manage workflow execution by interacting with the agent, facilitating actions such as skipping a step or rerunning a previous step.
• Context Compression (Coming Soon) harnesses insights from extensive past executions to automatically determine the optimal prompt compression for routing tasks, enhancing execution efficiency and reducing costs.

---

2. AutoEval: Neuro-Symbolic Environment Synthesis for Resilient Agent Evaluation (Theory)

Evaluating Large Language Model (LLM) agents in industrial settings presents a fundamental dichotomy: static datasets fail to capture non-stationary environments (data drift) and implicit logic, while production testing carries unacceptable operational risk. We present AutoEval, a framework for Counterfactual World Generation that synthesizes executable evaluation environments directly from natural language policy documents.

Unlike stochastic simulators that hallucinate states or collapse modes, AutoEval utilizes a neuro-symbolic architecture—combining LLM-based multi-agent debate for logic extraction with Breadth-First Search (BFS) constraint propagation—to guarantee 100% path coverage of the policy logic.

1. Introduction

The transition of LLMs from passive chatbots to active agents requires robust evaluation frameworks. Current methodologies face three critical failures:

1. Data Drift: Static benchmarks (e.g., AgentBench) rapidly depreciate as production environments evolve.
2. Implicit Logic: Real-world workflows (TSGs) contain hidden branching logic missed by standard sets.
3. Combinatorial Explosion: Naive "One-Shot" LLM generation fails on complex TSGs. A workflow with sequential branching factors (e.g., [5, 6, 3, 2, 5, 2, 3, 6]) generates >21,000 unique paths. Dumping the entire context into an LLM results in context overflow and mode collapse, failing to enumerate the long tail of edge cases.

We introduce AutoEval, a framework that bridges the gap between data coverage and case coverage. AutoEval treats documentation not as unstructured text, but as latent code, compiling natural language policies into executable "Toolstubs" that simulate API responses and ground truth trajectories.

Key Contributions:

• Neuro-Symbolic Extraction: A novel pipeline using Multi-Agent "Critics" for implicit logic detection and symbolic BFS for exhaustive path exploration.
• Environment Synthesis: A mechanism to generate "Toolstubs" (mock environments) that are mathematically consistent with real-world Knowledge Bases (KBs).
• Scalable Systematic Workflow: A step-by-step decomposition approach that handles large-context combinatorial explosion, enabling the generation of tens of thousands of coherent test cases where naive methods fail.
• Drift Resilience: Empirical evidence demonstrating AutoEval's ability to maintain benchmarking accuracy in shifting environments where static datasets fail.

2. Problem Formulation: The Latent POMDP

We define a Troubleshooting Guide (TSG) as a partial specification of a Partially Observable Markov Decision Process (POMDP), $\mathcal{M} = \langle S, A, T, R, \Omega, O \rangle$.

• The Challenge: The transition function $T(s'|s,a)$ is latent, encoded in unstructured text (the TSG).
• The Goal: Construct a synthetic evaluation environment $\mathcal{E}_{syn}$ such that an agent $\pi$ evaluated on $\mathcal{E}_{syn}$ accurately predicts performance in the real production environment $\mathcal{E}_{real}$.
• The Constraint: $\mathcal{E}_{syn}$ must cover all logical branches $B$ in the TSG, including implicit edge cases (e.g., "If CPU is high..."), which are rarely represented in historical logs.

3. Methodology: Neuro-Symbolic Compilation

AutoEval operates as a compiler, adopting a systematic, step-by-step workflow to manage context window limits and combinatorial complexity. Instead of attempting "one-shot" generation from the full document, it decomposes the problem via a four-phase architecture.

3.1 Phase 1: Ingestion & DAG Construction

The system parses the raw markdown of the policy documents. It maps nodes (steps) and transitions to construct a preliminary Directed Acyclic Graph (DAG) representing the explicit workflow.

3.2 Phase 2: Multi-Agent Logic Discovery (The "Critic")

Standard parsing misses implicit logic. We employ an LLM-based Multi-Agent Debate mechanism (Proposer <-> Critic) to analyze nodes for conditional dependencies not explicitly linked in the text (e.g., implicit "Else" conditions).

$$ \text{LogicTree} = \text{Consensus}(\text{Debate}(\text{Node}_i, \text{Context})) $$

This step converts linear text instructions into branching logic trees, uncovering hidden constraints often missed by single-pass extraction.

3.3 Phase 3: BFS Symbolic Execution Engine

To ensure comprehensive evaluation, we reject Monte Carlo sampling in favor of symbolic search.

• Constraint Propagation: The engine simulates a user traversing the DAG step-by-step. By limiting the LLM's context to the current node's logic, we avoid context-window overload.
• Path Splitting: At every Condition Axis (decision point detected by the Critic), the engine bifurcates the timeline (e.g., Path A: Disk Full, Path B: Disk OK).
• Coverage Guarantee: This BFS approach mathematically guarantees coverage of all detected logic paths. Even with combinatorial explosion (e.g., $N=21,600$ paths), the systematic traversal ensures every valid permutation is instantiated as a unique "Toolstub."

3.4 Phase 4: Data Refinement & Knowledge Base Grounding

To prevent hallucinations common in synthetic data, generated queries and states are validated against Reference Knowledge Bases (e.g., Kusto log schemas, API definitions). This ensures the "Toolstubs" return realistic schemas and values, maintaining consistency with real-world data structures (The "Sim-to-Real" Link).