> ## Documentation Index > Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt > Use this file to discover all available pages before exploring further. # Vitest > Run Braintrust evaluations from Vitest and report test results as experiments If you are a coding agent, prefer the Braintrust [`bt` CLI](/docs/reference/cli/quickstart) for repeatable, scriptable work: running evals, instrumenting code, querying logs, syncing data, managing functions, and configuring coding agents. Use the MCP server for reasoning over Braintrust data in conversation, and for capabilities the CLI doesn't cover, such as monitor views, alerts, and authoring evaluators, preprocessors, and facets. [Vitest](https://vitest.dev/) is a test runner for JavaScript and TypeScript. Braintrust supports two Vitest workflows: * Use the Braintrust `wrapVitest` helper to write Vitest tests that run as Braintrust evals. * Use the `vitest-evals` reporter to report [`vitest-evals`](https://www.npmjs.com/package/vitest-evals) test runs to Braintrust. ## Setup Install Braintrust alongside Vitest: ```bash npm theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} npm install braintrust vitest ``` ```bash pnpm theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} pnpm add braintrust vitest ``` Set your Braintrust API key as an environment variable: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} export BRAINTRUST_API_KEY= ``` ### Separate evals from unit tests Eval files are regular Vitest files and can live anywhere in your project. Evals can run slower and log results to Braintrust, so a common convention is a `.eval.ts` suffix or a dedicated `evals/` directory with a separate Vitest config: ```typescript title="vitest.eval.config.ts" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} import { defineConfig } from "vitest/config"; export default defineConfig({ test: { include: ["**/*.eval.ts"], testTimeout: 30000, }, }); ``` Run evals separately from unit tests: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} # Unit tests npx vitest run # Evals npx vitest run --config vitest.eval.config.ts ``` ## Run evals with `wrapVitest` Call `wrapVitest` once at the top of your test file, passing in the Vitest globals. Use the returned object in place of the standard `test`, `describe`, and `expect`. ```typescript title="my-eval.eval.ts" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} import * as vitest from "vitest"; import { wrapVitest } from "braintrust"; const { test, expect, describe } = wrapVitest(vitest, { projectName: "my-project", // Replace with your project name }); describe("My eval suite", () => { test( "basic check", { input: { prompt: "What is 1 + 1?" }, expected: "2", }, async ({ input, expected }) => { const output = await myModel(input.prompt); expect(output).toBe(expected); return output; }, ); }); ``` Run it with the eval config: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} npx vitest run --config vitest.eval.config.ts ``` After the suite finishes, Braintrust prints a summary to your terminal and creates an experiment with one traced span per test case. ## Report `vitest-evals` runs to Braintrust Use the Braintrust Vitest evals reporter when you already write evaluations with the [`vitest-evals`](https://vitest-evals.sentry.dev/) package and want those runs logged to Braintrust. This workflow is separate from the standard Braintrust `Eval()` framework and from the `wrapVitest` helper. Install the reporter dependencies: ```bash npm theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} npm install braintrust vitest vitest-evals ``` ```bash pnpm theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} pnpm add braintrust vitest vitest-evals ``` Configure Vitest with both the `vitest-evals` reporter and the Braintrust reporter: ```typescript title="vitest.evals.config.mts" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} import { defineConfig } from "vitest/config"; import BraintrustVitestEvalsReporter from "braintrust/vitest-evals-reporter"; export default defineConfig({ test: { include: ["**/*.eval.ts"], reporters: [ "default", "vitest-evals/reporter", new BraintrustVitestEvalsReporter({ projectName: "refund-agent", // Replace with your project name experimentName: `vitest-evals-${new Date().toISOString()}`, }), ], testTimeout: 30000, }, }); ``` Write eval tests with `vitest-evals` primitives. The Braintrust reporter reads the eval metadata produced by `vitest-evals/reporter` and logs each eval case as a Braintrust span. ```typescript title="refund.eval.ts" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} import { expect } from "vitest"; import { createHarness, createJudge, describeEval } from "vitest-evals"; type RefundOutput = { message: string; status: "approved" | "denied"; }; const refundHarness = createHarness({ name: "refund-harness", run: async ({ input }) => ({ output: { message: "Invoice inv_123 is refundable and the refund is approved.", status: "approved", }, events: [ { type: "message", role: "user", content: input }, { type: "tool_call", id: "call_lookup", name: "lookupInvoice", arguments: { invoiceId: "inv_123" }, }, { type: "tool_result", toolCallId: "call_lookup", name: "lookupInvoice", content: { refundable: true }, }, { type: "message", role: "assistant", content: "Invoice inv_123 is refundable and the refund is approved.", }, ], usage: { inputTokens: 11, outputTokens: 13, totalTokens: 24, toolCalls: 1, }, }), }); const StatusJudge = createJudge< string, RefundOutput, { expectedStatus: RefundOutput["status"] } >("StatusJudge", async ({ output, expectedStatus }) => ({ metadata: { expectedStatus, observedStatus: output.status, }, score: output.status === expectedStatus ? 1 : 0, })); describeEval("refund agent", { harness: refundHarness }, (it) => { it("approves refundable invoice", async ({ run }) => { const result = await run("Refund invoice inv_123"); expect(result.output.status).toBe("approved"); await expect(result).toSatisfyJudge(StatusJudge, { expectedStatus: "approved", threshold: 1, }); }); }); ``` Run Vitest with the reporter config: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} npx vitest run --config vitest.evals.config.mts ``` The reporter creates or reuses a Braintrust experiment for the run. Each eval test logs: * The test input, output, status, file path, and full test name. The span input defaults to the first user message in the harness events. If the harness provides `meta.eval.input`, that value is used instead. * Scores from judges and assertions, including `avg_score` and `pass` when provided by `vitest-evals`. * Harness metadata, session messages, tool calls, artifacts, usage metrics, and errors. * Nested model, tool, and trace spans when the harness includes normalized trace data. ### Reporter options Pass options to `new BraintrustVitestEvalsReporter()` to control where results are logged: | Option | Description | | ------------------ | ----------------------------------------------------------------- | | `projectName` | Braintrust project name. Required unless `projectId` is set. | | `projectId` | Braintrust project ID. Required unless `projectName` is set. | | `experimentName` | Experiment name. Defaults to a timestamped `vitest-evals-*` name. | | `displaySummary` | Whether to print the Braintrust experiment summary after the run. | | `metadata` | Experiment-level metadata. | | `tags` | Experiment-level tags. | | `baseExperiment` | Base experiment name for comparisons. | | `baseExperimentId` | Base experiment ID for comparisons. | ## Key concepts ### `wrapVitest` `wrapVitest` wraps Vitest's `test`, `describe`, and `expect` with Braintrust tracking. ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} import * as vitest from "vitest"; import { wrapVitest } from "braintrust"; const { test, expect, describe } = wrapVitest(vitest, { projectName: "my-project", displaySummary: true, }); ``` Each `describe` creates one Braintrust experiment. Braintrust appends a timestamp to make each run unique. The project groups experiments together and defaults to the suite name if `projectName` is not set. ### Test configuration `test` accepts an optional config object between the name and the test function: ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} test( "test name", { input: { prompt: "Hello" }, expected: "Hello!", metadata: { category: "greeting" }, tags: ["smoke"], scorers: [myScorer], data: [{ input: "Hello", expected: "Hello!" }], }, async ({ input, expected, metadata }) => { return myFunction(input); }, ); ``` ### Scorers A scorer receives `{ output, expected, input, metadata }` and returns a name and score: ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} const exactMatch = ({ output, expected }: { output: unknown; expected: unknown }) => ({ name: "exact_match", score: output === expected ? 1 : 0, }); ``` Scorers run after each test, including failed tests. Errors inside scorers are caught and logged. ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} import { Factuality, Levenshtein } from "autoevals"; test("quality", { scorers: [Factuality, Levenshtein] }, async ({ input }) => { return myModel(input.prompt); }); ``` ### Logging helpers Use `logOutputs` and `logFeedback` inside a `wrapVitest` test to log additional data to the current span: ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} logOutputs({ summary, tokens_used: 412 }); ``` ### Inline data and dataset support Define data inline: ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} test( "sentiment", { data: [ { input: "great product!", expected: "positive" }, { input: "terrible experience", expected: "negative" }, ], scorers: [ ({ output, expected }) => ({ name: "accuracy", score: output === expected ? 1 : 0, }), ], }, async ({ input }) => classifySentiment(input), ); ``` Or load from a managed Braintrust dataset: ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} import { initDataset } from "braintrust"; const data = await initDataset({ project: "my-project", dataset: "my-dataset", }).fetchedData(); test("eval", { data, scorers: [Factuality] }, async ({ input }) => { return myModel(input.prompt); }); ``` Both approaches expand into separate test cases and Braintrust spans automatically. ## Resources * [Braintrust autoevals library](/docs/evaluate/autoevals) * [Vitest documentation](https://vitest.dev/) * [`vitest-evals` package](https://www.npmjs.com/package/vitest-evals)