Documentation

Getting Started

TestHide is a self-hosted CI/CD platform built around one idea: when a build or test fails, you should get an explanation — not just a red X. It runs your pipelines on your own agents, then an AI layer (an LLM Investigator plus diagnostic ML models trained on your own history) tells you why a run failed — the likely root cause, whether it's flaky, the most similar past failures, and where to look. Everything runs on your own infrastructure; your logs and source never leave it. It ships as a Docker Compose stack from Docker Hub — no source build required. This guide takes you from an empty server to a first diagnosed build.

What is TestHide?

In one line: failure intelligence for CI/CD. TestHide ingests test reports and runs CI builds on distributed agents, then analyses every failure with AI. The headline is the Investigator — an LLM root-cause agent that reads the failing log, runs tool calls across your build and test data, and returns a plain-language verdict with a visible per-investigation cost meter. It is multi-provider (Google / OpenAI / Anthropic) and can run a fully on-device local model, so an investigation can complete without a single byte leaving your network. Underneath, TestHide vectorises each payload (MiniLM embeddings for text, CLIP for screenshots) and trains specialised models on your own history, surfacing insights through the in-app dashboard and a REST API (/api/v3/). The diagnostic models include:

• Root-Cause Classifier — multi-modal failure categorisation (text + numeric + image), a transformer classifier on a DistilBERT / BERT / RoBERTa backbone.
• Flakiness Predictor — gradient-boosted scoring over rolling test history.
• Failure Retriever — contrastive embeddings + FAISS index for nearest-failure search.
• OOD Detector — autoencoder + isolation forest for novelty scoring.
• Visual Diff — element detection (YOLO) + learned SSIM ensemble for screenshots.
• Log Signature Miner — master-pattern extraction over log lines.
• Bug Linker — dense retrieval over linked issue (Jira) embeddings.
• Emerging Issues — changepoint / z-score trend detection over failure groups.

Which models are active on a given instance depends on the tier's AI feature flags from the synced license; the Free baseline enables root-cause and flakiness. See the tiers & pricing page for what each license unlocks.

Why teams choose TestHide

Architecture at a glance

A standard install is a single Docker Compose project (name: testhide) made up of an application core plus a self-contained observability stack. Everything is pulled from Docker Hub (thuesdays/*) — there is no host repository to clone.

The default Root-Cause backbone is distilbert-base-uncased (RC_MODEL_NAME); diagnostic summaries use a local Phi-3.5-mini GGUF model, with an optional Gemini / OpenAI / Anthropic fallback when keys are configured.

System requirements

The AI worker reserves 5 GB and may grow to 12 GB during training, while ai-api ranges 1.5–6 GB; the two are anti-correlated by design, so 32 GB is the realistic floor.

Quickstart

The steps below stand up a working instance. They mirror the published docker-compose.yaml and .env.example; the installation guide covers each variable in depth.

1 · Prepare the deployment directory

mkdir -p /opt/testhide/{ssl,data/mongo,ssh_keys,sandbox_data,releases,monitoring_scripts}
cd /opt/testhide

curl -O https://testhide.com/static/landing/docker-compose.yaml
curl -O https://testhide.com/static/landing/.env.example && cp .env.example .env

MONGO_DATA_PATH defaults to /opt/testhide/data/mongo; that host directory must exist and be writable before first launch.

2 · Configure the environment

Edit .env. At a minimum set the public URLs, a unique JWT_SECRET, and the database / Grafana credentials. Replace every CHANGE_ME… placeholder before booting.

# Domain & URLs
PUBLIC_URL=https://testhide.yourcompany.com
API_URL=https://testhide.yourcompany.com:7771
WS_URL=wss://testhide.yourcompany.com:7771
FRONTEND_PUBLIC_URL=https://testhide.yourcompany.com

# Security  (settings.py refuses to boot in prod/staging with the dev-default secret)
ENVIRONMENT=production
DEBUG=false
JWT_SECRET=$(python3 -c "import secrets; print(secrets.token_urlsafe(64))")

# Data
MONGO_USER=testhide_user
MONGO_PASS=<strong-password>
MONGO_DATA_PATH=/opt/testhide/data/mongo
REDIS_PASSWORD=<strong-password>

# Observability
GRAFANA_ADMIN_PASSWORD=<strong-password>

3 · Place TLS certificates

Nginx terminates TLS. Drop your certificate and key in ./ssl/ on the host (that directory is bind-mounted read-only into the frontend at /etc/ssl). The container paths are set by CERT_FILE / CERT_KEY, which default to /etc/ssl/testhide.crt and /etc/ssl/testhide.key.

# Place your real cert + key as ssl/testhide.crt and ssl/testhide.key
chmod 600 ssl/testhide.key

# ...or a self-signed cert for evaluation only:
openssl req -x509 -nodes -newkey rsa:4096 -days 365 \
  -keyout ssl/testhide.key -out ssl/testhide.crt \
  -subj "/CN=testhide.local" && chmod 600 ssl/testhide.key

Grafana reuses the same cert by default (GRAFANA_CERT_HOST_CRT / _KEY); override only if it needs a separate certificate.

4 · Launch the stack

docker compose up -d --force-recreate
docker compose logs -f backend ai-api ai-worker

First boot pulls several GB of images plus the AI models — allow 10–15 minutes. The UI comes up on port 443, the REST API and WebSocket on 7771, and Grafana dashboards on 3000. Prometheus binds to 127.0.0.1:9090 (loopback only) by default.

5 · Verify

docker compose ps
curl -sk https://localhost/api/v3/health | python3 -m json.tool

A healthy instance returns an ok status with Mongo and Redis connected. If a service is restarting, check docker compose logs <service> — the most common first-boot failure is a missing or default JWT_SECRET in production mode (see the callout above).

Connect a build agent

Builds run on agents (nodes). Install one on any machine and point it at your server with a license key; every supported platform is covered in the installation guide. The quickest path on Linux:

curl -fsSL https://dl.testhide.com/install.sh | sudo bash
sudo testhide config --url wss://YOUR_DOMAIN:7771 --license-key YOUR_LICENSE_KEY
sudo systemctl enable --now testhide

The agent appears on the Nodes page as available once it connects. Agent WebSocket auth is always enforced — every agent must present a valid token derived from its license key or the connection is rejected before the upgrade.

Run your first build

There are two ways to get test results into TestHide:

1. Run it from TestHide. Create a job, add a test_provider build step pointing at your test suite, and run it on an agent. Results, artifacts, and logs are captured automatically. recommended
2. Push from your existing CI. Add the matching framework reporter plugin (pytest, unittest, JS/TS, or .NET) so your runner emits a TestHide report file, then have a test_provider step ingest it.

When a build fails, the AI layer diagnoses it automatically: open the build's Overview and Investigate tabs to see the root cause, similar past failures, and suspect commits. Model accuracy improves as labelled history accumulates — watch training progress on the AI Assist dashboard at /ai-assist.

Reports & Builds

Builds are the central entity in Testhide. Every CI run produces a build record that aggregates its pipeline, test results, AI insights, LLM-eval steps, artifacts, and the captured environment. The builds list lives at /builds; a single build opens at /build/:id/overview inside a shell that keeps a left navigation rail visible across every build view.

Build lifecycle & statuses

A build moves through a fixed set of canonical statuses. The wire values are lowercase American spellings (the canonical source is app/build/models.py → Build); the client tolerates British cancelled but emits canceled.

The build page

BuildShellComponent renders a persistent navigation rail beside every build view. Each entry routes under /build/:id.

The rail also carries cross-build links (View / Edit Job, Run) and a job build-history panel.

Build actions & permissions

Actions are available from the rail and the overview, each gated by a BUILDS_* permission. Routes below come from the server route table (app/routes.py); handlers live in app/build/build_actions_view.py, build_start_view.py and build_delete_view.py.

Matrix builds

A matrix build has one parent document and N child builds — one per parameter combination (OS, browser, shard, etc.). The parent stores a child_rollup read-model so the overview can show a "N/M completed · K failed" summary live, without recounting on the client. Restarting only failed children is supported per child via restart_child, and Retry Flaky Tests works at both parent (all children) and single-child scope.

Data retention

A daily cron deletes builds (and their jobs' artifacts and AI data) older than the tier's retention_days; builds flagged Keep Forever are exempt. The Free baseline is 7 days (app/services/tier_enforcement.py); longer windows for paid tiers come from the synced license rather than being hard-coded.

Build Steps

Build steps are the executable units of work inside a Testhide job. The step list is edited on the Build tab of the job editor (/jobs/edit/:id) and is stored on the job and its builds as build_step[]. Each step has a type and a typed configuration; the agent runs them in order. The llm_eval type adds LLM-as-judge / snapshot evaluations as a native part of the pipeline.

Step types

The supported step types are enforced by the job validator (VALID_STEP_TYPES in app/job/validators.py). Unknown types are rejected with UNKNOWN_STEP_TYPE on save.

Most script steps share an always_run flag (run even if a previous step failed) and an optional timeout_seconds (1–86400). Steps can be reordered by drag-and-drop in the editor; step names must be unique within a job.

llm_eval — YAML reference

The llm_eval step nests its judge and threshold under typed sub-objects. The field names below match the backend parser (app/build/llm_eval_step.py) and the job-editor form.

build_step:
  - type: llm_eval
    name: "customer-support-quality"
    input_dataset: data/customer-questions.jsonl   # optional candidate file (.jsonl, relative path, <= 50 MB)
    # cases:                                        # ...or inline cases instead of input_dataset
    #   - id: case-1
    #     expected: "the snapshot / golden answer"
    #     actual: "candidate output"               # optional; omit for smoke mode (auto-pass)
    judge:
      kind: llm_as_judge            # llm_as_judge | snapshot
      model: gpt-4o-mini            # judge model (default: gemini-2.0-flash)
      prompt: registry://judges/customer-support-quality/v3
      temperature: 0.0
      # mode: exact                 # snapshot only: exact | fuzzy | semantic (default exact)
    threshold:
      metric: judge_score
      min: 0.80                     # pass threshold, must be in [0.0, 1.0]
    parallelism: 4                  # concurrent cases (>= 1, server cap 32)
    cost_budget_usd: 0.50           # abort the step if estimated cost exceeds this (server cap 50.0)
    on_fail: pr_check_fail          # pr_check_fail | warn | continue

Step execution flow

For most step types the agent runs the script/command locally and reports an exit code. The llm_eval step is special: it is executed server-side over WebSocket so the judge has access to the build data and prompt registry.

1. The agent's build executor reaches an llm_eval step and sends an LLM_EVAL_STEP_REQUEST WebSocket message (build_id, step_index, eval_yaml) to the backend.
2. The backend handler (app/ws_handler/handlers/llm_eval.py) enforces the AI-evals license quota and server caps (max 500 inline cases, cost budget ≤ $50), then builds the judge — LLMJudge for llm_as_judge (resolving any registry:// prompt) or SnapshotJudge for snapshot.
3. The work runs as a detached task, so PING / BUILD_FINISHED keep flowing. EvalRunner.run_for_build scores each case, streaming an LLM_EVAL_PROGRESS frame back to the agent per case.
4. The result is persisted into the build document at cfg.build_step[step_index].eval (and the run row in db_eval_runs) so the AI Evals tab can render it.
5. The backend returns an LLM_EVAL_STEP_RESULT message (exit code, pass/fail counts, pass rate, cost, tokens, per-case judge outputs).
6. The agent applies on_fail: pr_check_fail marks the step (and build) failed when below threshold, while warn / continue keep the build green.

Viewing results

Open a build and select the AI Evals tab in the build rail (/build/:id/ai-evals). It loads GET /api/v3/builds/:id/evals (permission BUILDS_READ) and lists every llm_eval step with:

• Step name (eval_id) and step_index.
• Pass / fail outcome with pass_rate and the passed/failed/total_cases counts.
• Estimated cost (cost_usd) and tokens_used, aggregated across cases.
• Regression / improvement deltas vs. the previous run.
• Expandable per-case judge output (case_id, verdict, score, reasoning, raw response, cost, tokens, latency).
• For snapshot judges, a "View diff" action opening the snapshot diff viewer.

Plugins

Testhide ingests test results from your CI pipeline. The official reporter plugins are what you add to your test runner so it emits results in a format Testhide understands. Each plugin writes a single Testhide Report Format v1 file (junittests.xml) — JUnit-extended XML with a stable failure id, a resolution, captured output, attachments and suite metadata — which the build agent then collects and uploads. Plugins exist for pytest unittest JavaScript / TypeScript .NET.

The common model

Every plugin produces the same on-disk contract, so the agent and the AI layer parse results identically regardless of language. The flow is always the same:

1. Add the reporter to your test runner — a CLI flag, a config-file entry, a logger, or an attribute, depending on the framework.
2. Run your tests. The plugin writes junittests.xml incrementally: each test result is flushed immediately, so a partial report survives even an interrupted run.
3. The Testhide agent collects the report. A test_provider build step runs the runner on an agent, which reads the report file and pushes the build to the Testhide REST API.

The agent (and any direct API caller) authenticates to the Testhide server with a Personal Access Token. The token, the server URL, and the project a build belongs to are configured once on the agent / job — not inside each plugin. The plugins themselves never talk to the network (the optional exception is Jira enrichment); they only write the report file. Run identifiers such as the build number and branch travel as suite metadata that you attach on the command line (e.g. build=1042, branch=main); the reserved build and branch keys get special handling in Testhide, everything else is stored as free-form metadata.

Compatibility overview

All plugins write to junittests.xml by default and are safe under parallel / sharded execution: each worker writes to a temp chunk that is atomically merged (file-locked) into the final report.

pytest

Package testhide-pytest-plugin (PyPI, Python ≥ 3.8). It registers itself as a pytest plugin via the pytest11 entry point — there is nothing to import. Reporting is off until you pass the --report-xml flag, which both enables the plugin and sets the output path.

Install

pip install testhide-pytest-plugin

Enable & run

pytest --report-xml=junittests.xml

Suite name & run metadata

Set the <testsuite name="..."> and attach build/run identifiers with repeatable --meta KEY=VALUE options:

pytest --report-xml=junittests.xml \
       --suite-name=api-tests \
       --meta build=1042 --meta branch=main

Options

• Parallelism — fully compatible with pytest-xdist; add -n auto and the plugin merges results from every worker.
• Flaky / retries — works with pytest-rerunfailures; every rerun of a failing test is recorded as a separate <testcase> for accurate instability tracking.
• Attachments & metadata — inject per-test artifact links and suite metadata through the pytest_testhide_get_test_case_properties and pytest_testhide_add_metadata hooks.

# parallel run with reruns
pytest -n auto --reruns 5 --report-xml=junittests.xml

unittest

Package testhide-unittest-plugin (PyPI, Python ≥ 3.8). Python's built-in unittest has no plugin system, so the reporter ships as a thin wrapper module you invoke instead of python -m unittest. Use the optional [jira] extra for Jira enrichment.

Install

pip install testhide-unittest-plugin
# optional Jira enrichment:
pip install "testhide-unittest-plugin[jira]"

Enable & run (CLI)

Run the wrapper exactly like unittest; anything unittest accepts is passed through, and the Testhide options are consumed by the wrapper:

# discover & run, writing the report:
python -m testhide_unittest discover -s tests -p "test_*.py" \
       --report-xml junittests.xml --suite-name api-tests \
       --meta build=1042 --meta branch=main

Options

The wrapper exits 0 when all tests pass and 1 otherwise, so it drops straight into CI.

Programmatic use & per-test enrichment

import unittest
import testhide_unittest as th

class LoginTests(unittest.TestCase):
    def test_login(self):
        "User can log in with valid credentials."   # docstring -> docstr (automatic)
        th.attach("/tmp/screenshot.png")            # repeatable; images/logs/json
        th.set_info('{"env": "staging"}')           # free-form context
        th.set_jira("PROJ-123")                     # link a ticket
        ...

Or drive it from your own entry point with the runner, passing suite metadata directly:

import unittest
import testhide_unittest

runner = testhide_unittest.TesthideTestRunner(
    report_path="junittests.xml",
    metadata={"build": "1042", "branch": "main"},
)
unittest.main(testRunner=runner)

The writer is parallel-safe: each test is written to a temp chunk under .{report}_temp/ and atomically merged (file-locked) into the final report.

JavaScript & TypeScript

Package @testhide/reporters (npm, Node ≥ 14). One package exposes a sub-path per framework — Jest, Mocha, Vitest and Playwright — that you register as a reporter. The frameworks are optional peer dependencies, so you only install the one you use. Every reporter takes the same options object: { outputPath, suiteName, meta } (outputPath also reads the TESTHIDE_REPORT_XML env var).

Install

npm install --save-dev @testhide/reporters

Enable per framework

module.exports = {
  reporters: [
    'default',
    ['@testhide/reporters/jest', {
      outputPath: 'junittests.xml',
      suiteName: 'api-tests',
      meta: { build: '1042', branch: 'main' },
    }],
  ],
};

mocha --reporter @testhide/reporters/mocha \
      --reporter-options outputPath=junittests.xml

export default {
  test: {
    reporters: [
      ['@testhide/reporters/vitest', { outputPath: 'junittests.xml' }],
    ],
  },
};

export default {
  reporter: [
    ['@testhide/reporters/playwright', { outputPath: 'junittests.xml' }],
  ],
};

.NET

The .NET reporter ships as a single VSTest logger that works for xUnit, NUnit and MSTest, because they all run through dotnet test. The only package you need is Testhide.Reporting.VSTest (its FriendlyName is testhide); the shared writer Testhide.Reporting.Core is pulled in automatically.

Install & use

dotnet add package Testhide.Reporting.VSTest
dotnet test --logger "testhide;LogFilePath=junittests.xml"

Per-framework meta-packages (Testhide.Reporting.Xunit, .NUnit, .MSTest) exist purely for discoverability — they contain no code and just depend on the VSTest logger.

Logger parameters

Parameters are name=value pairs, ;-separated:

Example with metadata

dotnet test --logger "testhide;LogFilePath=junittests.xml;SuiteName=api-tests;meta.build=1042;meta.branch=main"

• Source mapping — file / line are captured when the framework provides them (e.g. with portable PDBs / SourceLink).
• Per-test metadata — test traits named docstr / jira / info map to report properties, and result attachments become attachment properties.

Getting reports into Testhide

Once your runner writes junittests.xml, a test_provider build step on the agent collects it and the build is pushed to the Testhide API. Point the job's report_paths (Test reporting → report paths) at the file; the agent parses it after the run and uploads the structured results — no extra API calls from your tests. The agent authenticates with a Personal Access Token; the server URL and the target project are part of the agent / job configuration, not the plugin.

Nodes & Agents

A node (or agent) is a worker machine that runs your builds. Each agent opens a persistent WebSocket back to the backend, advertises its labels and capabilities, and receives build steps to execute. Nodes are managed under Settings → Nodes; the backing collection is db_nodes. This guide covers the node model, the health system, installing agents on every platform, pools, per-node Python environments, the Docker agent pool, and the soft-delete lifecycle.

Static vs dynamic nodes

Every node carries two classification fields: node_type (static or dynamic) and source (binary or docker).

A node's stable identity is its node_uuid (MAC/hostname-independent), falling back to secure_id then name on reconnect. The secure_id is the agent handshake identity and is never serialized to the browser.

Node statuses

Disconnected nodes are recovered by a reconciliation task that fixes busy nodes with no active builds and revives unavailable nodes that still hold a verified WebSocket. You can run it on demand with Force Reconcile.

Node health & the penalty system

Each node accumulates a health_penalty driven only by genuine infrastructure faults — build-content failures (your failing tests) and indeterminate outcomes are fail-open and add zero penalty. The score shown in the UI is derived as score = 100 − penalty × 5.

• Critical reasons (disk_full, read_only_fs, bad_interpreter, docker_daemon_down) jump straight to the quarantine threshold.
• Healing — each clean build decrements the penalty; the penalty also time-decays, so a transient infra blip recovers on its own.
• Disconnect grace — one disconnect is tolerated; restart/update-driven disconnects add no penalty.
• Release residual — releasing a node from quarantine leaves it one heal below the threshold so a still-sick host re-quarantines on the next fault rather than flapping back to available.

Installing agents

Download the agent from Settings → Nodes → Download Agent. The modal is populated from the release manifest on the CDN (R2): it prefers launcher_assets / launcher_version when present and falls back to the raw client assets. Platforms surfaced: Windows x64, Linux x64, macOS Intel, and macOS ARM.

The launcher wraps the client binary and handles auto-update: it watches the release manifest and migrates to the published launcher_version without a manual reinstall, so a fleet can be rolled forward centrally.

Linux quickstart

curl -fsSL https://dl.testhide.com/install.sh | sudo bash
sudo testhide config --url wss://YOUR_DOMAIN:7771 --license-key YOUR_LICENSE_KEY
sudo systemctl enable --now testhide

Docker agent image

docker run -d \
  -e TESTHIDE_URL=wss://your-host:7771 \
  -e TESTHIDE_LICENSE_KEY=<license-key> \
  -e TESTHIDE_LABELS=docker,ephemeral \
  -e TESTHIDE_NODE_TYPE=dynamic \
  -e TESTHIDE_SOURCE=docker \
  --network testhide-net \
  thuesdays/testhide-agent:latest

Agent authentication token

Every agent authenticates its WebSocket handshake by sending ?token=SHA256(licenseKey) as a query parameter (browsers can't set WS headers, so the token rides on the URL). The backend validates it against the configured instance license key and rejects with HTTP 401 before any WS upgrade — an unauthenticated client never enters the hub.

GET /api/v3/node/connect?token=<sha256-hex-of-license-key>
# Backend accepts either SHA256(rawKey) or SHA256(UUID-normalized key).

The license key is provisioned out-of-band on the agent host — it is never learned from the backend. Provide it via any of:

• the TESTHIDE_LICENSE_KEY environment variable,
• a testhide_override.json file, or
• the agent's config command.

Node pools

A node pool is a named group of nodes that builds target via execution_target: pool:<name>. Routing resolves candidate nodes by pool and by label. Manage pools under Settings → Nodes → Pools (the node_pools collection).

A node's own concurrent_executors caps how many builds it runs at once. A pool cannot be deleted while active nodes still reference it — unassign the pool from all nodes first.

Per-node Python ENV management

Each node can host managed Python environments, driven over the agent WebSocket from POST /api/v3/node/{id}/envs (requires NODES_WRITE). Supported actions: list, create, delete, reinstall_requirements, list_packages, check_health.

• Type — virtualenv (created from a base interpreter) or pyenv (install/manage interpreter versions).
• Location — global or workspace.
• Requirements — a requirements blob (≤ 64 KB) installed/reinstalled into the env.
• Inspect — list installed packages and run a health check.

POST /api/v3/node/{id}/envs
Content-Type: application/json
Requires: NODES_WRITE

{
  "action": "create",
  "type": "virtualenv",
  "name": "api-tests",
  "location": "workspace",
  "base_python": "3.11",
  "requirements": "pytest==8.2.0\nrequests"
}

Per-node remote operations

From a node's row or detail view you can drive a range of remote operations. The non-destructive ones require NODES_READ; the destructive/fleet ones require the superuser REMOTE_CONTROL_EXECUTE permission.

• SSH terminal — a browser-based xterm. The backend provisions a one-off SSH key to the target username, the agent confirms deployment, then the backend bridges stdin/stdout over WebSocket. Requires ssh_enabled on the node and REMOTE_CONTROL_EXECUTE.
• Quarantine / Release — manually take a node out of dispatch or return it. Release restores the pre-quarantine status, drops the penalty to one heal below threshold, and refreshes the stored health snapshot. (REMOTE_CONTROL_EXECUTE)
• Reset penalty — clear health_penalty (optionally also resetting status) for one or many nodes. (REMOTE_CONTROL_EXECUTE)
• Force Reconcile — run the reconciliation task immediately to fix stuck busy nodes and revive falsely-offline ones. (REMOTE_CONTROL_EXECUTE)
• Clean disk — instruct selected agents to clear their working/temp space. (REMOTE_CONTROL_EXECUTE)
• Get log — pull the agent's recent log over RPC. The backend resolves the node's live ws_id first so multi-instance reconnects don't time out. (NODES_READ)
• Remote screen — view the node's live screen / screencast for debugging GUI runs.
• Execute / Restart / Force-update / Install tooling — run a shell command (batch / powershell / python / bash, server-enforced 10-min wall + 5-min idle timeouts), restart the agent, force an update, or push Python/Git installers. All REMOTE_CONTROL_EXECUTE.

Docker pool & spawning + resource limits

Ephemeral dynamic agents are spawned as containers from the image configured in Configuration → Docker Agents. The backend never touches the Docker socket directly — all container lifecycle goes through a hardened sidecar, and spawns are serialized with a Redis lock so multiple backend workers can't race the max_containers limit.

POST   /api/v3/nodes/docker-agents/spawn        # body: { labels, memory_limit, instance_id }
GET    /api/v3/nodes/docker-agents              # list tracked containers (live status)
POST   /api/v3/nodes/docker-agents/{id}/stop    # stop
POST   /api/v3/nodes/docker-agents/{id}/start   # start
DELETE /api/v3/nodes/docker-agents/{id}         # destroy (stop + remove + soft-delete node)
GET|POST /api/v3/nodes/docker-agents/config     # read / update config

On spawn, the backend injects the WS URL and license key as environment (TESTHIDE_URL, TESTHIDE_KEY, TESTHIDE_LICENSE_KEY, TESTHIDE_LABELS, TESTHIDE_NODE_TYPE=dynamic, TESTHIDE_SOURCE=docker, plus TESTHIDE_INSTANCE_ID and TESTHIDE_SERVICE_MODE=1), attaches the container to the testhide-net bridge, and polls up to 90 s for the agent to connect. Spawn progress and live container logs stream to the UI over WebSocket events.

Each spawned agent is capped at the configured CPU and PID limits and runs with no-new-privileges so a runaway build can't DoS the host. An idle reaper destroys containers that have been idle longer than docker_agent_idle_timeout_min. Destroy stops and removes the container and soft-deletes the node. Spawn, destroy, start, stop, and config-write are all REMOTE_CONTROL_EXECUTE; listing and reading config require NODES_READ.

Delete, restore & purge

Deleting a node is a soft delete: the document is moved from db_nodes to the db_deleted_nodes archive with deleted_at / deleted_by metadata. Any active builds running on the node are failed first so they don't become permanent orphans. Single delete, bulk delete, and Docker-agent destroy all use this path. Deleting requires the NODES_DELETE permission.

• Deleted Nodes archive — browse soft-deleted nodes at GET /api/v3/nodes/deleted (the "Deleted Nodes" view). Requires NODES_READ.
• Restore — POST /api/v3/nodes/deleted/{id}/restore moves an archived node back to db_nodes; it returns as unavailable until it reconnects. (NODES_WRITE)
• Purge (single) — POST /api/v3/nodes/deleted/{id}/purge permanently hard-deletes one archived record. Cannot be undone. (NODES_DELETE)
• Purge all — POST /api/v3/nodes/deleted/purge permanently empties the entire archive in one shot. (NODES_DELETE)

External CI (Jenkins & TeamCity)

TestHide interoperates bidirectionally with Jenkins and TeamCity. A TestHide job can depend on an external build (wait for it to pass before running), be triggered by one (an external build that finishes starts a TestHide build), and trigger one (when a TestHide build finishes it kicks off an external build). Linked builds show a live external-CI badge on their build cards, and external Jenkins/TeamCity nodes are rendered alongside internal build dependencies on the build Dependencies page.

Overview

The integration ties an external CI server to a TestHide job through three independent mechanisms — a dependency gate, an inbound trigger, and an outbound action — all configured per job on the External CI tab of the job editor. Status flows both ways: TestHide polls the external server (and accepts inbound webhook pushes) to keep build-card badges and the Dependencies graph current, and it calls the external server's API to start builds and read results.

1 · Connect a CI server

Register each Jenkins or TeamCity server once under Configuration → Endpoints and add a new endpoint (type jenkins or teamcity). This requires the CONFIGURATION admin permission.

• Jenkins — Base URL + Username + API token (create the token under the Jenkins user's Configure → API Token). Outbound triggers send a crumb-protected buildWithParameters request.
• TeamCity — Base URL + API token (a bearer access token), or a username / password pair.
• Inbound-webhook secret (optional) — a per-endpoint shared secret used to authenticate notifications that the CI server pushes back to TestHide (see Triggers).
• TLS verification — a toggle to verify the server's certificate; turn it off only for an internal server using a self-signed certificate.

After filling the form, use the Test Connection button (POST /api/v3/integrations/ci/test) to validate the URL and credentials before saving. A green result confirms TestHide can reach the server and authenticate; the job discovery picker (POST /api/v3/integrations/ci/jobs) then lists the external jobs / build configurations.

2 · The External CI job tab

Open a job at /jobs/edit/:id and select the External CI tab. It has three independent subsections; configure any combination. Saving requires JOBS_WRITE.

Dependencies (gate)

The job's builds wait until the chosen external Jenkins/TeamCity build passes before they run. Fields:

• Endpoint — which registered CI server to query.
• External job / build configuration (ref) — the Jenkins job name or TeamCity build-configuration id to watch.
• Branch — restrict the gate to a specific branch.
• Timeout (optional, timeout_minutes) — how long to hold the build before the gate fails it.

A failed external dependency cancels the TestHide build; a still-running one holds it in the queue. A terminal-but-not- success state (FAILURE / ABORTED, and UNSTABLE unless the dep opts into unstable_is_success) drives the gate to a decision rather than waiting forever. Resolved dep states are sticky — a transient UNKNOWN from a poll error never regresses an already-terminal dependency or re-closes an open gate.

Triggers (inbound)

When the chosen external build finishes with a matching status, it starts this TestHide job. This is powered by an inbound webhook: point your Jenkins or TeamCity notification at the TestHide notify endpoint and authenticate it with the endpoint's secret. The trigger's on_status list (default ["SUCCESS"]) controls which finishing states fire the job.

POST {TestHide}/api/v3/integrations/ci/notify/{endpointId}
X-Testhide-CI-Token: 
Content-Type: application/json

Replace {TestHide} with your public base URL, {endpointId} with the endpoint id from step 1, and the header value with that endpoint's inbound-webhook secret. Inbound deliveries are deduplicated on (endpoint, ref, number, state) (Redis-backed, with an in-process fallback), so a webhook retry never double-fires the job.

Actions (outbound)

When this TestHide build finishes — on success, on failure, or always — TestHide triggers an external Jenkins/TeamCity build, optionally passing parameters. Parameter values support placeholders that are substituted at trigger time:

The outbound action is the trigger_external_ci post-build step; it reads configuration.external_ci.actions and fires once on the won terminal write so a retried finalize never double-triggers the external build.

3 · Live status & badges

Linked builds show an External-CI badge (provider icon + state) on their build cards. Statuses update live over WebSocket: a leader-only poller refreshes external state every 15 seconds (with a 30 s startup delay and a 20 s per-pass runtime cap), and — if an inbound webhook is configured — updates arrive instantly when the external server pushes a notification.

The build Dependencies page (/build/:id/dependencies) renders external Jenkins/TeamCity nodes alongside internal build dependencies in the same graph. External nodes are color-coded by status and are click-through to the external build in Jenkins/TeamCity.

4 · Security & operations

• Credentials encrypted at rest — API tokens and passwords are encrypted in the database and never logged or returned to the browser. Set SECRET_ENCRYPTION_KEY in the environment so the backend can encrypt and decrypt them.
• Authenticated inbound webhooks — every notification to /api/v3/integrations/ci/notify/{endpointId} must present the per-endpoint secret via the X-Testhide-CI-Token header (or ?token=); the compare is timing-safe and requests without a valid token are rejected with 401.
• Brute-force throttle — repeated bad tokens are counted per endpoint (not per IP, so a misconfigured CI controller can't ban its own trusted internal IP); over the threshold the notify path answers 429 instead.
• On-prem CI allowed — endpoints on private (RFC 1918) networks are permitted, so internally hosted Jenkins/TeamCity servers work without exposing them publicly.
• Upstream failures surface inline — when an external server is unreachable or errors, TestHide returns an inline 424 (Failed Dependency) error rather than a maintenance redirect, so the cause is clear.
• Circuit breaker — a per-endpoint circuit breaker backs off a server that is down. While an endpoint is in cooldown the poller treats it as UNKNOWN (still pending, retry later) rather than hammering it.

5 · Limitations

• The dependency gate timeout is bounded by the job's max queue-wait — a gate cannot hold a build longer than the job is allowed to wait in the queue (its max_queue_wait_minutes ceiling).
• Without an inbound webhook, external state refreshes on a ~15 second poll cadence (leader-only); configure a webhook for instant updates.
• Inbound dedupe and trigger fan-out are bounded; during a Redis outage the in-process fallback is per-replica, so a duplicate webhook delivered to a different replica may produce at-least-once trigger semantics (the gated-dep update path stays idempotent).

Examples

Inbound webhook (trigger this job)

Point your CI server's notification at this URL with the secret header and a JSON body:

POST https://testhide.yourcompany.com/api/v3/integrations/ci/notify/65f0a1c2d3e4
X-Testhide-CI-Token: s3cr3t-endpoint-token
Content-Type: application/json

{
  "status": "SUCCESS",
  "job": "backend-integration",
  "branch": "main",
  "build_number": 1042
}

Jenkins post-build "Notify TestHide" (curl)

Add a post-build shell step (or an Execute shell stage) that posts the result back to TestHide so it can trigger the linked job:

curl -sS -X POST \
  -H "X-Testhide-CI-Token: s3cr3t-endpoint-token" \
  -H "Content-Type: application/json" \
  -d "{\"status\":\"$BUILD_STATUS\",\"job\":\"$JOB_NAME\",\"branch\":\"$GIT_BRANCH\",\"build_number\":\"$BUILD_NUMBER\"}" \
  https://testhide.yourcompany.com/api/v3/integrations/ci/notify/65f0a1c2d3e4

AI Assist Dashboard

The dashboard at /ai-assist is the operations console for the AI-Assist subsystem — the layered build-failure-diagnosis pipeline (the rules-v1 rule engine → the RCA engine → optional LLM deep investigation) plus the diagnostic / prediction models that feed it. It surfaces live worker health, per-component diagnostics, training controls, and the federated model-exchange status. The route is guarded by the AI_ASSIST_READ permission and the ai_feature_dashboard license capability; mutating actions additionally require AI_ASSIST_EXECUTE.

Worker stats banner

The header polls GET /api/v3/ai/admin/stats every 60 seconds through the shared AiService stats stream (timer(0, 60000)). The endpoint returns the single worker document (_id="global_ai_worker") from db_ai_worker_stats:

A small health helper classifies the worker as healthy, warning (queue_size > 1000 or error rate > 10 %), or critical (failed / processed ratio > 50 %). A Refresh action forces an immediate re-poll, and a collapsible console tails the worker log via GET /api/v3/log every 2 seconds.

Diagnostic model cards

Below the banner a responsive grid renders one card per diagnostic component. Diagnostics are loaded from GET /api/v3/ai/admin/diagnostics, which returns the latest report per component from db_ai_diagnostics (with the matching training_history_<name> merged in). The current component set is:

Each card surfaces:

• Status — healthy / warning / error / running, with a loaded-in-memory indicator. A benchmark result for a non-model card (corpus, dataset, yolo) reads as a benign N/A (“not applicable”) rather than an error — those components are raw data stores or have their own audit route, so they have no golden-set benchmark.
• Top issues — the first issues from the latest diagnostic run.
• Staleness & version — model-registry metadata flags models not trained in the last 30 days.
• Run Check — POSTs /api/v3/ai/admin/diagnostics/:component to mark the component pending, then polls /api/v3/ai/admin/diagnostics on a 10 s interval until the report leaves the running/pending state.
• Details — opens a dialog with advanced stats, a file-integrity table, and parameter analysis.

Training & maintenance menu

A kebab menu in the header drives training and maintenance, each item gated by AI_ASSIST_EXECUTE. These enqueue work for the AI worker via POST /api/v3/ai/admin/diagnostics/run (body { "component": "<name>", "config": { … } }), or via the per-component POST /api/v3/ai/admin/diagnostics/:component trigger. The accepted component names are allow-listed server-side (system_train_heavy, system_train_fast, dataset, root_cause, retriever, templates, ood, yolo, emerging, corpus, flakiness, build_rca, training_lock).

• Train Fast — a quick incremental pass over the lightweight models (system_train_fast).
• Heavy Training — opens the train dialog to pick a backbone (DistilBERT / BERT / RoBERTa), epochs, batch size, learning rate, and a freeze backbone toggle (recommended for small datasets); runs as system_train_heavy with the chosen config.
• Train a single model — a submenu for retriever, templates, OOD, YOLO, emerging, and root_cause.
• Convert to ONNX — exports the trained classifier for faster CPU inference.
• Rebuild Index — rebuilds the retriever FAISS index.
• Force Unlock — clears a stale training lock left by a crashed run.
• Clear Queue — deletes corrupted / empty items from the worker input queue (confirm dialog).
• Build Golden Flakiness — materializes a flakiness golden set from the DB for benchmarking.
• Restart Worker — sends a restart signal to the AI worker process (confirm dialog).

A live training monitor card polls the training lock (GET /api/v3/ai/admin/diagnostics/status/training_lock) every 30 seconds, shows a determinate progress bar while a run is active, and offers a Terminate action that posts { "component": "training_lock", "signal": "terminate" } to POST /api/v3/ai/admin/diagnostics/signal. The status view also self-heals: if it sees a running state but the on-disk training.lock file is gone (past a short grace period) it resets the component to idle.

Model exchange & quick links

The dashboard also exposes the federated model-sync card (upload / download / both): it reads GET /api/v3/ai/model-sync/status, triggers a transfer with POST /api/v3/ai/model-sync/run, and then polls status on a 4 s interval until any in-flight transfer clears. A Rule Catalog link is badged with the count of pending auto-suggested Layer-1 rules from GET /api/v3/ai/rule-suggestions/count.

Each model card's Details action opens the per-model detail page at /ai-assist/model/:name (see the AI Model Detail guide), and the toolbar links reach the Eval Explorer (benchmark history), the Cost dashboard (/api/v3/ai/costs/daily), and the Activity History page (/api/v3/ai/activity).

Admin endpoints used by the dashboard

AI Model Detail Pages

Every diagnostic / prediction model has a dedicated detail page at /ai-assist/model/:name. The page surfaces live diagnostics, input-drift, training history, benchmark trends, recent predictions, and per-model controls (retrain, version promote/rollback, inference-mode, and ad-hoc inference). The route is guarded by the ai_feature_dashboard license capability; mutating actions require the AI_ASSIST_EXECUTE permission.

The models

The canonical model ids are defined by KNOWN_MODELS in app/ai_assist/model_registry.py. The detail-page route accepts any of these names (and the corpus data asset):

Detail page tabs

The active tab is reflected in the URL via a ?tab=<slug> query param (TAB_SLUGS = ['overview', 'training', 'benchmarks', 'configuration', 'predictions', 'actions']), so tabs are deep-linkable and survive back/forward navigation. A stat strip above the tabs shows status, last-trained age, training duration, and the number of logged runs and benchmarks.

Actions tab

Each action below requires AI_ASSIST_EXECUTE:

• Trigger Retrain — queues a training run for this model.
• Download Model — downloads the trained artifact (GET /api/v3/ai/models/:name/download, served as a ZIP).
• Run Benchmark — queues the benchmark suite against the golden set.
• Model version — pick a registered version from GET /api/v3/ai/model-registry/:name/versions and Promote it (POST …/:version/promote) or Rollback to the previous active one (POST …/rollback).
• Inference mode — switch between rules (heuristics only), shadow (ML runs but does not act), and ml (ML active) via GET/POST /api/v3/ai/models/:name/mode.
• Run on sample — ad-hoc inference for the models in the backend allow-list _VALID_MODELS = {root_cause, retriever, yolo}: paste a log/error string, an image URL (yolo), or a JSON payload. It POSTs to /api/v3/ai/inference and polls GET /api/v3/ai/inference/:request_id for the result.

The model-detail endpoint

All tabs (except drift and predictions, which have their own endpoints) are populated from a single call. The handler is AIAssistModelDetailView in app/ai_assist/api/admin.py:

GET /api/v3/ai/admin/model/:name
Authorization: Bearer <token>
Requires: CONFIGURATION_READ

Example response (name=root_cause):

{
  "ok": true,
  "data": {
    "name": "root_cause",
    "diagnostics": {
      "component": "root_cause",
      "status": "healthy",
      "issues": [],
      "last_trained": "2026-06-05T08:14:02",
      "training_result": "ok",
      "training_duration_ms": 41200,
      "advanced_stats": { "accuracy": 0.91, "labels": 12 }
    },
    "training_history": [
      { "ts": "2026-06-05T08:14:02", "result": "ok", "error": null, "duration_ms": 41200 }
    ],
    "benchmarks": [
      { "model_name": "root_cause_classifier", "version": "1.3.0",
        "passed": true, "primary_metric": 0.88, "created_at": "2026-06-05T09:01:00" }
    ],
    "worker_stats": { "queue_size": 0, "processed_total": 5123 }
  }
}

The endpoint is backed by db_ai_diagnostics (the current diagnostics doc plus training_history_<name>, last 10 runs) and db_model_benchmarks (last 10 runs). Benchmark rows are looked up under both the UI short name and the runner's canonical name (e.g. flakiness ↔ flakiness_predictor, build_rca → root_cause_classifier) so they survive a reload. For the corpus asset the audit is computed and persisted on demand when no cached doc exists, so a fresh install still renders a populated page.

Navigation

From the AI Assist Dashboard, click Details on any model card. The page lazy-loads via the Angular router and re-fetches on every visit; subsequent refreshes are silent (content stays visible while a background reload runs).

Cloud LLM model selection (chat, diagnosis, Auto-PR)

The detail pages above cover the bundled ML models. The separate cloud LLM that powers Ask AI chat, deep diagnosis, and Auto-PR is config-driven from the LLM Providers page — there are no model environment variables. You pick the provider and model in the UI; only the API keys and provider priority live in settings.

Models are chosen per tier, so each kind of task can use a different model:

• Chat — fast, cheaper model for the Ask AI conversational assistant.
• Deep diagnosis — stronger model for root-cause investigation.
• Code (Auto-PR) — code-specialised model for generating fixes; if no code model is configured for a provider it falls back to that provider's deep-diagnosis model.

AI Insights Tab

The AI Insights tab on the build detail page gives a unified view of what the background ML pipeline concluded about each test in a build — status, flakiness label and score, root-cause prediction, coarse failure type, and which model produced each insight. The tab is loaded on demand (a lazy child of the build-detail view) and reads the build's insight documents. It requires the BUILDS_READ permission. The handlers live in app/build/build_insights_view.py.

What it shows

On open, the tab fires two endpoints in parallel (a single forkJoin) via BuildInsightsService:

Both endpoints enforce project scope: the build's project id (pi) is checked against the caller's allowed projects before any insight is returned — a caller with a global BUILDS_READ still gets 403 for a build in a project outside their token scopes.

Insight document fields

Insights are stored in db_ai_insights using compact short keys; the list endpoint expands them to the names below. (The list view surfaces only the prediction label + score; uncertainty is detail-level.)

Pagination

The list uses keyset pagination on _id (newest first). Pass the next_cursor from the previous response as the cursor query param; cursors are ObjectId strings. next_cursor is null on the final page (returned only when the page was full). An optional status filter narrows the list to one status value.

GET /api/v3/builds/<id>/insights?limit=100&cursor=<next_cursor>&status=failed

The list response (v2 envelope) looks like:

{
  "ok": true,
  "data": {
    "build_id": "665f1c2a9b1e4a0012ab34cd",
    "total": 47,
    "items": [
      {
        "insight_id": "665f1c2a9b1e4a0012ab34ce",
        "test_name": "tests/api/test_login.py::test_expired_token",
        "status": "failed",
        "flaky": "Random",
        "flakiness_score": 72,
        "prediction": { "label": "env_issue", "score": 0.84 },
        "source_kind": "root_cause",
        "decision_category": "auto",
        "coarse_failure_type": "network",
        "created_at": "2026-06-05T09:12:44+00:00"
      }
    ],
    "next_cursor": "665f1c2a9b1e4a0012ab34ce"
  }
}

The summary response aggregates the same collection:

{
  "ok": true,
  "data": {
    "build_id": "665f1c2a9b1e4a0012ab34cd",
    "total": 47,
    "status_counts": { "failed": 23, "passed": 24 },
    "prediction_labels": { "env_issue": 5, "product_bug": 10, "infra_issue": 3 },
    "flaky_count": 8,
    "flakiness_breakdown": { "Random": 4, "Rare": 3, "Systematic": 1 },
    "coarse_failure_types": { "memory": 3, "network": 2 },
    "source_kinds": { "root_cause": 12, "flakiness": 8, "ood": 3 }
  }
}

Insights vs. the build-level diagnosis

AI Insights are per-test background ML inference that runs automatically on every result — no configuration required. They are distinct from two neighbouring surfaces:

• The compact build diagnosis banner / chip (the denormalized ai summary on the build: src, cat, conf, verdict, recs) summarises the whole build's failure cause from the rules-v1 → RCA → LLM pipeline — one verdict per build, not per test.
• The AI Evals tab shows results of LLM-judge evaluations you explicitly configure as llm_eval build steps, whereas AI Insights are produced automatically.

AI Evals & Benchmarks

Testhide ships two related but distinct evaluation surfaces. The Eval Explorer at /ai-assist/eval-explorer is the benchmark history for the ML prediction models (it requires AI_ASSIST_READ). The Evals section at /evals is the LLM-eval suite — golden-set evaluations run by pluggable judges (it requires EVALS_READ). This page covers both.

Eval Explorer — model benchmark history

Reachable from the AI Assist Dashboard, the Eval Explorer is a flat table of benchmark runs across the prediction models (e.g. flakiness, build_rca). Each row is one benchmark evaluation against a golden set: a PASS/FAIL regression badge, the model name, version, the primary metric, all metrics, and the run date. Filter by model pill and by status (All / PASS / FAIL). The same benchmark data also appears on each model's detail page under the Benchmarks tab.

Evals — the LLM-eval suite

The /evals section lists LLM evaluations and lets you drill into each one. An eval scores model or prompt output against a golden set using a pluggable judge. Evals are aggregated from the build records themselves: every llm_eval build step stores its result under cfg.build_step[i].eval, and the eval endpoints group those results by eval_id (a build carries an eval_ids[] array for fast lookup). There is no separate eval document — the eval name is the eval_id from the step YAML.

The judge registry

Judges live in app/ai_core/eval/judges and are exported from the package __init__. The registry currently includes:

The eval detail page

Open an eval from /evals to reach its detail page. It is backed by the following endpoints (all require EVALS_READ and enforce per-project token scope):

Judge output

Every eval run stores its per-case judge output in eval.judge_outputs[] (surfaced on the run-detail page as cases). For the llm judge the model is asked to return a JSON object that is parsed and validated; the required keys are score, verdict, and reasoning:

{
  "score": 0.92,
  "verdict": "pass",
  "reasoning": "The output correctly identifies the root cause and is concise."
}

• score — a float in [0.0, 1.0]; anything outside the range is rejected.
• verdict — one of pass, fail, regression, improvement.
• reasoning — one short paragraph explaining the verdict.

Each executed judge call is also written to the prompt-registry audit log (db_prompt_uses) and to db_ai_calls for cost tracking, with the parsed output, token count, cost, and latency. A malformed or timed-out judge response is recorded as verdict: fail rather than crashing the run, and a per-case cost_budget_usd cap returns verdict: cost_exceeded when the call costs more than the budget.

Baseline approval

Snapshot evals use an approval workflow. An approver holding the EVALS_APPROVE permission can promote a run's output to the new baseline so future snapshot runs diff against it and only fail when the output meaningfully changes. The relevant endpoints (all EVALS_APPROVE, all project-scoped):

CLI

The eval harness has a CLI for running and inspecting evals directly from a build record:

# Run a snapshot eval (mode overrides the manifest default)
python -m app.ai_core.eval run --eval my-eval --candidate cases.jsonl --mode semantic

# Run an llm_eval step against a specific build + step index
python -m app.ai_core.eval run-for-build --build <build_id> --step 2 \
  --model gemini-2.0-flash --threshold 0.80 --cost-budget 1.0 \
  --parallelism 4 --on-fail pr_check_fail

# Show recent eval runs from build records
python -m app.ai_core.eval show --eval my-eval --runs 10 --json

Prompt Registry

The Prompt Registry stores versioned prompt templates with SHA-256 content deduplication and semantic versioning. Prompts are referenced by eval configurations and llm_eval steps through the registry:// URI scheme, and every resolved-and-executed use is logged for cost and audit reporting. The package is app/ai_assist/prompt_registry; documents live in db_prompts, db_prompt_versions, and db_prompt_uses.

registry:// references

A reference resolves a prompt name (which is path-like, e.g. judges/customer-support-quality) to a concrete version:

registry://<name>/<version>
registry://<name>/latest
registry://<name>              # bare name is treated as latest

# Examples
registry://judges/sql-correctness/1.2.0
registry://judges/customer-support-quality/latest
registry://root-cause-summarizer

At resolve time the backend looks up the prompt in db_prompts and the matching record in db_prompt_versions. When the version token is latest (or omitted), the most-recently-inserted non-deprecated version is selected. The resolved prompt exposes its model, temperature, declared variables, and an optional output_schema; render() substitutes the {{input}}, {{expected}}, and {{actual}} placeholders.

Versioning & SHA-256 dedup

Versions are not supplied by hand — the registry assigns them. On upsert, the new content is hashed (SHA-256) and compared against the latest stored version for that name:

• If the hash matches the latest version, the existing version is returned unchanged — a true dedup, no new record is written.
• If the content differs, the patch component is auto-bumped (e.g. 1.2.0 → 1.2.1) and a new PromptVersion is inserted.
• The first version of a brand-new prompt starts at 0.1.0.

Versions must be valid MAJOR.MINOR.PATCH semver, and a version can be flagged deprecated so latest skips it.

Document model

Browsing prompts

The frontend PromptRegistryService reads the registry through two REST endpoints, both gated by the PROMPTS_READ permission (granted to any authenticated user by default — prompts are judge configurations, not secrets):

The prompt picker dialog (prompt-dialog) lists available prompts, previews the selected version's template, and inserts the chosen registry://<name>/<version> reference back into the form field. Both endpoints also support the Testhide-API-Version: 2 envelope while remaining backwards-compatible with the legacy v1 shape.

CLI usage

# List all prompt names, latest versions, and owners
python -m app.ai_assist.prompt_registry list

# Show a resolved prompt version (content + model + temperature)
python -m app.ai_assist.prompt_registry show registry://judges/sql-correctness/1.2.0

# Unified diff between two versions
python -m app.ai_assist.prompt_registry diff \
  registry://judges/sql-correctness/1.0.0 \
  registry://judges/sql-correctness/1.2.0

# Upsert from a file (auto-assigns / bumps the version; dedups on identical content)
python -m app.ai_assist.prompt_registry upsert judges/sql-correctness prompt.txt

LLM Cost & Pricing

Every cloud LLM call Testhide makes — AI investigation, RCA summarisation, the llm eval judge, and auto-fix drafting — is priced and recorded so spend is visible and capped. The single source of truth for cost computation is app/ai_core/pricing.py; the runtime knobs live in settings.py as LLM_* environment variables.

Supported providers (bring-your-own key)

The AI Narrator / Investigator routes to whichever provider you have configured a key for. Native SDKs back Google Gemini, OpenAI and Anthropic Claude; one OpenAI-compatible backend additionally serves Groq, Mistral, DeepInfra, OpenRouter and Hugging Face at their own base URLs. A fully local GGUF model is the always-available, offline last resort.

Keys, base URLs, per-provider monthly budgets and daily request limits are stored in the database (db_llm_providers) and editable in the UI — no redeploy. The per-tier provider chain is likewise DB-configurable (db_llm_routing), overriding the LLM_*_PROVIDER_CHAIN env defaults below.

The cost formula

Cost is computed per call from the token split and the model's per-1K-token input/output prices:

cost_usd = (prompt_tokens / 1000) * input_price_per_1k
         + (completion_tokens / 1000) * output_price_per_1k

The result is rounded to 6 decimal places. compute_cost(model, prompt_tokens, completion_tokens) returns (cost_usd, unknown); unknown is True only when the provider cannot even be inferred from the model name, in which case the cost is 0.0 and a warning is logged (an accounting gap to fix, never a blocked call).

Where costs are recorded

• db_ai_calls — one row per cloud LLM call with caller, model, provider, cost_usd, prompt_tokens, completion_tokens, latency_ms, and the build / eval / job / project ids. This is what the Cost dashboard reads.
• db_prompt_uses — the prompt-registry audit log; each resolved-and-executed prompt records its own cost_usd and tokens_used.

The Cost dashboard endpoint is GET /api/v3/ai/costs/daily (requires AI_ASSIST_READ). It aggregates db_ai_calls by day with group_by of model (default), caller, user, project, or all, over a window of up to 90 days.

Environment variables

Per-user-per-day call quotas (chat / deep) are configured in the UI — AI Assist settings, defaults 25 / 2; the per-call timeout (60 s) is baked in. The former AI_QUOTA_* and LLM_REQUEST_TIMEOUT_S env vars were removed.

Price-resolution order

compute_cost resolves a model's rate in this order, stopping at the first match:

1. Model override — LLM_PRICING_OVERRIDES, longest matching prefix wins.
2. Provider table (when the caller passes provider=) — the same model id can cost differently on different hosts (e.g. an open-weight Llama on Groq vs DeepInfra), so a provider-specific table is consulted first when the provider is known.
3. Local model — if LLM_LOCAL_MODEL_FREE is on and the model is a local prefix and the resolved provider is local/empty, cost is $0 (a hosted Llama is never mistaken for free).
4. Gemini table — built-in Google / Gemini per-model prices.
5. Anthropic table — built-in Anthropic per-model prices (longer prefixes matched first).
6. OpenAI table — built-in OpenAI per-model prices.
7. Groq / Mistral / DeepInfra tables — built-in per-model prices for the open-weight router hosts.
8. Per-provider default — an unlisted model from a known provider is priced at a conservative provider-typical rate (so a new/renamed model is never billed at $0).
9. Blended legacy fallback — old model aliases keyed by a single blended per-1K rate.

If none of the above match (the provider cannot be inferred), the model is truly unknown: cost is 0.0 and unknown=True.

Built-in default price tables

Prices are USD per 1K tokens (input / output), reviewed 2026-06-14. These are the code-level defaults; the live, operator-editable rates live in the model catalog (db_llm_models) and win over these tables. A representative subset is shown below — pricing.py additionally carries Groq, Mistral and DeepInfra tables.

Google / Gemini

Anthropic

OpenAI

Per-provider defaults (unlisted models)

Examples

Correct an out-of-date Gemini Flash rate and add a price for a brand-new model without touching code:

LLM_PRICING_OVERRIDES='{"gemini-2.5-flash":[0.0003,0.0025],"gpt-4o":[0.005,0.015]}'

Set a conservative per-provider default for any unlisted model from those providers:

LLM_PROVIDER_DEFAULT_PRICES='{"openai":[0.005,0.015],"google":[0.00125,0.005]}'

Cap total monthly spend at $250 and stop billing local inference at zero:

LLM_MONTHLY_COST_CEILING_USD=250
LLM_LOCAL_MODEL_FREE=false

Worked figure: a gemini-2.0-flash call with 800 prompt tokens and 200 completion tokens costs (800/1000) × 0.0001 + (200/1000) × 0.0004 = 0.00016 USD.

Permissions & Roles

Access control in Testhide is permission-based. Route handlers are decorated with @require_permission('CODE') (the caller must hold all listed codes) or @require_role('role'). The authenticated user's role determines which permission codes they carry. Superusers and service accounts bypass every permission check — the decorator short-circuits for them, so they implicitly hold everything.

The RESOURCE_OPERATION model

A permission code is <RESOURCE>_<OPERATION>. There are exactly four operations — READ, WRITE, EXECUTE, and DELETE. Each role is expanded from a base list of resources crossed with all four operations, plus a few explicit extras (evals, security-admin). Granting the BUILDS resource to a role therefore yields BUILDS_READ, BUILDS_WRITE, BUILDS_EXECUTE, and BUILDS_DELETE.

Resources by role

The two base resource lists are USER_PERM (granted to the standard user role) and SUPERUSER_PERM (granted to superuser and service_account). The table below is taken verbatim from app/permissions.py.

Notable gated operations

• REMOTE_CONTROL_EXECUTE — gates dangerous fleet / node operations: quarantine, Docker container spawn, force-reconcile, clean disk, install-python, and PassEncode. Superuser / service account only — the REMOTE_CONTROL resource is not in USER_PERM.
• NODES_DELETE — gates node delete and purge. Regular users carry it (NODES is a user resource), but it is checked explicitly on the destructive node routes.
• AI_FIX_READ / AI_FIX_WRITE — gate the Auto-PR-Fix feature. READ shows the auto-fix badge and the "Suggested fix routing" card on Investigate; WRITE enables the Draft proposal / Create PR actions. AI_FIX is in SUPERUSER_PERM only, so it is off by default for regular users until an admin grants it.
• SYSTEM_READ / SYSTEM_WRITE — the System Errors panel. Superuser / service account only, because it exposes internal infrastructure detail.
• EVALS_APPROVE — approve snapshot baselines and mark / clear known issues. Held by superusers, service accounts, and the named eval_approver role.
• SECURITY_ADMIN — manage banned IPs (the banned-IPs admin endpoints). Held by superusers, service accounts, and the named security_admin role. Not granted to regular users.

Built-in roles

The role catalog is defined in PERMISSIONS in app/permissions.py:

Service accounts

Service accounts are headless users without UI access. The CI agent authenticates its WebSocket handshake with a hashed license-key token, while the account authenticates against the REST API like any other user (typically via a Personal Access Token). Service accounts inherit the full superuser permission set and bypass @require_permission.

LDAP / SSO

Directory integration (available on Cloud Team & Enterprise tiers) maps directory users onto Testhide roles. LDAP uses LDAPS by default — LDAP_USE_SSL=0 drops to plain LDAP and is warned against loudly. The relevant environment variables:

LDAP_URL=ldaps://ldap.corp.example.com:636
LDAP_BIND_DN=cn=svc-testhide,ou=services,dc=corp,dc=example,dc=com
LDAP_BIND_PASSWORD=<password>
LDAP_USER_BASE=ou=users,dc=corp,dc=example,dc=com
LDAP_USE_SSL=1            # 1 = LDAPS (default), 0 = plain LDAP (warns loudly)
LDAP_TLS_INSECURE=0       # 1 = skip cert verification (not recommended)
LDAP_TLS_VERSION=TLSv1.2  # minimum TLS version for LDAPS

Security

Testhide ships with defense-in-depth hardening: JWT secret enforcement, authenticated agent and frontend WebSockets, per-project webhook HMAC signatures, container isolation via a Docker sidecar, supply-chain scanning (SBOM + Trivy + pinned images), an IP allowlist on the database dump endpoint, and TLS on outbound integrations.

JWT authentication

Browser sessions use signed JWTs. The backend refuses to start in production / staging when JWT_SECRET is still the development default, and warns if it is shorter than 32 characters. Generate a strong secret:

python -c "import secrets; print(secrets.token_urlsafe(64))"

A standard login token is valid for 1 hour; a "remember me" login extends it to 7 days. To revoke all outstanding sessions for a user, increment token_version on their user document — the auth middleware rejects any JWT whose embedded version no longer matches.

Agent WebSocket authentication

Every node agent must authenticate its WebSocket handshake. The agent computes SHA256(licenseKey) (lowercase hex) and sends it as a query parameter:

wss://<host>/api/v3/node/connect?token=<SHA256(licenseKey)>

The backend validates the token against the instance licenseKey (accepting both the raw-key hash and a UUID-normalized hash) with a constant-time comparison. A missing or invalid token is rejected with HTTP 401 before the WebSocket is upgraded, so an unauthenticated client never enters the connection hub.

Frontend WebSocket authentication

The UI WebSocket is not public. The auth middleware runs on the handshake (the token is passed via the ?token= query parameter, since browsers cannot set custom WS headers), rejecting anonymous connections and injecting the authenticated user plus their accessible project IDs onto the session. The pub/sub router then enforces per-topic, project-scoped authorization on every subscribe — a client can only subscribe to topics for projects it can access.

Node secrets

The legacy root_password and vnc_password node fields were dead code and have been removed entirely. A node's secure_id (its agent-handshake identity) is never serialized to browser clients — it is stripped from all node API responses unless an internal caller explicitly opts in.

Inbound webhook HMAC signatures

Inbound git webhooks (POST /api/v3/git/notifyCommit) are verified with HMAC-SHA256 using a per-project secret. The signature is computed over the raw request body and the project is identified by header:

X-Testhide-Project: <project_id>          # ObjectId, hex 24
X-Testhide-Signature: sha256=<hex_digest> # HMAC-SHA256(secret, raw_body)

# Generate / rotate a project's webhook secret at any time:
POST /api/v3/project/<id>/webhook_secret/rotate   # requires PROJECTS_WRITE

Docker sidecar isolation

/var/run/docker.sock is not mounted into the backend container. All Docker operations go through a narrow sidecar service over an internal bridge network. The sidecar enforces allowlists for images, exec patterns, and volumes, and requires a bearer token.

Supply chain

Database-dump IP allowlist

The database dump endpoint is guarded by an IP allowlist (decorator @require_ip_allowlist(env_var="DB_DUMP_ALLOWED_IPS")). Entries may be single IPs or CIDR ranges:

DB_DUMP_ALLOWED_IPS=127.0.0.1,10.0.0.0/8
TESTHIDE_TRUST_PROXY=1    # trust X-Forwarded-For behind a load balancer (else use the TCP peer)

Integration security — Jira & Slack

Outbound integrations use TLS and are credentialed through environment variables (never persisted to browser clients):

# Jira (Bug Linker)
JIRA_BASE_URL=https://yourorg.atlassian.net
JIRA_EMAIL=svc-testhide@yourorg.com
JIRA_TOKEN=<api-token>

# Slack notifications
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T.../B.../...

API Tokens (Personal Access Tokens)

Personal Access Tokens (PATs) authenticate against the Testhide REST API without a session cookie. Tokens are long-lived, optionally scoped, and revocable — the recommended auth method for CI scripts, build agents, and external integrations. Every token belongs to the user who created it and inherits that user's project access; a token can never exceed its owner's reach.

Token format

Every token starts with the prefix th_live_ followed by 40 lowercase hex characters (48 characters total). The plaintext is shown once at creation and never stored — only a SHA-256 hash is kept in the db_api_tokens collection.

th_live_53449c66048ca4916f6b357ff45440b113b63c59

Who can create tokens

Any authenticated user can create, list, and revoke their own tokens — there is no special role requirement, and managing your own tokens requires no scope. The optional project_scopes narrowing (below) is validated against the caller's own project access, so a user cannot mint a token for a project they cannot reach. Superusers and service accounts are exempt from that project-access check because they already see every project.

Creating a token

Open Settings → Configuration → API Tokens and click New API Token.

Via the REST API:

curl -X POST https://localhost:8080/api/v3/auth/api-tokens \
  -H "Authorization: Bearer <your-jwt>" \
  -H "Content-Type: application/json" \
  -d '{"name":"ci-runner","expires_days":90,"scopes":["builds:write","jobs:read"]}'

Response 201 Created:

{
  "ok": true,
  "data": {
    "token": "th_live_…",
    "id": "6643a1b2c3d4e5f6a7b8c9d0",
    "user_id": "…",
    "name": "ci-runner",
    "scopes": ["builds:write", "jobs:read"],
    "project_scopes": null,
    "created_at": "2026-05-13T12:00:00+00:00",
    "expires_at": "2026-08-11T12:00:00+00:00",
    "last_used_at": null,
    "request_count": 0
  }
}

Using a token

Pass the token in the standard Authorization header (or, where headers are unavailable, as a ?token= query parameter):

curl https://localhost:8080/api/v3/builds \
  -H "Authorization: Bearer th_live_53449c66048ca4916f6b357ff45440b113b63c59"

The auth middleware detects the th_live_ prefix, hashes the value, looks it up, validates expiry and (if scoped) scopes, then injects the owning user into the request context identically to a JWT session. Each successful request atomically updates last_used_at and increments request_count.

Scopes

A token with no scopes (or containing the wildcard *) has Full Access. Each write scope automatically implies its read counterpart, so you never need to select both. Scope enforcement maps the request method + URL path prefix to a required scope; if the token's scopes don't satisfy it, the request is denied. The full scope vocabulary lives in app/auth/scopes.py:

Project scoping

Pass project_scopes as an array of project IDs to restrict a token to specific projects, independent of its scope grants. The backend validates that each ID is a well-formed 24-character ObjectId, that the project exists, and that the caller can access it. An empty or omitted value means the token inherits the owner's full project access.

Listing tokens

GET /api/v3/auth/api-tokens

Response: { "ok": true, "data": { "tokens": [...], "count": 1 } }

Token hashes are never returned. Up to 200 of the most recently created tokens are listed (newest first). last_used_at and request_count reflect the most recent successful use.

Revoking tokens

DELETE /api/v3/auth/api-tokens/{token_id}   — revoke one (must be yours)
DELETE /api/v3/auth/api-tokens              — revoke ALL for the current user

Revoking a token you don't own (or an unknown ID) returns 404.

Limits & best practices

• Maximum 50 tokens per user. Creating beyond the cap returns 422 — revoke unused tokens first.
• Use short-lived tokens (e.g. 90 days) for CI jobs and rotate them in your secrets manager.
• Grant only the scopes your integration actually needs; add project_scopes to confine the blast radius.
• Never embed tokens in source code — use environment variables or a secrets vault.

Error responses

Tiers & Quotas

Plan limits are enforced inside the backend by TierEnforcementService. The per-tier limits and feature flags are never configured by environment variables — they are delivered by the signed license your instance syncs from the TestHide license server and cached in configuration.license_data. The only fallback is a hard-coded Free-tier baseline (FREE_TIER_LIMITS / FREE_TIER_CAPABILITIES in app/license/manager.py), applied before the first sync and on any downgrade. A daily cron handles billing-period rollover.

Enforcement is always on

Enforcement is a hard-coded constant (_ENFORCE = True) baked into app/services/tier_enforcement.py. There is deliberately no env var or config flag that can switch it to “log only”: the former TIER_ENFORCEMENT_MODE knob was removed precisely because anyone who could read or run the image could otherwise disable tier limits. Enforcement can only ever be made stricter, never looser.

Plan comparison

The values below are the canonical tier capabilities seeded on the license server (testhide_service/population.py → Tier / TierCapability), which is what a signed license actually carries. The cloud prices are taken from the same seed data; the authoritative limits for any given instance are whatever its synced license reports.

^† ai_monthly_budget_usd defaults to 0 (unlimited) on every tier; it is an operator-tunable per-license ceiling on cloud-LLM spend, layered under the per-provider and global cost caps.

Free-tier baseline (hard-coded fallback)

When no license has synced yet — or after a downgrade — the backend falls back to these exact values (app/license/manager.py, mirrored in tier_enforcement.py):

max_concurrent_builds  = 1
max_nodes              = 5
max_users              = 1
retention_days         = 7
ai_requests_monthly    = 0     # 0 = unlimited (self-hosted has no cloud AI quota)
ai_monthly_budget_usd  = 0     # 0 = unlimited

# capabilities: only these two are on; every other flag is False
ai_root_cause = True
ai_flakiness  = True

On the Free tier every other AI model and platform feature is denied until a paid license is synced. Note the 0 → unlimited convention for ai_requests_monthly: a self-hosted instance never has a cloud AI-request quota, so Free is “unlimited” there even though it is the lowest tier.

How limits are sourced

• License sync — LicenseManager.check_and_sync() calls the license server, then writes { valid, tier, capabilities, limits } into configuration.license_data. A limit of 0 from the server (meaning “unlimited”) is normalised to the -1 sentinel.
• Cadence — the license worker re-syncs on a periodic schedule and on instance/user changes; it reports usage telemetry, then refreshes capabilities. A successful sync timestamp drives the grace window (below).
• Reads — TierEnforcementService reads limits from license_data["limits"] and feature flags from license_data["capabilities"]; LicenseManager.can(feature) / can_use_feature_sync() delegate to the same source, so there is a single source of truth.

What gets gated

• Concurrent builds — can_start_build() counts builds in Build.ACTIVE_STATUSES and blocks when the count reaches max_concurrent_builds; over-cap builds stay pending until a slot frees.
• User seats — enforced in auth/views.py::UserAdd and services/user.py::UserService.create_user against max_users (service accounts and soft-deleted users are excluded from the count).
• Connected nodes — max_nodes is propagated to the node fleet; nodes beyond the cap are flagged license_validity=false.
• AI evals / month — the live gate is LicenseManager.check_ai_evals_limit() (called from ai_assist/api/general.py::_check_license), counting authoritative configuration.global_options['ai_requests_this_month']. The parallel TierEnforcementService.can_use_ai_eval() exists only for parity/tests and must not be wired in as a second gate.
• AI feature flags — can_use_feature() / can_use_ai_model() gate individual models (ai_faiss, ai_ood, ai_visual_classifier, ai_investigator, ai_feature_dashboard, …) against the licensed capabilities.

Quota banners

• 80% used — info banner, e.g. “AI evals: 8,123 / 10,000 (81%) this billing period”.
• 95% used — yellow “approaching quota” warning banner.
• 100% used — red banner, and AI calls return 429 Quota Exceeded.

On the self-hosted Free tier ai_requests_monthly is 0 (unlimited), so no monthly AI-request banner is shown. Per-user/day soft limits still apply via the UI-editable quotas in AI Assist settings (defaults: 25 chat / 2 deep; the former AI_QUOTA_* env vars were removed).

Billing-period rollover

A cron at 00:05 UTC daily (app/cron_tasks.py → tier_billing_rollover, spec 5 0 * * *) calls reset_billing_period_if_due(). When billing_period_end has passed, the active usage counter (db_usage_counters) is archived to db_usage_history and a fresh counter opens for the next period. A Redis lock (cron:lock:billing_rollover) prevents a double rollover across instances. The rollover is idempotent — re-running after a partial failure is safe.

License grace & downgrade

Transient licensing outages keep the last-good tier, so a normal license-server blip never disrupts paying customers. The grace window is generous:

• Within grace — the last successfully synced tier and limits stay in effect. A 5xx from the license server or a connectivity error keeps existing state.
• Beyond grace — no successful sync for longer than LICENSE_GRACE_SECONDS (default 14 days) stops the instance from failing open: it is downgraded to Free and the invalid state is propagated to all nodes. The first transient error never triggers this — a prior recorded success is required.
• Explicit rejection — a known-bad code (license.not_found / inactive / revoked), or a license the server reports as valid=false (expired / inactive), causes an immediate downgrade to Free regardless of grace.

Configuration & Environment Variables

Every TestHide component is configured through environment variables. This is the authoritative reference: operator-facing settings (server, datastores, security, observability, LLM providers, license, integrations, agents) each get a full row with default and purpose; the deep AI/ML tuning knobs are listed in compact tables at the end. Defaults shown are the in-code defaults from testhide/settings.py, database.py, gunicorn.conf.py, the app/ service modules, the license server’s config/settings.py, the Angular scripts/set-env.js, and the .NET ConfigService.cs. The curated operator subset is mirrored in static/landing/.env.example.

Backend — Server, networking & TLS

Backend — MongoDB

The effective URI is assembled as mongodb://USER:PASS@HOST:PORT/DB?authSource=AUTH. The host data path (MONGO_DATA_PATH) is a Docker-compose volume binding, not read by the app.

Backend — Redis

Backend — Auth & security

Backend — Gunicorn

Backend — Process-mode toggles

The same image runs as backend, ai-api, or ai-worker depending on these flags (set in docker-compose).

Backend — Build/queue reliability

Per-user AI quotas (Ask-AI chat / deep diagnosis) are configured in the UI (AI Assist settings; defaults 25 / 2 per day) — the former AI_QUOTA_* env vars were removed. The deprecated MAX_BUILD_RETRIES and TIER_ENFORCEMENT_MODE vars were removed entirely — enforcement is hard-coded on (see the callout above).

Backend — Observability

GRAFANA_ADMIN_PASSWORD, GRAFANA_ROOT_URL, GRAFANA_CERT_HOST_CRT/KEY, PROMETHEUS_PORT_BIND, and PUBLIC_URL are consumed by the docker-compose / Grafana / nginx stack, not by the Python app directly — see the Deployment table below.

Backend — LLM providers & keys

LLM_PRICING_OVERRIDES='{"gemini-2.5-flash":[0.0003,0.0025],"gpt-4o":[0.005,0.015]}'
LLM_PROVIDER_DEFAULT_PRICES='{"openai":[0.005,0.015],"google":[0.00125,0.005]}'

Backend — Local LLM (on-device GGUF)

Inference threads, GPU-layer offload and CPU/GPU selection are now sized automatically from the host (CPU cores, and CUDA + free VRAM when present) — the former AI_LLM_THREADS, AI_LLM_GPU_LAYERS and AI_FORCE_CPU env vars are no longer required. Use AI_TRAIN_CPU_ONLY=1 if you need to force CPU on a GPU host.

Backend — Edge AI (on-agent inference)

Backend — AI worker memory & concurrency

The worker’s soft memory ceiling and its executor pool sizes (inference / IO / LLM / model-load) are auto-derived from the container’s RAM budget and usable CPU cores — the former AI_MEMORY_LIMIT_MB and AI_ASSIST_*_WORKERS env vars are no longer required.

Backend — Model-sync (federated model exchange)

Backend — License

Backend — Docker sidecar (SEC-004)

Backend — LDAP / Active Directory (SEC-007)

Backend — vCenter

Backend — Integrations & monitoring sandbox

Backend — Native math-lib thread caps

The native math-lib thread-fan-out caps (OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, NUMEXPR_NUM_THREADS) and the related Intel/HF knobs (KMP_BLOCKTIME, KMP_AFFINITY, KMP_DUPLICATE_LIB_OK, TOKENIZERS_PARALLELISM, HF_DATASETS_NUM_PROC, PYTORCH_CUDA_ALLOC_CONF) are sized automatically from the usable CPU-core count and set by the training pipeline — they are no longer required on a normal install. They remain available as a deployment-environment override only for an exotic host.

Frontend (Angular build-time)

Consumed by testhide_ui/scripts/set-env.js when generating src/assets/env.js — baked into the bundle at build time.

License server (Django — config/settings.py)

.NET agent (TESTHIDE_* overrides)

These override the agent’s on-disk config.json (resolved at C:\ProgramData\Testhide\config.json / ~/.config/testhide). They take effect on next launch.

Deployment / Docker (compose, nginx, Grafana)

These are read by docker-compose / the surrounding stack rather than the Python app directly.

Compose also accepts ${VAR:-default} substitutions, e.g.:

SIDECAR_ALLOWED_IMAGES=${SIDECAR_ALLOWED_IMAGES:-thuesdays/testhide-agent:latest,thuesdays/testhide-backend:latest}
SIDECAR_ALLOWED_EXEC_PATTERNS=${SIDECAR_ALLOWED_EXEC_PATTERNS:-^python\s,^bash\s,^sh\s}

Advanced AI/ML tuning

Root-cause classifier (RC_*)

The classifier batch size is auto-sized from the capacity tier (the former RC_BATCH_SIZE env var is no longer required). The RC_TEST_SIZE and RC_CLEANLAB_* env vars were removed — they had no effect on the shipped CPU pipeline.

Failure retriever / FAISS (RC_* retriever knobs)

Build RCA classifier (BUILD_RCA_*)

OOD detector (AI_OOD_*)

Visual classifier / YOLO (AI_VISUAL_*, AI_YOLO_*)

GPU vs. CPU selection and the YOLO batch size are auto-detected (CUDA + free VRAM) — the former AI_VISUAL_USE_GPU, AI_YOLO_FORCE_CPU and AI_YOLO_BATCH env vars are no longer required.

Flakiness predictor (AI_FLAKY_*)

The flakiness predictor is a gradient-boosted classifier with no epoch concept — the former AI_FLAKINESS_EPOCHS env var had no effect and was removed.

Log-signature miner (LOGSIGN_* / AI_LOGSIGN_*)

Dataset, budget & lock knobs

Dataset shard/row/size caps, the memory-budget chunking factors, corpus and embedding-cache caps, the inference batch cap and vectorizer row concurrency are now derived automatically from the container’s RAM budget and CPU-core count — the former AI_DATASET_MAX_MB, AI_DATASET_MAX_ROWS, AI_DATASET_SHARD_ROWS, AI_COMPACT_AFTER_SHARDS, AI_BUDGET_*, AI_CORPUS_MAX_PER_CLASS, AI_TEXT_CACHE_MAX_ENTRIES, AI_INFER_MAX_BATCH, ROW_CONCURRENCY, AI_PRECOMPUTE_MAX_CONCURRENT and OFFLINE_EMBED env vars are no longer required.

ETA estimator

The ETA estimator's statistical tunings (sample thresholds, duration sanity range, retention, cache TTLs, robust-stats parameters) are baked-in constants — the former ETA_* env vars were removed; there is nothing to configure.

SCM circuit breaker (SCM_CB_*)

Health

Server liveness probe. No authentication required.

curl https://your-testhide-server/api/v3/healthz

import requests

r = requests.get("https://your-testhide-server/api/v3/healthz")
print(r.json())

{"status": "ok", "version": "3.14.2", "build": "a1b2c3d"}

Authentication

Obtain and manage JWT tokens. Session tokens expire in 8 hours; API tokens never expire unless revoked.

curl -X POST https://your-testhide-server/api/v3/authenticate \
  -H "Content-Type: application/json" \
  -d '{"username": "admin@corp.com", "password": "s3cr3t!"}'

import requests

r = requests.post(
    "https://your-testhide-server/api/v3/authenticate",
    json={"username": "admin@corp.com", "password": "s3cr3t!"}
)
token = r.json()["token"]
print(token)

{"token": "eyJhbGci...", "expires_at": "2026-05-21T14:00:00Z", "user_id": 42}

curl -X POST https://your-testhide-server/api/v3/logout \
  -H "Authorization: Bearer $TOKEN"

import requests

requests.post(
    "https://your-testhide-server/api/v3/logout",
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

(no content)

curl https://your-testhide-server/api/v3/me \
  -H "Authorization: Bearer $TOKEN"

import requests

r = requests.get(
    "https://your-testhide-server/api/v3/me",
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)
print(r.json())

{"id": 42, "email": "admin@corp.com", "role": "admin", "projects": [1, 3, 7]}

curl https://your-testhide-server/api/v3/config \
  -H "Authorization: Bearer $TOKEN"

import requests

r = requests.get(
    "https://your-testhide-server/api/v3/config",
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)
print(r.json())

{"version": "3.14.2", "global_options": {"ai_assist": {"enabled": true}, "max_parallel_builds": 20}}

curl -X POST https://your-testhide-server/api/v3/sso \
  -H "Content-Type: application/json" \
  -d '{"saml_response": "PHNhbWxwOlJlc3BvbnNl..."}'

import requests

r = requests.post(
    "https://your-testhide-server/api/v3/sso",
    json={"saml_response": "PHNhbWxwOlJlc3BvbnNl..."}
)
print(r.json()["token"])

{"token": "eyJhbGci...", "user_id": 42, "expires_at": "2026-05-21T14:00:00Z"}

curl -X POST https://your-testhide-server/api/v3/reset-password \
  -H "Content-Type: application/json" \
  -d '{"email": "dev@corp.com"}'

import requests

requests.post(
    "https://your-testhide-server/api/v3/reset-password",
    json={"email": "dev@corp.com"}
)

{"detail": "Reset link sent if the account exists."}

curl -X POST https://your-testhide-server/api/v3/reset-password/confirm \
  -H "Content-Type: application/json" \
  -d '{"token": "abc123xyz", "new_password": "Str0ng!Pass"}'

import requests

requests.post(
    "https://your-testhide-server/api/v3/reset-password/confirm",
    json={"token": "abc123xyz", "new_password": "Str0ng!Pass"}
)

{"detail": "Password updated successfully."}

curl https://your-testhide-server/api/v3/api-tokens \
  -H "Authorization: Bearer $TOKEN"

import requests

r = requests.get(
    "https://your-testhide-server/api/v3/api-tokens",
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)
print(r.json())

[{"id": 1, "name": "CI Bot", "created_at": "2026-01-10T09:00:00Z", "last_used": "2026-05-19T12:30:00Z"}]

curl -X POST https://your-testhide-server/api/v3/api-tokens \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "GitHub Actions Bot"}'

import requests

r = requests.post(
    "https://your-testhide-server/api/v3/api-tokens",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={"name": "GitHub Actions Bot"}
)
secret = r.json()["secret"]  # save this — shown only once

{"id": 5, "name": "GitHub Actions Bot", "secret": "th_live_xK9p...Q2rZ", "created_at": "2026-05-20T10:00:00Z"}

curl -X DELETE https://your-testhide-server/api/v3/api-tokens/5 \
  -H "Authorization: Bearer $TOKEN"

import requests

requests.delete(
    "https://your-testhide-server/api/v3/api-tokens/5",
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

(no content)

Users

Manage platform users. Admin role required for create/update operations.

curl "https://your-testhide-server/api/v3/users?page=1&limit=20" \
  -H "Authorization: Bearer $TOKEN"

import requests

r = requests.get(
    "https://your-testhide-server/api/v3/users",
    params={"page": 1, "limit": 20},
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)
print(r.json())

{"total": 84, "results": [{"id": 1, "email": "alice@corp.com", "role": "admin", "active": true}]}

curl -X POST https://your-testhide-server/api/v3/users \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"email": "bob@corp.com", "role": "developer", "project_ids": [1, 3]}'

import requests

r = requests.post(
    "https://your-testhide-server/api/v3/users",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={"email": "bob@corp.com", "role": "developer", "project_ids": [1, 3]}
)
print(r.json()["id"])

{"id": 88, "email": "bob@corp.com", "role": "developer", "invited_at": "2026-05-20T10:00:00Z"}

curl https://your-testhide-server/api/v3/users/88 \
  -H "Authorization: Bearer $TOKEN"

import requests

r = requests.get(
    "https://your-testhide-server/api/v3/users/88",
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)
print(r.json())

{"id": 88, "email": "bob@corp.com", "role": "developer", "active": true, "projects": [1, 3], "last_login": "2026-05-19T08:00:00Z"}

curl -X PUT https://your-testhide-server/api/v3/users/88 \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"role": "admin", "project_ids": [1, 3, 7]}'

import requests

r = requests.put(
    "https://your-testhide-server/api/v3/users/88",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={"role": "admin", "project_ids": [1, 3, 7]}
)
print(r.json())

{"id": 88, "email": "bob@corp.com", "role": "admin", "project_ids": [1, 3, 7]}

curl https://your-testhide-server/api/v3/users/88/permissions \
  -H "Authorization: Bearer $TOKEN"

import requests

r = requests.get(
    "https://your-testhide-server/api/v3/users/88/permissions",
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)
print(r.json())

{"user_id": 88, "can_start_builds": true, "can_manage_nodes": false, "is_admin": false, "projects": {"1": "write", "3": "read"}}

Projects

Projects group jobs, builds, and settings into isolated workspaces.

curl https://your-testhide-server/api/v3/projects \
  -H "Authorization: Bearer $TOKEN"

import requests
r = requests.get("https://your-testhide-server/api/v3/projects", headers={"Authorization": "Bearer YOUR_TOKEN"})
for p in r.json(): print(p["id"], p["name"])

[{"id": 1, "name": "Backend API", "slug": "backend-api", "jobs_count": 12}]

curl -X POST https://your-testhide-server/api/v3/projects \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "Mobile App", "description": "iOS/Android tests", "scm_endpoint_id": 2}'

import requests
r = requests.post(
    "https://your-testhide-server/api/v3/projects",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={"name": "Mobile App", "description": "iOS/Android tests", "scm_endpoint_id": 2}
)
print(r.json()["id"])

{"id": 12, "name": "Mobile App", "slug": "mobile-app", "webhook_secret": "wh_sec_abc..."}

curl https://your-testhide-server/api/v3/projects/12 \
  -H "Authorization: Bearer $TOKEN"

import requests
r = requests.get("https://your-testhide-server/api/v3/projects/12", headers={"Authorization": "Bearer YOUR_TOKEN"})
print(r.json())

{"id": 12, "name": "Mobile App", "jobs_count": 8, "last_build_at": "2026-05-19T18:00:00Z", "webhook_url": "https://your-testhide-server/webhook/proj/12"}

curl -X PUT https://your-testhide-server/api/v3/projects/12 \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "Mobile App v2", "scm_endpoint_id": 3}'

import requests
r = requests.put("https://your-testhide-server/api/v3/projects/12", headers={"Authorization": "Bearer YOUR_TOKEN"}, json={"name": "Mobile App v2", "scm_endpoint_id": 3})
print(r.json())

{"id": 12, "name": "Mobile App v2", "scm_endpoint_id": 3}

curl -X POST https://your-testhide-server/api/v3/projects/12/rotate-webhook-secret \
  -H "Authorization: Bearer $TOKEN"

import requests
r = requests.post("https://your-testhide-server/api/v3/projects/12/rotate-webhook-secret", headers={"Authorization": "Bearer YOUR_TOKEN"})
print(r.json()["webhook_secret"])

{"webhook_secret": "wh_sec_xK9p...Q2rZ"}

Jobs

Jobs define the test pipeline — what runs, on which nodes, with which configuration. A Job produces Builds when triggered.

curl "https://your-testhide-server/api/v3/jobs?project_id=1&enabled=true" \
  -H "Authorization: Bearer $TOKEN"

import requests
r = requests.get("https://your-testhide-server/api/v3/jobs", params={"project_id": 1, "enabled": "true"}, headers={"Authorization": "Bearer YOUR_TOKEN"})
for job in r.json(): print(job["id"], job["name"])

[{"id": 55, "name": "Regression Suite", "project_id": 1, "node_pool_id": 2, "enabled": true}]

curl -X POST https://your-testhide-server/api/v3/jobs \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Smoke Tests",
    "project_id": 1,
    "node_pool_id": 2,
    "timeout_minutes": 30,
    "max_parallel": 3
  }'

import requests
r = requests.post(
    "https://your-testhide-server/api/v3/jobs",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={"name": "Smoke Tests", "project_id": 1, "node_pool_id": 2, "timeout_minutes": 30, "max_parallel": 3}
)
print(r.json()["id"])

{"id": 56, "name": "Smoke Tests", "project_id": 1, "enabled": true}

curl https://your-testhide-server/api/v3/jobs/55 \
  -H "Authorization: Bearer $TOKEN"

import requests
r = requests.get("https://your-testhide-server/api/v3/jobs/55", headers={"Authorization": "Bearer YOUR_TOKEN"})
print(r.json())

{"id": 55, "name": "Regression Suite", "node_pool_id": 2, "last_build_id": 1024, "last_build_status": "passed"}

curl -X PUT https://your-testhide-server/api/v3/jobs/55 \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false, "timeout_minutes": 90}'

import requests
r = requests.put("https://your-testhide-server/api/v3/jobs/55", headers={"Authorization": "Bearer YOUR_TOKEN"}, json={"enabled": False, "timeout_minutes": 90})
print(r.json())

{"id": 55, "enabled": false, "timeout_minutes": 90}

curl -X DELETE https://your-testhide-server/api/v3/jobs/55 \
  -H "Authorization: Bearer $TOKEN"

import requests
requests.delete("https://your-testhide-server/api/v3/jobs/55", headers={"Authorization": "Bearer YOUR_TOKEN"})

(no content)

curl https://your-testhide-server/api/v3/jobs/55/configuration \
  -H "Authorization: Bearer $TOKEN"

import requests
r = requests.get("https://your-testhide-server/api/v3/jobs/55/configuration", headers={"Authorization": "Bearer YOUR_TOKEN"})
print(r.json())

{"setup": ["pip install -r requirements.txt"], "command": "pytest tests/ -x", "env": {"ENV": "staging"}, "artifacts": ["reports/*.xml"]}

curl -X PUT https://your-testhide-server/api/v3/jobs/55/configuration \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "setup": ["pip install -r requirements.txt"],
    "command": "pytest tests/ --junitxml=report.xml",
    "env": {"ENV": "staging", "DB_HOST": "db.internal"},
    "artifacts": ["report.xml", "logs/*.log"]
  }'

import requests
r = requests.put(
    "https://your-testhide-server/api/v3/jobs/55/configuration",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={
        "setup": ["pip install -r requirements.txt"],
        "command": "pytest tests/ --junitxml=report.xml",
        "env": {"ENV": "staging", "DB_HOST": "db.internal"},
        "artifacts": ["report.xml", "logs/*.log"]
    }
)
print(r.status_code)

{"job_id": 55, "updated_at": "2026-05-20T10:15:00Z"}

curl -X POST https://your-testhide-server/api/v3/jobs/55/export \
  -H "Authorization: Bearer $TOKEN" \
  -o job_55_export.json

import requests, json
r = requests.post("https://your-testhide-server/api/v3/jobs/55/export", headers={"Authorization": "Bearer YOUR_TOKEN"})
with open("job_55_export.json", "w") as f:
    json.dump(r.json(), f, indent=2)

{"export_version": "1", "job": {"name": "Regression Suite", ...}, "configuration": {...}}

curl -X POST https://your-testhide-server/api/v3/jobs/import \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"project_id": 3, "bundle": {...}}'

import requests, json
bundle = json.load(open("job_55_export.json"))
r = requests.post(
    "https://your-testhide-server/api/v3/jobs/import",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={"project_id": 3, "bundle": bundle}
)
print("New job ID:", r.json()["id"])

{"id": 99, "name": "Regression Suite (imported)", "project_id": 3}

Builds

Builds are triggered executions of a Job. Each build runs the pipeline on CI agents and collects test results and artifacts.

curl -X POST https://your-testhide-server/api/v3/builds/start \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "job_id": 55,
    "branch": "main",
    "commit_sha": "a1b2c3d4",
    "env": {"RUN_SLOW": "true"}
  }'

import requests
r = requests.post(
    "https://your-testhide-server/api/v3/builds/start",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={"job_id": 55, "branch": "main", "commit_sha": "a1b2c3d4", "env": {"RUN_SLOW": "true"}}
)
build_id = r.json()["id"]
print("Build started:", build_id)

{"id": 1025, "job_id": 55, "status": "queued", "branch": "main", "commit_sha": "a1b2c3d4", "created_at": "2026-05-20T10:00:00Z"}

curl "https://your-testhide-server/api/v3/builds?job_id=55&status=failed&limit=10" \
  -H "Authorization: Bearer $TOKEN"

import requests
r = requests.get(
    "https://your-testhide-server/api/v3/builds",
    params={"job_id": 55, "status": "failed", "limit": 10},
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)
print(r.json())

{"total": 3, "results": [{"id": 1024, "status": "failed", "branch": "main", "duration_s": 312, "failed_tests": 7}]}

curl https://your-testhide-server/api/v3/builds/1024 \
  -H "Authorization: Bearer $TOKEN"

import requests
r = requests.get("https://your-testhide-server/api/v3/builds/1024", headers={"Authorization": "Bearer YOUR_TOKEN"})
print(r.json())

{"id": 1024, "job_id": 55, "status": "passed", "duration_s": 287, "total_tests": 412, "passed": 410, "failed": 0, "skipped": 2, "node_id": 7}

curl -X POST https://your-testhide-server/api/v3/builds/1024/restart \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"failed_only": true}'

import requests
r = requests.post(
    "https://your-testhide-server/api/v3/builds/1024/restart",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={"failed_only": True}
)
print("New build ID:", r.json()["id"])

{"id": 1026, "status": "queued", "restarted_from": 1024}

curl -X POST https://your-testhide-server/api/v3/builds/1025/stop \
  -H "Authorization: Bearer $TOKEN"

import requests
requests.post("https://your-testhide-server/api/v3/builds/1025/stop", headers={"Authorization": "Bearer YOUR_TOKEN"})

{"id": 1025, "status": "aborted"}

curl -X DELETE https://your-testhide-server/api/v3/builds/1020 \
  -H "Authorization: Bearer $TOKEN"

import requests
requests.delete("https://your-testhide-server/api/v3/builds/1020", headers={"Authorization": "Bearer YOUR_TOKEN"})

(no content)

curl -X POST https://your-testhide-server/api/v3/builds/1025/push_to_top \
  -H "Authorization: Bearer $TOKEN"

import requests
requests.post("https://your-testhide-server/api/v3/builds/1025/push_to_top", headers={"Authorization": "Bearer YOUR_TOKEN"})

{"id": 1025, "queue_position": 1}

curl -X POST https://your-testhide-server/api/v3/builds/1024/keep-forever \
  -H "Authorization: Bearer $TOKEN"

import requests
requests.post("https://your-testhide-server/api/v3/builds/1024/keep-forever", headers={"Authorization": "Bearer YOUR_TOKEN"})

{"id": 1024, "keep_forever": true}

curl "https://your-testhide-server/api/v3/builds/1024/pipeline_logs?offset=0&limit=200" \
  -H "Authorization: Bearer $TOKEN"

import requests
r = requests.get(
    "https://your-testhide-server/api/v3/builds/1024/pipeline_logs",
    params={"offset": 0, "limit": 200},
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)
for line in r.json()["lines"]: print(line)

{"total_lines": 1240, "lines": ["[10:00:01] Setup started", "[10:00:03] pip install OK", "..."]}

curl -N https://your-testhide-server/api/v3/builds/1025/stream \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: text/event-stream"

import requests

with requests.get(
    "https://your-testhide-server/api/v3/builds/1025/stream",
    headers={"Authorization": "Bearer YOUR_TOKEN", "Accept": "text/event-stream"},
    stream=True
) as r:
    for line in r.iter_lines():
        if line:
            print(line.decode())

data: {"type": "log", "data": "[10:00:05] PASSED tests/test_api.py::test_login"}
data: {"type": "status", "data": "passed"}
data: {"type": "done"}

curl -X POST https://your-testhide-server/api/v3/builds/1024/comment \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"text": "Flaky DB connection — infra issue, not code."}'

import requests
r = requests.post(
    "https://your-testhide-server/api/v3/builds/1024/comment",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={"text": "Flaky DB connection — infra issue, not code."}
)
print(r.json())

{"id": 88, "build_id": 1024, "text": "Flaky DB connection...", "author": "alice@corp.com"}

curl -X POST https://your-testhide-server/api/v3/builds/1024/retry-flaky \
  -H "Authorization: Bearer $TOKEN"

import requests
r = requests.post("https://your-testhide-server/api/v3/builds/1024/retry-flaky", headers={"Authorization": "Bearer YOUR_TOKEN"})
print("New build:", r.json()["id"], "| Retrying", r.json()["test_count"], "tests")

{"id": 1027, "status": "queued", "test_count": 4, "restarted_from": 1024}