Introduction

TestHide is an agentic CI/CD platform optimized for automated testing. Unlike general-purpose CI tools, TestHide treats test results as first-class citizens, using AI to analyze failures, detect flakiness, and correlate errors across builds.

Smart Analysis

Auto-classification of failures using ML (Stack traces, Screenshots).

Resilient Agents

Cross-platform agents (Windows/Linux/macOS) with WebSocket persistence.

Matrix Builds

Run tests across thousands of configurations in parallel.

Flakiness Detection

Statistical tracking of unstable tests with quarantine workflows.

System Architecture

TestHide consists of three main components that communicate via WebSocket and REST APIs.

Backend Server

Python/aiohttp REST API + WebSocket server. Handles job scheduling, build orchestration, AI analysis, and database operations.

Port: 8000 (HTTP/WS)

Frontend UI

Angular SPA with IDE-like interface. Provides job management, build monitoring, test analytics, and agent control.

Port: 4200 (nginx)

Agents (testhide_client)

Cross-platform C# executables. Connect via WebSocket, execute builds, collect results, and report back in real-time.

Win/Linux/macOS

Data Flow

  1. Job Trigger: User or webhook initiates a build via Frontend/API
  2. Queue: Backend adds build to queue and notifies available agents
  3. Execution: Agent claims build, executes steps, streams logs via WebSocket
  4. Analysis: Backend receives results, AI models analyze failures
  5. Storage: Results persisted to MongoDB, artifacts to file storage

Quick Start

Deploy TestHide instantly using Docker Compose. This configuration includes all required services.

docker-compose.yaml
version: '3.8'
name: testhide

services:
  # ==========================================
  # FRONTEND (Nginx + Angular)
  # ==========================================
  frontend:
    container_name: testhide_frontend
    image: thuesdays/testhide-frontend:latest
    restart: always
    ports:
      - "80:80"
      - "443:443"
      - "7771:7771"
    env_file:
      - .env
    environment:
      - CERT_FILE=${CERT_FILE}
      - CERT_KEY=${CERT_KEY}
    volumes:
      - ./ssl:/etc/ssl:ro
      - ./nginx/api.conf:/etc/nginx/conf.d/api.conf:ro
      - testhide_static_data:/usr/share/nginx/html/static:ro
    depends_on:
      - backend
      - ai-api
    networks:
      - testhide-net
    deploy:
      resources:
        limits:
          cpus: "0.25"
          memory: 256m

  # ==========================================
  # BACKEND (Python API)
  # ==========================================
  backend:
    image: thuesdays/testhide-backend:latest
    restart: unless-stopped
    env_file:
      - .env
    environment:
      - LOAD_AI_MODELS=false
      - RUN_AI_WORKER=false
      - GUNICORN_WORKERS=2
      - COMPOSE_PROJECT_NAME=testhide
    volumes:
      - testhide_static_data:/app/static
      - ./releases:/app/releases
      - ./monitoring_scripts:/app/monitoring_scripts
      - ./gunicorn.conf.py:/app/gunicorn.conf.py:ro
      - ./ssl:/etc/ssl:ro
      - ./sandbox_data:/app/sandbox_data
      - /var/run/docker.sock:/var/run/docker.sock
      - ./docker-compose.prod.yaml:/app/docker-compose.yaml:ro
    depends_on:
      mongo:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - testhide-net
    healthcheck:
      disable: true
    mem_limit: 1.5g
    memswap_limit: 1.5g
    deploy:
      replicas: 2
      resources:
        limits:
          cpus: "2.0"
          memory: 1.5g
        reservations:
          cpus: "0.5"
          memory: 768m

  # ==========================================
  # AI API (HTTP only — serves /api/v3/ai/)
  # ==========================================
  ai-api:
    container_name: testhide_ai_api
    image: thuesdays/testhide-backend:latest
    restart: unless-stopped
    env_file:
      - .env
    environment:
      - LOAD_AI_MODELS=true
      - RUN_AI_WORKER=false
      - GUNICORN_WORKERS=2
    volumes:
      - testhide_static_data:/app/static
      - ./sandbox_data:/app/sandbox_data
      - ./releases:/app/releases
      - ./monitoring_scripts:/app/monitoring_scripts
      - ./gunicorn.conf.py:/app/gunicorn.conf.py:ro
      - ./ssl:/etc/ssl:ro
    depends_on:
      mongo:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - testhide-net
    healthcheck:
      disable: true
    mem_limit: 5g
    memswap_limit: 5g
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 5g
        reservations:
          cpus: "0.5"
          memory: 3g

  # ==========================================
  # AI WORKER (Background: training, vectorization)
  # ==========================================
  ai-worker:
    container_name: testhide_ai_worker
    image: thuesdays/testhide-backend:latest
    restart: unless-stopped
    env_file:
      - .env
    environment:
      - LOAD_AI_MODELS=true
      - RUN_AI_WORKER=true
      - GUNICORN_WORKERS=1
    volumes:
      - testhide_static_data:/app/static
      - ./sandbox_data:/app/sandbox_data
      - ./releases:/app/releases
      - ./monitoring_scripts:/app/monitoring_scripts
      - ./gunicorn.conf.py:/app/gunicorn.conf.py:ro
      - ./ssl:/etc/ssl:ro
    depends_on:
      mongo:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      testhide-net:
        aliases:
          - ai-worker
    healthcheck:
      disable: true
    mem_limit: 7g
    memswap_limit: 7g
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 7g
        reservations:
          cpus: "0.5"
          memory: 4g

  # ==========================================
  # DATABASES
  # ==========================================
  mongo:
    container_name: testhide_mongo
    image: 'mongo:8.2.3'
    restart: always
    command: [ "mongod", "--auth", "--wiredTigerCacheSizeGB", "2" ]
    environment:
      - MONGO_INITDB_ROOT_USERNAME=${MONGO_USER}
      - MONGO_INITDB_ROOT_PASSWORD=${MONGO_PASS}
    ports:
      - "27017:27017"
    volumes:
      - ${MONGO_DATA_PATH}:/data/db
    networks:
      - testhide-net
    healthcheck:
      test: [ "CMD", "mongosh", "--norc", "--quiet", "-u", "${MONGO_USER}", "-p", "${MONGO_PASS}", "--authenticationDatabase", "admin", "--eval", "db.adminCommand('ping')" ]
      interval: 30s
      timeout: 5s
      retries: 5
    deploy:
      resources:
        limits:
          cpus: "1.5"
          memory: 3g
        reservations:
          cpus: "0.5"
          memory: 1.5g

  redis:
    container_name: testhide_redis
    image: redis:7-alpine
    restart: always
    command: [ "redis-server", "--maxmemory", "512mb", "--maxmemory-policy", "allkeys-lru", "--appendonly", "no", "--requirepass", "${REDIS_PASSWORD}" ]
    networks:
      - testhide-net
    healthcheck:
      test: [ "CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping" ]
      interval: 10s
      timeout: 3s
      retries: 5
    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: 768m

  redisinsight:
    container_name: testhide_redisinsight
    image: redis/redisinsight:latest
    restart: always
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - testhide-net
    deploy:
      resources:
        limits:
          cpus: "0.25"
          memory: 256m

# ==========================================
# VOLUMES & NETWORKS
# ==========================================
volumes:
  testhide_static_data:
    name: testhide_static_data

networks:
  testhide-net:
    name: testhide-net

Deployment Steps

  1. Create a .env file with your secrets:
    .env
    MONGO_PASSWORD=your_secure_password
    JWT_SECRET=your_super_secret_jwt_key
  2. Run docker-compose up -d
  3. Access the frontend at http://localhost:4200
  4. Default login: admin / admin

WebSocket URL for Agents: Configure your agents to connect to ws://your-server:8000 (or wss:// with SSL).

Installation & Configuration

Environment Variables

VariableDefaultDescription
MONGO_HOSTlocalhostMongoDB server hostname.
MONGO_PORT27017MongoDB server port.
MONGO_DB_NAMEtesthide_databaseMongoDB database name.
MONGO_USERtesthide_userMongoDB authentication username.
MONGO_PASS(empty)MongoDB authentication password.
JWT_SECRET<default>Secret key for signing JWT auth tokens. Change in production!
TPS_HEARTBEAT_TIMEOUT_SEC300Agent timeout (seconds) before marking as offline.
CLIENT_MAX_SIZE_MB4096Max upload size for artifacts (in MB).
USE_SSLfalseEnable HTTPS/WSS. Requires CERT_KEY and CERT_FILE.
CERT_KEY / CERT_FILE(none)Paths to SSL certificate files for HTTPS.
CORS_ORIGINS*Comma-separated list of allowed CORS origins.
DEBUGfalseEnable debug mode (verbose logging).
FRONTEND_PUBLIC_URL(none)Public URL of the frontend for email links.

AI Configuration Variables

VariableDefaultDescription
Variable ClassificationNameDefaultDescription
Global Settings
Master SwitchAI_ENABLE_AI_ASSISTtrueGlobal kill-switch for all AI background workers.
Feature Toggles (Enable/Disable Models)
Feature ToggleAI_FEATURE_AI_ROOT_CAUSE_ENABLEDtrueEnable Root Cause analysis.
Feature ToggleAI_FEATURE_AI_FLAKINESS_ENABLEDtrueEnable Flakiness prediction.
Feature ToggleAI_FEATURE_AI_VISUAL_ENABLEDtrueEnable Visual Diff analysis.
Feature ToggleAI_FEATURE_AI_OOD_ENABLEDtrueEnable Out-of-Distribution detection.
Feature ToggleAI_FEATURE_AI_LOG_SIGNATURES_ENABLEDtrueEnable Log Signature mining.
Feature ToggleAI_FEATURE_AI_RETRIEVER_ENABLEDtrueEnable Failure Retriever (Search).
Feature ToggleAI_FEATURE_AI_EMERGING_ISSUES_ENABLEDtrueEnable Emerging Issues detection.
Feature ToggleAI_FEATURE_AI_BUG_LINKER_ENABLEDtrueEnable auto-linking to Jira/Issues.
Hardware & Performance
InfrastructureAI_CPU_LIMIT_CORES2Max CPU cores per worker process.
InfrastructureAI_OMP_NUM_THREADS1OpenMP threads (set 1 to avoid contention).
InfrastructureAI_MKL_NUM_THREADS1MKL threads (set 1 to avoid contention).
InfrastructureAI_HF_DATASETS_NUM_PROC2Parallel processes for HuggingFace datasets.
InfrastructureAI_TOKENIZERS_PARALLELISMfalseEnable parallelism in tokenizers.
Update Thresholds
ThresholdsAI_FAST_UPDATE_THRESHOLD500Samples to trigger light updates (FAISS/OOD).
ThresholdsAI_HEAVY_UPDATE_THRESHOLD1000Samples to trigger full model fine-tuning.
ThresholdsAI_HEAVY_MIN_INTERVAL_SEC18000Min seconds (5h) between heavy training runs.
Root Cause Model
Root CauseAI_RC_BATCH_SIZE8Training batch size.
Root CauseAI_RC_EPOCHS3Training epochs.
Root CauseAI_RC_MODEL_NAMEbert-base-uncasedBase HuggingFace model architecture.
Root CauseAI_RC_MAX_LEN256Max sequence length for text tokens.
Root CauseAI_RC_CLEANLAB_BACKENDtorchBackend for label noise correction.
Other Models
OtherAI_FLAKINESS_EPOCHS100Epochs for flakiness predictor training.
OtherAI_VISUAL_EPOCHS5Epochs for visual diff model.
OtherAI_OOD_AE_EPOCHS10Epochs for OOD Autoencoder.
OtherAI_RETRIEVER_EPOCHS200Epochs for Contrastive Retriever.
OtherAI_LOGSIGN_TOP_EXAMPLES3Number of examples to keep per log signature.

Agent Deployment (testhide_client)

Agents connect to the backend via WebSocket. The agent is a cross-platform C# executable that auto-updates and persists on the system.

1. Configure the Agent

Set the backend WebSocket URL using the config command:

Configure Agent (Windows)
# 1. Configure
testhide.exe config --url wss://testhide.company.com --instance-id my-server-01

# 2. Run
testhide.exe run

# View config
testhide.exe config --show

2. Config Command Options

FlagShortDescription
--url-uWebSocket server URL (ws:// or wss://)
--instance-id-iInstance ID for licensing/identification
--show-sDisplay current configuration
--help-hShow help message

3. Start the Agent

Run Agent
# Start the agent using saved configuration
testhide.exe run

# Verify connection in the logs:
# [INFO] Successfully connected to wss://...
# [INFO] Registered with ID: ...

4. config.json Structure

The agent stores configuration in config.json next to the executable:

config.json
{
  "WebSocketUrl": "wss://testhide.company.com",
  "InstanceId": "my-server-id",
  "UpdateCheckIntervalSeconds": 3600,
  "MinDiskSpaceGbForCleanup": 10.0,
  "LogCleanupDays": 5,
  "MaxDisconnectionsAllowed": 1,
  "GitCheckoutTimeoutSeconds": 600,
  "IncludeTestCasesInSnapshots": false,
  "DirectoriesToClean": [],
  "StrayProcessNamesToKill": [],
  "PythonEnv": {
    "AutoInstallPython": false,
    "AutoInstallPyenv": false,
    "AutoInstallRequirements": true,
    "FallbackToVirtualenv": true
  }
}

Requirements: The agent machine should have Python 3.9+ and Git installed for most build steps. The agent supports Windows, Linux, and macOS.

Features: Remote Desktop access, system monitoring, auto-updates, label-based routing, concurrent build execution, and local AI analysis (when enabled).

Pipelines & Jobs

TestHide jobs define the build logic, triggers, and execution environment. You can manage them via the IDE UI or by placing YAML files in your repository.

UI Management

Use our 9-tab integrated editor for visual configuration. Includes Source Management, AI Analysis triggers, Matrix Config, and Build Environment overrides.

YAML Discovery

Commit testhide.yaml to your repository. TestHide automatically discovers and updates job configurations based on branch changes.

Build Steps

A job consists of a list of steps executed sequentially on the agent. Each step type provides specialized parsing and execution logic.

Windows Batch

Standard .bat/.cmd commands.

PowerShell

Execute .ps1 scripts on Windows agents.

Bash

Shell scripting for Linux/macOS.

Python

Run .py scripts in isolated venv.

Docker

Containerized build execution.

Test Provider

PyTest, JUnit, NUnit parsing.

Copy Artifacts

Manage build outputs.

Isolated Execution: Python steps can automatically manage pyenv and requirements.txt installations if enabled in the agent config.

Matrix Builds

Matrix builds allow you to run the same job across multiple configurations in parallel. TestHide generates a child build for each combination of axis values (Cartesian product).

Axis Types

You can define up to 2 axes per job:

Nodes Axis (by name/label)

Select specific agents or labels from a tree view. Each selected node becomes a matrix cell. Builds are routed to matching agents.

  • • Expandable tree for node/label selection
  • • Routes builds to matching agents
  • • Default value for single runs
Custom Axis

Define arbitrary values (one per line). Use for OS versions, browsers, environments, or any configuration dimension.

  • • Free-form text values
  • • Exposed as environment variables
  • • Available via %PARAM_NAME% substitution

Configuration Example

Matrix Configuration (job_parameters)
job_parameters:
  # Custom Axis: OS versions
  - type: matrix
    matrix_type: custom_axis
    parameter_name: OS
    values:
      - windows-2022
      - ubuntu-22.04
      - macos-14
  
  # Nodes Axis: Run on specific agents
  - type: matrix
    matrix_type: nodes
    parameter_name: AGENT_LABEL
    values:
      - gpu-runner
      - standard-runner

# Result: 6 child builds (3 OS × 2 Agents)
# Child keys: "windows-2022-gpu-runner", "ubuntu-22.04-standard-runner", etc.

Generated Child Build Structure

For each combination, TestHide generates a child configuration with:

matrix_configuration (auto-generated)
{
  "windows-2022-gpu-runner": {
    "envs": [
      { "env_name": "OS", "env_value": "windows-2022" },
      { "env_name": "AGENT_LABEL", "env_value": "gpu-runner" }
    ],
    "job_parameters": [
      { "parameter_name": "OS", "defaultValue": "windows-2022" },
      { "parameter_name": "AGENT_LABEL", "defaultValue": "gpu-runner" }
    ],
    "label_expr": ["gpu-runner"]  // For agent routing
  },
  // ... more combinations
}

Using Matrix Values in Scripts

Build Script
# Environment variables are automatically set
echo "Running on OS: $OS"
echo "Agent Label: $AGENT_LABEL"

# Or use substitution syntax
echo "Testing on %OS% with %AGENT_LABEL%"

Parallel Execution: Matrix child builds run in parallel across available agents. The parent build waits for all children to complete. Parent duration ≈ max child duration (not sum).

Selective Restart: You can restart specific matrix children using a boolean selection mask, e.g., {"windows-2022-gpu-runner": true, "ubuntu-22.04-standard-runner": false}.

Artifacts & Reports

TestHide automatically captures and indexes build outputs, test results, and media attachments.

Test Results

Auto-parsing for JUnit, PyTest, and NUnit formats. TestHide tracks every test case history, execution time, and stability (flakiness).

Media & Snapshots

Store screenshots, videos, and HTML reports. AI analysis triggers automatically on new failure screenshots to group similar issues.

Configuring Artifacts

artifact-config.yaml
post_steps:
  # Capture JUnit reports for analytics
  - type: test_provider
    provider: pytest
    path: "**/reports/*.xml"
  
  # Upload logs and screenshots
  - type: copy_artifacts
    source: "logs/**/*.log"
    target: "build_logs"
  
  - type: copy_artifacts
    source: "screenshots/*.png"
    target: "failures"

AI Configuration

TestHide utilizes a suite of specialized machine learning models to analyze test results in real-time. These models are hosted on the backend and triggered automatically after build completion.

Root Cause

MultiModalClassifier: Classifies stack traces and logs into meaningful categories (e.g., Environment, Bug, Network).

Visual CNN

Uses a Convolutional Neural Network to cluster failures by visual similarity in screenshots.

Flakiness

Statistical model that predicts test stability based on historical transitions and execution patterns.

OOD Detection

TinyAutoencoder: Out-of-Distribution detector that identifies novel errors never seen before by the system.

Bug Linker

Uses text embeddings to correlate test failures with existing Jira issues or internal bug reports.

Model Release: Models are stored in the AI_MODELS_RELEASE_DIR and can be updated without restarting the main server. New samples are processed in batches of AI_FAST_THRESHOLD.

Permissions & Roles

TestHide implements a multi-tenant RBAC system with global roles and granular project-scoped permissions. Permissions are stored in the user profile as a list of strings and validated by the backend middleware.

System Roles

Role IDDescriptionCapability
userStandard corporate user.Access limited to explicitly assigned projects and basic read-only global views.
superuserSystem Administrator.Full bypass of all permission checks. Access to all projects, global settings, and user management.
service_accountInternal System Account.Used for automated agents and background workers with wide system permissions.

Project-Scoped Permissions

Project permissions follow a specific string pattern: PROJECT_{id}_{OP}. These are automatically granted to project creators and can be assigned to other users by Project Admins.

PROJECT_..._READ
View Only

Access to build history, test reports, artifacts, and configuration (read-only).

PROJECT_..._WRITE
Configuration

Ability to modify job settings, update SCM endpoints, and manage project metadata.

PROJECT_..._EXECUTE
Pipeline Control

Permission to trigger new builds, restart failed stages, and stop active pipelines.

PROJECT_..._DELETE
Data Removal

High-level access to permanently delete builds, reports, or the entire project.

Global Permission Categories

System-wide actions are governed by category-based strings (e.g., JOBS_READ, NODES_WRITE). Major categories include:

REPORTSJOBSPROJECTSBUILDSNODESUSERSAI_ASSISTCONFIGURATIONMONITORING

SSO & Authentication

Secure your instance with modern authentication providers.

Local JWT

Standard token-based auth with encrypted password storage. Managed via the admin panel.

OIDC / Google

Connect your corporate identity provider. Support for Google, GitLab, and custom OIDC.

Security: All API requests require a Bearer token. Agent communication is secured via a persistent WebSocket challenge.

Integrations

Git Flow

Auto-trigger on PR/Commit. GitHub, GitLab, Bitbucket supported.

Issue Tracking

Bilateral Jira integration. Links failed tests to Jira tickets.

ChatOps

Real-time status cards in Teams and Slack channels.

Git Providers

Connect your repository to automatically trigger builds. TestHide supports SSH and HTTPS access protocols.

source-config.yaml
source_management:
  type: git
  url: "git@github.com:company/repo.git"
  branch: "main"
  credentials_id: "github-ssh-key"
  poll_interval: "5m"

Jira Bug Linker

Enable AI-assisted bug tracking. TestHide can search for existing Jira issues that match the current failure signature or create new ones.

AI Insight: When enabled, TestHide adds a "Bug Search" button to every failed test report, querying your Jira instance for similar stack traces.

Notifications

Send detailed report cards to Microsoft Teams or Slack. Notifications can be triggered on success, failure, or stability change (flaky detection).

notifications.yaml
post_steps:
  - type: msteams_notification
    webhook_url: "https://outlook.office.com/webhook/..."
    events: [failed, flaky]
    template: "Build #{build_number} failed on {branch}"

Incoming Webhooks

Trigger build jobs from external systems via simple HTTP POST requests.

CURL Trigger
curl -X POST https://testhide.com/api/v3/job/{job_id}/trigger \
     -H "Authorization: Bearer <TOKEN>" \
     -d '{"branch": "feature/ui-fix", "parameters": {"DEBUG": "true"}}'

API Data Reference

Overview

The TestHide API is a RESTful interface that allows you to manage jobs, builds, and agents programmatically. All responses are returned in JSON format.

Full interactive documentation: /api/v3/documentation

Authentication

Authenticate your requests using the Authorization: Bearer <TOKEN> header.

Headers
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
Content-Type: application/json

Build Object Structure

GET /api/v3/build/{id}
{
  "id": "60d5ec...",
  "status": "failed",
  "build_number": 421,
  "branch": "main",
  "duration": 452,
  "started_at": "2024-03-20T10:30:00Z",
  "tests": {
    "passed": 142,
    "failed": 3,
    "skipped": 5,
    "flaky": 1
  },
  "tps": {
    "flaky_tests": ["test_login_timeout"],
    "new_failures": ["test_checkout_flow"],
    "probable_cause": "Network latency in secondary DC"
  }
}