Introduction

TestHide is an agentic CI/CD platform optimized for automated testing. Unlike general-purpose CI tools, TestHide treats test results as first-class citizens, using AI to analyze failures, detect flakiness, and correlate errors across builds.

Smart Analysis

Auto-classification of failures using ML (Stack traces, Screenshots).

Resilient Agents

Cross-platform agents (Windows/Linux/macOS) with WebSocket persistence.

Matrix Builds

Run tests across thousands of configurations in parallel.

Flakiness Detection

Statistical tracking of unstable tests with quarantine workflows.

System Architecture

TestHide consists of three main components that communicate via WebSocket and REST APIs.

Backend Server

Python/aiohttp REST API + WebSocket server. Handles job scheduling, build orchestration, AI analysis, and database operations.

Port: 8000 (HTTP/WS)

Frontend UI

Angular SPA with IDE-like interface. Provides job management, build monitoring, test analytics, and agent control.

Port: 4200 (nginx)

Agents (testhide_client)

Cross-platform C# executables. Connect via WebSocket, execute builds, collect results, and report back in real-time.

Win/Linux/macOS

Data Flow

  1. Job Trigger: User or webhook initiates a build via Frontend/API
  2. Queue: Backend adds build to queue and notifies available agents
  3. Execution: Agent claims build, executes steps, streams logs via WebSocket
  4. Analysis: Backend receives results, AI models analyze failures
  5. Storage: Results persisted to MongoDB, artifacts to file storage

Quick Start

Deploy TestHide instantly using Docker Compose. This configuration includes all required services.

docker-compose.yaml
version: "3.8"

services:
  # ========================================
  # MongoDB Database
  # ========================================
  mongo:
    image: mongo:6.0
    container_name: testhide-mongo
    restart: unless-stopped
    environment:
      MONGO_INITDB_DATABASE: testhide_database
      MONGO_INITDB_ROOT_USERNAME: testhide_user
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_PASSWORD:-your_secure_password}
    volumes:
      - mongo_data:/data/db
    ports:
      - "27017:27017"

  # ========================================
  # TestHide Backend (Python/aiohttp)
  # ========================================
  backend:
    image: testhide/server:latest
    container_name: testhide-backend
    restart: unless-stopped
    ports:
      - "8000:8000"
    environment:
      # MongoDB Connection
      MONGO_HOST: mongo
      MONGO_PORT: "27017"
      MONGO_DB_NAME: testhide_database
      MONGO_USER: testhide_user
      MONGO_PASS: ${MONGO_PASSWORD:-your_secure_password}
      
      # Security
      JWT_SECRET: ${JWT_SECRET:-change_this_in_production}
      
      # Agent Configuration
      TPS_HEARTBEAT_TIMEOUT_SEC: "300"
      CLIENT_MAX_SIZE_MB: "4096"
      
      # AI Configuration (optional)
      AI_FORCE_CPU: "1"
      AI_ASSIST_LOAD_WORKERS: "3"
      
      # Frontend URL for emails
      FRONTEND_PUBLIC_URL: http://localhost:4200
    depends_on:
      - mongo
    volumes:
      - backend_data:/app/static
      - ai_models:/app/releases

  # ========================================
  # TestHide Frontend (Angular)
  # ========================================
  frontend:
    image: testhide/ui:latest
    container_name: testhide-frontend
    restart: unless-stopped
    ports:
      - "4200:80"
    environment:
      API_URL: http://backend:8000
    depends_on:
      - backend

volumes:
  mongo_data:
  backend_data:
  ai_models:

Deployment Steps

  1. Create a .env file with your secrets:
    .env
    MONGO_PASSWORD=your_secure_password
JWT_SECRET=your_super_secret_jwt_key
  2. Run docker-compose up -d
  3. Access the frontend at http://localhost:4200
  4. Default login: admin / admin

WebSocket URL for Agents: Configure your agents to connect to ws://your-server:8000 (or wss:// with SSL).

Installation & Configuration

Environment Variables

VariableDefaultDescription
MONGO_HOSTlocalhostMongoDB server hostname.
MONGO_PORT27017MongoDB server port.
MONGO_DB_NAMEtesthide_databaseMongoDB database name.
MONGO_USERtesthide_userMongoDB authentication username.
MONGO_PASS(empty)MongoDB authentication password.
JWT_SECRET<default>Secret key for signing JWT auth tokens. Change in production!
TPS_HEARTBEAT_TIMEOUT_SEC300Agent timeout (seconds) before marking as offline.
CLIENT_MAX_SIZE_MB4096Max upload size for artifacts (in MB).
USE_SSLfalseEnable HTTPS/WSS. Requires CERT_KEY and CERT_FILE.
CERT_KEY / CERT_FILE(none)Paths to SSL certificate files for HTTPS.
CORS_ORIGINS*Comma-separated list of allowed CORS origins.
DEBUGfalseEnable debug mode (verbose logging).
FRONTEND_PUBLIC_URL(none)Public URL of the frontend for email links.

AI Configuration Variables

VariableDefaultDescription
AI_FORCE_CPU1Force CPU inference (set 0 for GPU).
AI_ASSIST_LOAD_WORKERS3Number of parallel AI worker threads.
AI_FAST_THRESHOLD500Samples count to trigger FAISS/OOD updates.
JIRA_URL / JIRA_USERNAME / JIRA_PASSWORD(none)Jira integration credentials for Bug Linker AI.
HF_TOKEN(none)HuggingFace token for model downloads.

Agent Deployment (testhide_client)

Agents connect to the backend via WebSocket. The agent is a cross-platform C# executable that auto-updates and persists on the system.

1. Configure the Agent

Set the backend WebSocket URL using the config command:

Configure Agent (Windows)
# Set WebSocket URL
testhide.exe config --url ws://your-server:8000

# Or with SSL and instance ID
testhide.exe config --url wss://testhide.company.com --instance-id my-server-id

# View current configuration
testhide.exe config --show

2. Config Command Options

FlagShortDescription
--url-uWebSocket server URL (ws:// or wss://)
--instance-id-iInstance ID for licensing/identification
--show-sDisplay current configuration
--help-hShow help message

3. Start the Agent

Run Agent
# Simply run the executable (uses saved config)
testhide.exe

# Agent will connect to the configured WebSocket URL
# and register with the server automatically

4. config.json Structure

The agent stores configuration in config.json next to the executable:

config.json
{
  "WebSocketUrl": "wss://testhide.company.com",
  "InstanceId": "my-server-id",
  "UpdateCheckIntervalSeconds": 3600,
  "MinDiskSpaceGbForCleanup": 10.0,
  "LogCleanupDays": 5,
  "MaxDisconnectionsAllowed": 1,
  "GitCheckoutTimeoutSeconds": 600,
  "IncludeTestCasesInSnapshots": false,
  "DirectoriesToClean": [],
  "StrayProcessNamesToKill": [],
  "PythonEnv": {
    "AutoInstallPython": false,
    "AutoInstallPyenv": false,
    "AutoInstallRequirements": true,
    "FallbackToVirtualenv": true
  }
}

Requirements: The agent machine should have Python 3.9+ and Git installed for most build steps. The agent supports Windows, Linux, and macOS.

Features: Remote Desktop access, system monitoring, auto-updates, label-based routing, concurrent build execution, and local AI analysis (when enabled).

Pipelines & Jobs

TestHide jobs define the build logic, triggers, and execution environment. You can manage them via the IDE UI or by placing YAML files in your repository.

UI Management

Use our 9-tab integrated editor for visual configuration. Includes Source Management, AI Analysis triggers, Matrix Config, and Build Environment overrides.

YAML Discovery

Commit testhide.yaml to your repository. TestHide automatically discovers and updates job configurations based on branch changes.

Build Steps

A job consists of a list of steps executed sequentially on the agent. Each step type provides specialized parsing and execution logic.

Windows Batch

Standard .bat/.cmd commands.

PowerShell

Execute .ps1 scripts on Windows agents.

Bash

Shell scripting for Linux/macOS.

Python

Run .py scripts in isolated venv.

Docker

Containerized build execution.

Test Provider

PyTest, JUnit, NUnit parsing.

Copy Artifacts

Manage build outputs.

Isolated Execution: Python steps can automatically manage pyenv and requirements.txt installations if enabled in the agent config.

Matrix Builds

Matrix builds allow you to run the same job across multiple configurations in parallel. TestHide generates a child build for each combination of axis values (Cartesian product).

Axis Types

You can define up to 2 axes per job:

Nodes Axis (by name/label)

Select specific agents or labels from a tree view. Each selected node becomes a matrix cell. Builds are routed to matching agents.

  • • Expandable tree for node/label selection
  • • Routes builds to matching agents
  • • Default value for single runs
Custom Axis

Define arbitrary values (one per line). Use for OS versions, browsers, environments, or any configuration dimension.

  • • Free-form text values
  • • Exposed as environment variables
  • • Available via %PARAM_NAME% substitution

Configuration Example

Matrix Configuration (job_parameters)
job_parameters:
  # Custom Axis: OS versions
  - type: matrix
    matrix_type: custom_axis
    parameter_name: OS
    values:
      - windows-2022
      - ubuntu-22.04
      - macos-14
  
  # Nodes Axis: Run on specific agents
  - type: matrix
    matrix_type: nodes
    parameter_name: AGENT_LABEL
    values:
      - gpu-runner
      - standard-runner

# Result: 6 child builds (3 OS × 2 Agents)
# Child keys: "windows-2022-gpu-runner", "ubuntu-22.04-standard-runner", etc.

Generated Child Build Structure

For each combination, TestHide generates a child configuration with:

matrix_configuration (auto-generated)
{
  "windows-2022-gpu-runner": {
    "envs": [
      { "env_name": "OS", "env_value": "windows-2022" },
      { "env_name": "AGENT_LABEL", "env_value": "gpu-runner" }
    ],
    "job_parameters": [
      { "parameter_name": "OS", "defaultValue": "windows-2022" },
      { "parameter_name": "AGENT_LABEL", "defaultValue": "gpu-runner" }
    ],
    "label_expr": ["gpu-runner"]  // For agent routing
  },
  // ... more combinations
}

Using Matrix Values in Scripts

Build Script
# Environment variables are automatically set
echo "Running on OS: $OS"
echo "Agent Label: $AGENT_LABEL"

# Or use substitution syntax
echo "Testing on %OS% with %AGENT_LABEL%"

Parallel Execution: Matrix child builds run in parallel across available agents. The parent build waits for all children to complete. Parent duration ≈ max child duration (not sum).

Selective Restart: You can restart specific matrix children using a boolean selection mask, e.g., {"windows-2022-gpu-runner": true, "ubuntu-22.04-standard-runner": false}.

Artifacts & Reports

TestHide automatically captures and indexes build outputs, test results, and media attachments.

Test Results

Auto-parsing for JUnit, PyTest, and NUnit formats. TestHide tracks every test case history, execution time, and stability (flakiness).

Media & Snapshots

Store screenshots, videos, and HTML reports. AI analysis triggers automatically on new failure screenshots to group similar issues.

Configuring Artifacts

artifact-config.yaml
post_steps:
  # Capture JUnit reports for analytics
  - type: test_provider
    provider: pytest
    path: "**/reports/*.xml"
  
  # Upload logs and screenshots
  - type: copy_artifacts
    source: "logs/**/*.log"
    target: "build_logs"
  
  - type: copy_artifacts
    source: "screenshots/*.png"
    target: "failures"

AI Configuration

TestHide utilizes a suite of specialized machine learning models to analyze test results in real-time. These models are hosted on the backend and triggered automatically after build completion.

Root Cause

MultiModalClassifier: Classifies stack traces and logs into meaningful categories (e.g., Environment, Bug, Network).

Visual CNN

Uses a Convolutional Neural Network to cluster failures by visual similarity in screenshots.

Flakiness

Statistical model that predicts test stability based on historical transitions and execution patterns.

OOD Detection

TinyAutoencoder: Out-of-Distribution detector that identifies novel errors never seen before by the system.

Bug Linker

Uses text embeddings to correlate test failures with existing Jira issues or internal bug reports.

Model Release: Models are stored in the AI_MODELS_RELEASE_DIR and can be updated without restarting the main server. New samples are processed in batches of AI_FAST_THRESHOLD.

Permissions & Roles

TestHide implements a multi-tenant RBAC system with global roles and granular project-scoped permissions. Permissions are stored in the user profile as a list of strings and validated by the backend middleware.

System Roles

Role IDDescriptionCapability
userStandard corporate user.Access limited to explicitly assigned projects and basic read-only global views.
superuserSystem Administrator.Full bypass of all permission checks. Access to all projects, global settings, and user management.
service_accountInternal System Account.Used for automated agents and background workers with wide system permissions.

Project-Scoped Permissions

Project permissions follow a specific string pattern: PROJECT_{id}_{OP}. These are automatically granted to project creators and can be assigned to other users by Project Admins.

PROJECT_..._READ
View Only

Access to build history, test reports, artifacts, and configuration (read-only).

PROJECT_..._WRITE
Configuration

Ability to modify job settings, update SCM endpoints, and manage project metadata.

PROJECT_..._EXECUTE
Pipeline Control

Permission to trigger new builds, restart failed stages, and stop active pipelines.

PROJECT_..._DELETE
Data Removal

High-level access to permanently delete builds, reports, or the entire project.

Global Permission Categories

System-wide actions are governed by category-based strings (e.g., JOBS_READ, NODES_WRITE). Major categories include:

REPORTSJOBSPROJECTSBUILDSNODESUSERSAI_ASSISTCONFIGURATIONMONITORING

SSO & Authentication

Secure your instance with modern authentication providers.

Local JWT

Standard token-based auth with encrypted password storage. Managed via the admin panel.

OIDC / Google

Connect your corporate identity provider. Support for Google, GitLab, and custom OIDC.

Security: All API requests require a Bearer token. Agent communication is secured via a persistent WebSocket challenge.

Integrations

Git Flow

Auto-trigger on PR/Commit. GitHub, GitLab, Bitbucket supported.

Issue Tracking

Bilateral Jira integration. Links failed tests to Jira tickets.

ChatOps

Real-time status cards in Teams and Slack channels.

Git Providers

Connect your repository to automatically trigger builds. TestHide supports SSH and HTTPS access protocols.

source-config.yaml
source_management:
  type: git
  url: "git@github.com:company/repo.git"
  branch: "main"
  credentials_id: "github-ssh-key"
  poll_interval: "5m"

Jira Bug Linker

Enable AI-assisted bug tracking. TestHide can search for existing Jira issues that match the current failure signature or create new ones.

AI Insight: When enabled, TestHide adds a "Bug Search" button to every failed test report, querying your Jira instance for similar stack traces.

Notifications

Send detailed report cards to Microsoft Teams or Slack. Notifications can be triggered on success, failure, or stability change (flaky detection).

notifications.yaml
post_steps:
  - type: msteams_notification
    webhook_url: "https://outlook.office.com/webhook/..."
    events: [failed, flaky]
    template: "Build #{build_number} failed on {branch}"

Incoming Webhooks

Trigger build jobs from external systems via simple HTTP POST requests.

CURL Trigger
curl -X POST https://testhide.com/api/v3/job/{job_id}/trigger \
     -H "Authorization: Bearer <TOKEN>" \
     -d '{"branch": "feature/ui-fix", "parameters": {"DEBUG": "true"}}'

API Data Reference

Overview

The TestHide API is a RESTful interface that allows you to manage jobs, builds, and agents programmatically. All responses are returned in JSON format.

Full interactive documentation: /api/v3/documentation

Authentication

Authenticate your requests using the Authorization: Bearer <TOKEN> header.

Headers
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
Content-Type: application/json

Build Object Structure

GET /api/v3/build/{id}
{
  "id": "60d5ec...",
  "status": "failed",
  "build_number": 421,
  "branch": "main",
  "duration": 452,
  "started_at": "2024-03-20T10:30:00Z",
  "tests": {
    "passed": 142,
    "failed": 3,
    "skipped": 5,
    "flaky": 1
  },
  "tps": {
    "flaky_tests": ["test_login_timeout"],
    "new_failures": ["test_checkout_flow"],
    "probable_cause": "Network latency in secondary DC"
  }
}