Introduction
TestHide is an agentic CI/CD platform optimized for automated testing. Unlike general-purpose CI tools, TestHide treats test results as first-class citizens, using AI to analyze failures, detect flakiness, and correlate errors across builds.
Smart Analysis
Auto-classification of failures using ML (Stack traces, Screenshots).
Resilient Agents
Cross-platform agents (Windows/Linux/macOS) with WebSocket persistence.
Matrix Builds
Run tests across thousands of configurations in parallel.
Flakiness Detection
Statistical tracking of unstable tests with quarantine workflows.
System Architecture
TestHide consists of three main components that communicate via WebSocket and REST APIs.
Backend Server
Python/aiohttp REST API + WebSocket server. Handles job scheduling, build orchestration, AI analysis, and database operations.
Port: 8000 (HTTP/WS)Frontend UI
Angular SPA with IDE-like interface. Provides job management, build monitoring, test analytics, and agent control.
Port: 4200 (nginx)Agents (testhide_client)
Cross-platform C# executables. Connect via WebSocket, execute builds, collect results, and report back in real-time.
Win/Linux/macOSData Flow
- Job Trigger: User or webhook initiates a build via Frontend/API
- Queue: Backend adds build to queue and notifies available agents
- Execution: Agent claims build, executes steps, streams logs via WebSocket
- Analysis: Backend receives results, AI models analyze failures
- Storage: Results persisted to MongoDB, artifacts to file storage
Quick Start
Deploy TestHide instantly using Docker Compose. This configuration includes all required services.
version: '3.8'
name: testhide
services:
# ==========================================
# FRONTEND (Nginx + Angular)
# ==========================================
frontend:
container_name: testhide_frontend
image: thuesdays/testhide-frontend:latest
restart: always
ports:
- "80:80"
- "443:443"
- "7771:7771"
env_file:
- .env
environment:
- CERT_FILE=${CERT_FILE}
- CERT_KEY=${CERT_KEY}
volumes:
- ./ssl:/etc/ssl:ro
- ./nginx/api.conf:/etc/nginx/conf.d/api.conf:ro
- testhide_static_data:/usr/share/nginx/html/static:ro
depends_on:
- backend
- ai-api
networks:
- testhide-net
deploy:
resources:
limits:
cpus: "0.25"
memory: 256m
# ==========================================
# BACKEND (Python API)
# ==========================================
backend:
image: thuesdays/testhide-backend:latest
restart: unless-stopped
env_file:
- .env
environment:
- LOAD_AI_MODELS=false
- RUN_AI_WORKER=false
- GUNICORN_WORKERS=2
- COMPOSE_PROJECT_NAME=testhide
volumes:
- testhide_static_data:/app/static
- ./releases:/app/releases
- ./monitoring_scripts:/app/monitoring_scripts
- ./gunicorn.conf.py:/app/gunicorn.conf.py:ro
- ./ssl:/etc/ssl:ro
- ./sandbox_data:/app/sandbox_data
- /var/run/docker.sock:/var/run/docker.sock
- ./docker-compose.prod.yaml:/app/docker-compose.yaml:ro
depends_on:
mongo:
condition: service_healthy
redis:
condition: service_healthy
networks:
- testhide-net
healthcheck:
disable: true
mem_limit: 1.5g
memswap_limit: 1.5g
deploy:
replicas: 2
resources:
limits:
cpus: "2.0"
memory: 1.5g
reservations:
cpus: "0.5"
memory: 768m
# ==========================================
# AI API (HTTP only — serves /api/v3/ai/)
# ==========================================
ai-api:
container_name: testhide_ai_api
image: thuesdays/testhide-backend:latest
restart: unless-stopped
env_file:
- .env
environment:
- LOAD_AI_MODELS=true
- RUN_AI_WORKER=false
- GUNICORN_WORKERS=2
volumes:
- testhide_static_data:/app/static
- ./sandbox_data:/app/sandbox_data
- ./releases:/app/releases
- ./monitoring_scripts:/app/monitoring_scripts
- ./gunicorn.conf.py:/app/gunicorn.conf.py:ro
- ./ssl:/etc/ssl:ro
depends_on:
mongo:
condition: service_healthy
redis:
condition: service_healthy
networks:
- testhide-net
healthcheck:
disable: true
mem_limit: 5g
memswap_limit: 5g
deploy:
resources:
limits:
cpus: "2.0"
memory: 5g
reservations:
cpus: "0.5"
memory: 3g
# ==========================================
# AI WORKER (Background: training, vectorization)
# ==========================================
ai-worker:
container_name: testhide_ai_worker
image: thuesdays/testhide-backend:latest
restart: unless-stopped
env_file:
- .env
environment:
- LOAD_AI_MODELS=true
- RUN_AI_WORKER=true
- GUNICORN_WORKERS=1
volumes:
- testhide_static_data:/app/static
- ./sandbox_data:/app/sandbox_data
- ./releases:/app/releases
- ./monitoring_scripts:/app/monitoring_scripts
- ./gunicorn.conf.py:/app/gunicorn.conf.py:ro
- ./ssl:/etc/ssl:ro
depends_on:
mongo:
condition: service_healthy
redis:
condition: service_healthy
networks:
testhide-net:
aliases:
- ai-worker
healthcheck:
disable: true
mem_limit: 7g
memswap_limit: 7g
deploy:
resources:
limits:
cpus: "2.0"
memory: 7g
reservations:
cpus: "0.5"
memory: 4g
# ==========================================
# DATABASES
# ==========================================
mongo:
container_name: testhide_mongo
image: 'mongo:8.2.3'
restart: always
command: [ "mongod", "--auth", "--wiredTigerCacheSizeGB", "2" ]
environment:
- MONGO_INITDB_ROOT_USERNAME=${MONGO_USER}
- MONGO_INITDB_ROOT_PASSWORD=${MONGO_PASS}
ports:
- "27017:27017"
volumes:
- ${MONGO_DATA_PATH}:/data/db
networks:
- testhide-net
healthcheck:
test: [ "CMD", "mongosh", "--norc", "--quiet", "-u", "${MONGO_USER}", "-p", "${MONGO_PASS}", "--authenticationDatabase", "admin", "--eval", "db.adminCommand('ping')" ]
interval: 30s
timeout: 5s
retries: 5
deploy:
resources:
limits:
cpus: "1.5"
memory: 3g
reservations:
cpus: "0.5"
memory: 1.5g
redis:
container_name: testhide_redis
image: redis:7-alpine
restart: always
command: [ "redis-server", "--maxmemory", "512mb", "--maxmemory-policy", "allkeys-lru", "--appendonly", "no", "--requirepass", "${REDIS_PASSWORD}" ]
networks:
- testhide-net
healthcheck:
test: [ "CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping" ]
interval: 10s
timeout: 3s
retries: 5
deploy:
resources:
limits:
cpus: "0.5"
memory: 768m
redisinsight:
container_name: testhide_redisinsight
image: redis/redisinsight:latest
restart: always
depends_on:
redis:
condition: service_healthy
networks:
- testhide-net
deploy:
resources:
limits:
cpus: "0.25"
memory: 256m
# ==========================================
# VOLUMES & NETWORKS
# ==========================================
volumes:
testhide_static_data:
name: testhide_static_data
networks:
testhide-net:
name: testhide-netDeployment Steps
- Create a
.envfile with your secrets:.envMONGO_PASSWORD=your_secure_password JWT_SECRET=your_super_secret_jwt_key
- Run
docker-compose up -d - Access the frontend at http://localhost:4200
- Default login: admin / admin
WebSocket URL for Agents: Configure your agents to connect to ws://your-server:8000 (or wss:// with SSL).
Installation & Configuration
Environment Variables
| Variable | Default | Description |
|---|---|---|
| MONGO_HOST | localhost | MongoDB server hostname. |
| MONGO_PORT | 27017 | MongoDB server port. |
| MONGO_DB_NAME | testhide_database | MongoDB database name. |
| MONGO_USER | testhide_user | MongoDB authentication username. |
| MONGO_PASS | (empty) | MongoDB authentication password. |
| JWT_SECRET | <default> | Secret key for signing JWT auth tokens. Change in production! |
| TPS_HEARTBEAT_TIMEOUT_SEC | 300 | Agent timeout (seconds) before marking as offline. |
| CLIENT_MAX_SIZE_MB | 4096 | Max upload size for artifacts (in MB). |
| USE_SSL | false | Enable HTTPS/WSS. Requires CERT_KEY and CERT_FILE. |
| CERT_KEY / CERT_FILE | (none) | Paths to SSL certificate files for HTTPS. |
| CORS_ORIGINS | * | Comma-separated list of allowed CORS origins. |
| DEBUG | false | Enable debug mode (verbose logging). |
| FRONTEND_PUBLIC_URL | (none) | Public URL of the frontend for email links. |
AI Configuration Variables
| Variable | Default | Description | |
|---|---|---|---|
| Variable Classification | Name | Default | Description |
|---|---|---|---|
| Global Settings | |||
| Master Switch | AI_ENABLE_AI_ASSIST | true | Global kill-switch for all AI background workers. |
| Feature Toggles (Enable/Disable Models) | |||
| Feature Toggle | AI_FEATURE_AI_ROOT_CAUSE_ENABLED | true | Enable Root Cause analysis. |
| Feature Toggle | AI_FEATURE_AI_FLAKINESS_ENABLED | true | Enable Flakiness prediction. |
| Feature Toggle | AI_FEATURE_AI_VISUAL_ENABLED | true | Enable Visual Diff analysis. |
| Feature Toggle | AI_FEATURE_AI_OOD_ENABLED | true | Enable Out-of-Distribution detection. |
| Feature Toggle | AI_FEATURE_AI_LOG_SIGNATURES_ENABLED | true | Enable Log Signature mining. |
| Feature Toggle | AI_FEATURE_AI_RETRIEVER_ENABLED | true | Enable Failure Retriever (Search). |
| Feature Toggle | AI_FEATURE_AI_EMERGING_ISSUES_ENABLED | true | Enable Emerging Issues detection. |
| Feature Toggle | AI_FEATURE_AI_BUG_LINKER_ENABLED | true | Enable auto-linking to Jira/Issues. |
| Hardware & Performance | |||
| Infrastructure | AI_CPU_LIMIT_CORES | 2 | Max CPU cores per worker process. |
| Infrastructure | AI_OMP_NUM_THREADS | 1 | OpenMP threads (set 1 to avoid contention). |
| Infrastructure | AI_MKL_NUM_THREADS | 1 | MKL threads (set 1 to avoid contention). |
| Infrastructure | AI_HF_DATASETS_NUM_PROC | 2 | Parallel processes for HuggingFace datasets. |
| Infrastructure | AI_TOKENIZERS_PARALLELISM | false | Enable parallelism in tokenizers. |
| Update Thresholds | |||
| Thresholds | AI_FAST_UPDATE_THRESHOLD | 500 | Samples to trigger light updates (FAISS/OOD). |
| Thresholds | AI_HEAVY_UPDATE_THRESHOLD | 1000 | Samples to trigger full model fine-tuning. |
| Thresholds | AI_HEAVY_MIN_INTERVAL_SEC | 18000 | Min seconds (5h) between heavy training runs. |
| Root Cause Model | |||
| Root Cause | AI_RC_BATCH_SIZE | 8 | Training batch size. |
| Root Cause | AI_RC_EPOCHS | 3 | Training epochs. |
| Root Cause | AI_RC_MODEL_NAME | bert-base-uncased | Base HuggingFace model architecture. |
| Root Cause | AI_RC_MAX_LEN | 256 | Max sequence length for text tokens. |
| Root Cause | AI_RC_CLEANLAB_BACKEND | torch | Backend for label noise correction. |
| Other Models | |||
| Other | AI_FLAKINESS_EPOCHS | 100 | Epochs for flakiness predictor training. |
| Other | AI_VISUAL_EPOCHS | 5 | Epochs for visual diff model. |
| Other | AI_OOD_AE_EPOCHS | 10 | Epochs for OOD Autoencoder. |
| Other | AI_RETRIEVER_EPOCHS | 200 | Epochs for Contrastive Retriever. |
| Other | AI_LOGSIGN_TOP_EXAMPLES | 3 | Number of examples to keep per log signature. |
Agent Deployment (testhide_client)
Agents connect to the backend via WebSocket. The agent is a cross-platform C# executable that auto-updates and persists on the system.
1. Configure the Agent
Set the backend WebSocket URL using the config command:
# 1. Configure testhide.exe config --url wss://testhide.company.com --instance-id my-server-01 # 2. Run testhide.exe run # View config testhide.exe config --show
2. Config Command Options
| Flag | Short | Description |
|---|---|---|
| --url | -u | WebSocket server URL (ws:// or wss://) |
| --instance-id | -i | Instance ID for licensing/identification |
| --show | -s | Display current configuration |
| --help | -h | Show help message |
3. Start the Agent
# Start the agent using saved configuration testhide.exe run # Verify connection in the logs: # [INFO] Successfully connected to wss://... # [INFO] Registered with ID: ...
4. config.json Structure
The agent stores configuration in config.json next to the executable:
{
"WebSocketUrl": "wss://testhide.company.com",
"InstanceId": "my-server-id",
"UpdateCheckIntervalSeconds": 3600,
"MinDiskSpaceGbForCleanup": 10.0,
"LogCleanupDays": 5,
"MaxDisconnectionsAllowed": 1,
"GitCheckoutTimeoutSeconds": 600,
"IncludeTestCasesInSnapshots": false,
"DirectoriesToClean": [],
"StrayProcessNamesToKill": [],
"PythonEnv": {
"AutoInstallPython": false,
"AutoInstallPyenv": false,
"AutoInstallRequirements": true,
"FallbackToVirtualenv": true
}
}Requirements: The agent machine should have Python 3.9+ and Git installed for most build steps. The agent supports Windows, Linux, and macOS.
Features: Remote Desktop access, system monitoring, auto-updates, label-based routing, concurrent build execution, and local AI analysis (when enabled).
Pipelines & Jobs
TestHide jobs define the build logic, triggers, and execution environment. You can manage them via the IDE UI or by placing YAML files in your repository.
UI Management
Use our 9-tab integrated editor for visual configuration. Includes Source Management, AI Analysis triggers, Matrix Config, and Build Environment overrides.
YAML Discovery
Commit testhide.yaml to your repository. TestHide automatically discovers and updates job configurations based on branch changes.
Build Steps
A job consists of a list of steps executed sequentially on the agent. Each step type provides specialized parsing and execution logic.
Windows Batch
Standard .bat/.cmd commands.
PowerShell
Execute .ps1 scripts on Windows agents.
Bash
Shell scripting for Linux/macOS.
Python
Run .py scripts in isolated venv.
Docker
Containerized build execution.
Test Provider
PyTest, JUnit, NUnit parsing.
Copy Artifacts
Manage build outputs.
Isolated Execution: Python steps can automatically manage pyenv and requirements.txt installations if enabled in the agent config.
Matrix Builds
Matrix builds allow you to run the same job across multiple configurations in parallel. TestHide generates a child build for each combination of axis values (Cartesian product).
Axis Types
You can define up to 2 axes per job:
Nodes Axis (by name/label)
Select specific agents or labels from a tree view. Each selected node becomes a matrix cell. Builds are routed to matching agents.
- • Expandable tree for node/label selection
- • Routes builds to matching agents
- • Default value for single runs
Custom Axis
Define arbitrary values (one per line). Use for OS versions, browsers, environments, or any configuration dimension.
- • Free-form text values
- • Exposed as environment variables
- • Available via %PARAM_NAME% substitution
Configuration Example
job_parameters:
# Custom Axis: OS versions
- type: matrix
matrix_type: custom_axis
parameter_name: OS
values:
- windows-2022
- ubuntu-22.04
- macos-14
# Nodes Axis: Run on specific agents
- type: matrix
matrix_type: nodes
parameter_name: AGENT_LABEL
values:
- gpu-runner
- standard-runner
# Result: 6 child builds (3 OS × 2 Agents)
# Child keys: "windows-2022-gpu-runner", "ubuntu-22.04-standard-runner", etc.Generated Child Build Structure
For each combination, TestHide generates a child configuration with:
{
"windows-2022-gpu-runner": {
"envs": [
{ "env_name": "OS", "env_value": "windows-2022" },
{ "env_name": "AGENT_LABEL", "env_value": "gpu-runner" }
],
"job_parameters": [
{ "parameter_name": "OS", "defaultValue": "windows-2022" },
{ "parameter_name": "AGENT_LABEL", "defaultValue": "gpu-runner" }
],
"label_expr": ["gpu-runner"] // For agent routing
},
// ... more combinations
}Using Matrix Values in Scripts
# Environment variables are automatically set echo "Running on OS: $OS" echo "Agent Label: $AGENT_LABEL" # Or use substitution syntax echo "Testing on %OS% with %AGENT_LABEL%"
Parallel Execution: Matrix child builds run in parallel across available agents. The parent build waits for all children to complete. Parent duration ≈ max child duration (not sum).
Selective Restart: You can restart specific matrix children using a boolean selection mask, e.g., {"windows-2022-gpu-runner": true, "ubuntu-22.04-standard-runner": false}.
Artifacts & Reports
TestHide automatically captures and indexes build outputs, test results, and media attachments.
Test Results
Auto-parsing for JUnit, PyTest, and NUnit formats. TestHide tracks every test case history, execution time, and stability (flakiness).
Media & Snapshots
Store screenshots, videos, and HTML reports. AI analysis triggers automatically on new failure screenshots to group similar issues.
Configuring Artifacts
post_steps:
# Capture JUnit reports for analytics
- type: test_provider
provider: pytest
path: "**/reports/*.xml"
# Upload logs and screenshots
- type: copy_artifacts
source: "logs/**/*.log"
target: "build_logs"
- type: copy_artifacts
source: "screenshots/*.png"
target: "failures"AI Configuration
TestHide utilizes a suite of specialized machine learning models to analyze test results in real-time. These models are hosted on the backend and triggered automatically after build completion.
Root Cause
MultiModalClassifier: Classifies stack traces and logs into meaningful categories (e.g., Environment, Bug, Network).
Visual CNN
Uses a Convolutional Neural Network to cluster failures by visual similarity in screenshots.
Flakiness
Statistical model that predicts test stability based on historical transitions and execution patterns.
OOD Detection
TinyAutoencoder: Out-of-Distribution detector that identifies novel errors never seen before by the system.
Bug Linker
Uses text embeddings to correlate test failures with existing Jira issues or internal bug reports.
Model Release: Models are stored in the AI_MODELS_RELEASE_DIR and can be updated without restarting the main server. New samples are processed in batches of AI_FAST_THRESHOLD.
Permissions & Roles
TestHide implements a multi-tenant RBAC system with global roles and granular project-scoped permissions. Permissions are stored in the user profile as a list of strings and validated by the backend middleware.
System Roles
| Role ID | Description | Capability |
|---|---|---|
| user | Standard corporate user. | Access limited to explicitly assigned projects and basic read-only global views. |
| superuser | System Administrator. | Full bypass of all permission checks. Access to all projects, global settings, and user management. |
| service_account | Internal System Account. | Used for automated agents and background workers with wide system permissions. |
Project-Scoped Permissions
Project permissions follow a specific string pattern: PROJECT_{id}_{OP}. These are automatically granted to project creators and can be assigned to other users by Project Admins.
Access to build history, test reports, artifacts, and configuration (read-only).
Ability to modify job settings, update SCM endpoints, and manage project metadata.
Permission to trigger new builds, restart failed stages, and stop active pipelines.
High-level access to permanently delete builds, reports, or the entire project.
Global Permission Categories
System-wide actions are governed by category-based strings (e.g., JOBS_READ, NODES_WRITE). Major categories include:
REPORTSJOBSPROJECTSBUILDSNODESUSERSAI_ASSISTCONFIGURATIONMONITORINGSSO & Authentication
Secure your instance with modern authentication providers.
Local JWT
Standard token-based auth with encrypted password storage. Managed via the admin panel.
OIDC / Google
Connect your corporate identity provider. Support for Google, GitLab, and custom OIDC.
Security: All API requests require a Bearer token. Agent communication is secured via a persistent WebSocket challenge.
Integrations
Git Flow
Auto-trigger on PR/Commit. GitHub, GitLab, Bitbucket supported.
Issue Tracking
Bilateral Jira integration. Links failed tests to Jira tickets.
ChatOps
Real-time status cards in Teams and Slack channels.
Git Providers
Connect your repository to automatically trigger builds. TestHide supports SSH and HTTPS access protocols.
source_management: type: git url: "git@github.com:company/repo.git" branch: "main" credentials_id: "github-ssh-key" poll_interval: "5m"
Jira Bug Linker
Enable AI-assisted bug tracking. TestHide can search for existing Jira issues that match the current failure signature or create new ones.
AI Insight: When enabled, TestHide adds a "Bug Search" button to every failed test report, querying your Jira instance for similar stack traces.
Notifications
Send detailed report cards to Microsoft Teams or Slack. Notifications can be triggered on success, failure, or stability change (flaky detection).
post_steps:
- type: msteams_notification
webhook_url: "https://outlook.office.com/webhook/..."
events: [failed, flaky]
template: "Build #{build_number} failed on {branch}"Incoming Webhooks
Trigger build jobs from external systems via simple HTTP POST requests.
curl -X POST https://testhide.com/api/v3/job/{job_id}/trigger \
-H "Authorization: Bearer <TOKEN>" \
-d '{"branch": "feature/ui-fix", "parameters": {"DEBUG": "true"}}'API Data Reference
Overview
The TestHide API is a RESTful interface that allows you to manage jobs, builds, and agents programmatically. All responses are returned in JSON format.
Full interactive documentation: /api/v3/documentation
Authentication
Authenticate your requests using the Authorization: Bearer <TOKEN> header.
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9... Content-Type: application/json
Build Object Structure
{
"id": "60d5ec...",
"status": "failed",
"build_number": 421,
"branch": "main",
"duration": 452,
"started_at": "2024-03-20T10:30:00Z",
"tests": {
"passed": 142,
"failed": 3,
"skipped": 5,
"flaky": 1
},
"tps": {
"flaky_tests": ["test_login_timeout"],
"new_failures": ["test_checkout_flow"],
"probable_cause": "Network latency in secondary DC"
}
}