/ plugins / report format
Testhide Report Format v1
The canonical contract every official Testhide reporter emits — a JUnit-extended XML (junittests.xml) with a stable failure id, a resolution, captured output, attachments and suite metadata. The build agent parses it identically for pytest, unittest, .NET and JS, so your data always lands correctly.
The one rule
Always emit the JUnit dialect: <testsuites> → <testsuite> → <testcase>. That's the only shape that carries the rich fields (fail_id, test_resolution, properties, system-out). Native NUnit / xUnit XML is a degraded fallback that drops them — which is exactly why you install the plugin.
1 · Status: stable. Plugins emit testhide_schema_version=1 as a suite property; a missing version is treated as 1 for back-compat.Document & suite
One <testsuite> per run, wrapped in <testsuites>. Counts should equal the sum of the testcases; the suite timestamp must be ISO-8601 UTC (trailing Z).
Suite attributes
| Attribute | Required | Notes |
|---|---|---|
name | optional | Runner / suite name (e.g. pytest, dotnet). Free text. |
timestamp | required | ISO-8601 UTC with trailing Z. The agent derives per-test start times from this + each time. |
hostname | optional | Auto-populated by the plugin. |
tests / failures / errors / skipped / time | recommended | Sum of children. The backend recomputes from the testcases, so a mismatch is a warning, not fatal. |
Suite <properties>
| Property | Required | Notes |
|---|---|---|
testhide_schema_version | required | Value 1 in v1 plugins. |
ip_address, hostname | auto | Populated by the plugin; identify the runner. |
build | optional | Reserved — shown as the build number. |
branch | optional | Reserved — the branch. |
| any other | optional | Free metadata: every name/value pair is captured verbatim. |
The <testcase>
One per test. A passing test has no outcome child; every other state has exactly one of <failure> / <error> / <skipped>.
Attributes
| Attribute | Required | Notes |
|---|---|---|
classname | required | Dotted path (package.module.Class); strip the source file extension. |
name | required | Test function/case name. Parametrization is stripped from the identity used for fail_id (the human name may keep it). |
time | required | Seconds, . decimal, invariant culture. setup + call + teardown. |
fail_id | required* | md5("module.class.function.ExceptionType(message)"). Non-empty on failure/error, empty otherwise. Stable across runs → the backend dedups failures and links Jira on it. |
test_resolution | required | Closed set (below). Parser default if absent: Unresolved. |
file, line | recommended | Enables stacktrace ↔ changed-file code-impact scoring. |
Outcome children
| State | Element | Notes |
|---|---|---|
| Passed | none | No outcome child; fail_id="". |
| Failed | <failure message="…"> | One-line message; full cleaned traceback in <![CDATA[…]]>. |
| Errored | <error message="…"> | Collection / import / teardown error (not an assertion). |
| Skipped / xfail | <skipped type="…" message="…"> | For an expected failure that failed (xfail), prefer <failure> + test_resolution="Known Issue" so it isn't a hard failure. |
Per-test <properties> & output
| Property | Notes |
|---|---|
docstr | Human-readable intent (the docstring). Fed to the text embedder. |
attachment | Repeatable. File path or URL — images, logs, JSON, binaries. The agent downloads & runs image/binary/config parsers (+ CLIP on images). Value must be non-empty. |
info | Free-form JSON or text context. |
jira (or issue) | Linked ticket(s). Either name is accepted. |
<system-out> | Execution log / steps / HTTP traces. Sanitized & truncated to 512 KB. Wrap in <![CDATA[…]]>. |
test_resolution — closed set
Default mapping: pass→Passed, skip→Skipped, import/collection failure→Collection Error, teardown failure→Teardown Error, xfail→Known Issue. Jira enrichment may override to Known Issue / Need to reopen / Resolved in branch / Verified at Branch based on the linked ticket's status.
A complete report
Every field, every outcome — passed, failed (attachments + info + captured output), collection error, skipped, and a known-issue/xfail. Point your job's report_paths at this file.
<?xml version="1.0" encoding="utf-8"?>
<testsuites>
<testsuite name="pytest" timestamp="2026-05-29T10:00:00.000Z" hostname="build-node-07"
tests="5" failures="2" errors="1" skipped="1" time="0.470">
<properties>
<property name="testhide_schema_version" value="1"/>
<property name="ip_address" value="10.0.0.7"/>
<property name="hostname" value="build-node-07"/>
<property name="build" value="1042"/>
<property name="branch" value="main"/>
</properties>
<!-- PASSED — no outcome child, empty fail_id -->
<testcase classname="shop.auth.LoginTests" name="test_login_ok"
file="tests/test_login.py" line="12" time="0.100"
fail_id="" test_resolution="Passed">
<properties>
<property name="docstr" value="User can log in with valid credentials."/>
</properties>
</testcase>
<!-- FAILED — <failure> + attachments + info JSON + captured output -->
<testcase classname="shop.checkout.CheckoutTests" name="test_total_with_tax"
file="tests/test_checkout.py" line="48" time="0.200"
fail_id="3f8a1c2b9d4e5f60718293a4b5c6d7e8" test_resolution="Unresolved">
<failure message="AssertionError: expected total 110.00, got 100.00"><![CDATA[
Traceback (most recent call last):
File "tests/test_checkout.py", line 52, in test_total_with_tax
assert order.total == Decimal("110.00")
AssertionError: expected total 110.00, got 100.00
]]></failure>
<properties>
<property name="docstr" value="Order total must include 10% tax."/>
<property name="attachment" value="https://artifacts.example.com/1042/checkout_fail.png"/>
<property name="attachment" value="/var/run/testhide/1042/checkout.har"/>
<property name="info" value="{"retries": 1, "env": "staging"}"/>
</properties>
<system-out><![CDATA[POST /cart/checkout -> 200
[assert] total == 110.00 FAILED (got 100.00)]]></system-out>
</testcase>
<!-- ERRORED — import / collection failure -->
<testcase classname="shop.reports.ImportSuite" name="test_imports"
file="tests/test_reports.py" line="1" time="0.050"
fail_id="aa11bb22cc33dd44ee55ff6677889900" test_resolution="Collection Error">
<error message="ModuleNotFoundError: No module named 'pandas'"><![CDATA[
ModuleNotFoundError: No module named 'pandas'
]]></error>
</testcase>
<!-- SKIPPED -->
<testcase classname="shop.payment.PaymentTests" name="test_refund"
file="tests/test_payment.py" line="55" time="0.000"
fail_id="" test_resolution="Skipped">
<skipped type="pytest.skip" message="refund API disabled in staging"/>
</testcase>
<!-- XFAIL / KNOWN ISSUE — a failure linked to a ticket, not a hard regression -->
<testcase classname="shop.payment.PaymentTests" name="test_gateway_timeout"
file="tests/test_payment.py" line="70" time="0.120"
fail_id="bb22cc33dd44ee55ff6677889900aa11" test_resolution="Known Issue">
<failure message="TimeoutError: gateway did not respond in 30s"><![CDATA[
TimeoutError: gateway did not respond in 30s
]]></failure>
<properties>
<property name="jira" value="PAY-417 Known Issue [gateway timeout under load]"/>
</properties>
</testcase>
</testsuite>
</testsuites>Behaviour & guarantees
Every official plugin shares these.
The agent synthesizes per-test timestamps from the suite timestamp + each time — emit an accurate suite timestamp and per-test time; per-test timestamps aren't needed.
Each worker writes a temp chunk under .{report}_temp/, atomically merged on finish — safe with pytest-xdist, sharded dotnet test, and Jest workers.
Deselect flaky tests by node-id from --quarantine-file, TESTHIDE_QUARANTINE_FILE, or .testhide_quarantine_file. One id per line; blanks and # comments ignored.
Optional: given creds, each plugin looks up the issue by fail_id and enriches the failure message + resolution — works offline, no backend round-trip.
UTF-8, valid XML 1.0 only (control chars stripped, emoji/CJK/Cyrillic kept); tracebacks & system-out in CDATA; no DTD/DOCTYPE (XXE-hardened).
This is v1. Additive property names don't bump the version (unknown names are ignored safely); breaking changes ship a new version + spec.
Ready to wire it up? Browse the plugins ↗