# CI failure triage A local tool for reviewing JUnit test artifacts across builds and retry attempts. It retains the original XML, keeps import identities immutable, and produces a standalone HTML report with build selection, search, classification filters, attempt details and XML/CSV downloads. The included history is synthetic: five builds, two jobs and two partial retries. It is an executed reference run, not a real company's CI history. Open `report.html` directly in a browser to inspect it. No server, account, or network connection is needed. ## Requirements Python 3.10 or newer with the standard-library `sqlite3` module. There are no third-party dependencies. The recorded run uses Python 3.12. Runtime version and SQLite version are retained in the acceptance measurements. ## Reproduce the delivered run ```sh python run_demo.py python verify.py python acceptance.py ``` `run_demo.py` replaces the generated files under `examples/`, `results/`, `figures/` and `report.html`, including `results/demo.sqlite`. Keep your own history in a different database path. The regression and acceptance commands use temporary databases and do not touch your own history. ## Use it with your test artifacts Keep one database per repository and comparison stream, such as a branch's sequential builds. Sequence numbers are explicit positive integers, unique within the database. Filesystem modification time and lexicographic build names are never used to guess order. ```sh python triage.py ingest junit-api.xml --db history.sqlite --build build-201 --sequence 201 --job api-linux python triage.py ingest junit-unit.xml --db history.sqlite --build build-201 --sequence 201 --job unit-linux python triage.py ingest junit-retry.xml --db history.sqlite --build build-201 --sequence 201 --job api-linux --attempt 2 --mode partial python triage.py report --db history.sqlite --out review python triage.py check --db history.sqlite --build build-201 ``` Open `review/index.html`. `review/analysis.json` contains the complete normalized history and base64-encoded original XML; `review/latest.csv` contains the latest classification. The report's CSV button exports the currently filtered rows. Original XML buttons download the exact ingested bytes. Several XML files from one job attempt can be supplied in one command. Their basenames and testcase identities must be distinct. All files are validated before the transaction starts. ```sh python triage.py ingest shard-a.xml shard-b.xml --db history.sqlite --build build-202 --sequence 202 --job api-linux ``` A validation-only check does not create a database: ```sh python triage.py ingest junit-api.xml --build build-201 --sequence 201 --job api-linux --dry-run ``` Dry run validates XML and arguments only. Immutable database identities and the earlier-full-attempt requirement for partial retries are checked by a real ingestion. ## Identity and classification rules A test identity is the exact tuple `(job, file, classname, name)`. Missing `file` and `classname` become empty strings; missing `name` is rejected. Parameterized tests need distinct names. Renaming or moving a test creates a new identity. This deliberate rule avoids guessing whether two similarly named tests are the same. An import identity is `(build, job, attempt)`. Reimporting the same artifact basenames, SHA-256 hashes and mode is idempotent. A different payload or mode under that identity is rejected. Build names and sequence assignments cannot be changed in place. - **Full attempt:** replaces that job's test membership for the build. A previously known test omitted from the latest full attempt is not reported, even if it passed earlier. - **Partial retry:** overlays only its supplied cases. It requires an earlier full attempt for that build/job. Unretried cases keep their last result. - **Retry pass:** final outcome is pass and an earlier observed attempt in the same build failed or errored. This is a candidate for investigation; the tool does not diagnose flakiness. - **New failure:** a final failure/error follows a pass in the immediately preceding build. - **Repeated failure:** the immediately preceding build also has a final failure/error. - **Failure without baseline:** final failure/error without an immediately preceding observed pass or failure. - **Recovered:** final pass follows a final failure/error in the preceding build, without a current failed-then-passed retry. - **Skipped / not reported:** remain separate from passes. The tool cannot infer whether a missing test was intentionally removed. Builds can be imported out of sequence. Comparisons are recomputed in numerical sequence order. A single read transaction keeps report snapshots consistent while another process ingests data. Summed testcase duration includes all observed attempts and is not pipeline wall time; missing duration remains unknown. ## Accepted XML and input boundaries The parser accepts a bounded JUnit-style subset: `testsuite`/`testsuites` roots, nested suites, namespaces, and `testcase` names, classnames, files and times. A testcase can have one `failure`, `error` or `skipped` child. Multiple outcome elements are rejected. Suite totals are not trusted; counts are calculated from parsed cases. An empty report is rejected rather than interpreted as a passing job. Framework-specific retry elements are not interpreted; retries are supplied as explicit artifacts and attempt numbers. Only UTF-8 XML is accepted. DTD/entity declarations and NUL bytes are rejected before XML parsing. Limits are 10 MiB combined input per attempt, 50,000 cases per attempt, 100 files, XML depth 64, 300,000 elements per file and bounded attributes. A retained failure message is truncated after 32,768 characters with a visible notice; the original XML remains intact. Test durations must be finite, nonnegative and no more than seven days. Artifact inputs must be regular `.xml` files, not symbolic links. The tool does not follow attachment paths or retrieve URLs found inside XML. SQL values are bound parameters. Report text uses DOM text nodes; embedded JSON escapes HTML delimiters. Spreadsheet formula prefixes in CSV cells are prefixed with an apostrophe. JSON and original XML preserve exact source values. Use a current supported Python runtime. Byte and structure limits reduce input exposure but do not turn the tool into a universal XML sandbox. See the Python XML security documentation below for interpreter/Expat considerations. ## Exit codes and transactions - `0`: successful ingest/report, or `check` with no final failures/errors. - `1`: `check` found a final failure/error. - `2`: invalid input, conflicting identity, unsupported database, I/O or SQLite error. `check` does not fail merely because tests were skipped or unreported. A release process that requires complete coverage must enforce that policy separately; the counts are in the JSON output. XML import does not replace the test runner's exit code. Ingestion uses `BEGIN IMMEDIATE`; artifact bytes and cases commit together. The write-lock timeout is 3 seconds. An interrupted transaction is rolled back on reopen. Existing unrelated or unsupported databases are rejected before schema changes. Opening a supported database performs no schema or header writes; exact duplicates and conflicting replays preserve the database bytes. Report exports replace each file atomically, but the entire output directory is not a transaction. The HTML is self-contained and does not depend on its sibling JSON or CSV being updated at the same instant. ## Files - `triage.py`: parser, database, classification and CLI. - `report.py`: offline HTML renderer and browser controls. - `fixtures.py`: deterministic synthetic JUnit generation. - `run_demo.py`: reference ingestion and result graphic. - `test_triage.py` / `verify.py`: behavioral regression suite and exact run log. - `acceptance.py`: actual concurrent imports, abrupt process exit, replay, size rejection, artifact round-trip, deterministic export and 10,000-case workload. - `completion/report.html`: expected and observed operating outcomes, timing and limitations. - `examples/manifest.json`: explicit identity and mode for every supplied fixture. - `results/summary.json`, `verification.json`, `completion.json`: recorded outcomes and source hashes. The 10,000-case run measures one local environment. It is not a production throughput guarantee. The accepted per-attempt limits do not establish a maximum unlimited-history database or browser report size. Archive or partition history according to the repository and comparison stream. ## Primary references - Python XML processing and security: https://docs.python.org/3/library/xml.html - Python SQLite transactions and parameter binding: https://docs.python.org/3/library/sqlite3.html - GitLab's documented JUnit subset and report semantics: https://docs.gitlab.com/ci/testing/unit_test_reports/ The tool implements its own explicit identity, retry and missing-result policies. It is not a GitLab, GitHub Actions or Jenkins connector and does not claim complete compatibility with every producer's JUnit extension. No real repository, provider login or production deployment is required for the delivered file-based workflow.