Rasa upgrade test history

You upgrade an agent’s dependencies, then its version check rejects an old test result in the README. Replacing that old version would make the check pass. It would also change what the result claims to have tested.

Keep the historical observation and update the current installation requirement separately. Verify the running agent on the new release before recording a new assessment.

This guide follows a recorded failure in the Rasa examples repository’s automatic upgrade job. Its repair exposed a second failure: a bullet could hide a stale installation requirement from the checker. You will reproduce that distinction with offline checks. No license, model, or speech service is needed.

Keep the dated result intact and check the current installation requirements.

The upgrade stopped on a preserved result

The recorded upgrade job selected a candidate Rasa version and advanced the maintained manifests and locks. Its catalog gate then reported:

[DRIFT] patterns/mcp-confirmation-gate  readme='3.21.0.dev5'

The candidate target in that log was 3.22.0.dev3. The old version came from a preserved README assessment. One reader of version fields honored the history marker; another still treated the marked assessment as the current version.

The failure was reproduced in a regression that migrated a temporary project while preserving its assessment. The original failing regression output included:

AssertionError: '3.21.0.dev5' is not None

----------------------------------------------------------------------
Ran 10 tests in 0.115s

FAILED (failures=1)

Those are excerpts from the recorded failure, not results from a live agent. The repair needed the version readers to agree about which claim each line made. It did not need a new date or version stamped onto the old assessment.

A bullet hid a current installation requirement

Preserving history fixed the original false alarm. But the earlier repaired reader still disagreed with itself about an installation field inside a bullet or block quote.

The following are the actual input lines from a new offline reproduction. Both temporary dependency files named the newer target. The README alone named the old version:

Installation: rasa-pro 3.21.0.dev5 # rasa-version-ignore: must not hide the current pin
- Installation: rasa-pro 3.21.0.dev5 # rasa-version-ignore: must not hide the current pin
> Installation: rasa-pro 3.21.0.dev5 # rasa-version-ignore: must not hide the current pin

The old-reader output and repaired-reader output show the difference:

Only the formatting changed. On the old reader, the bullet made the classifier call the line historical. It removed that line before looking for an installation version. A missing README version was permitted, so the checker accepted a project whose setup instructions disagreed with its manifest and lock.

That is the useful failure to test in another updater: does an exception erase a current requirement before the validator sees it? A test that only proves the old assessment stayed unchanged misses this direction entirely.

The reproduction script asserts that the real manifest and lock readers both see the target version before testing the README. Each output names the exact revision and source hash. The original job failure and this fresh comparison are separate recorded runs.

Keep an explicitly marked assessment such as this unchanged:

Verified with: rasa-pro 3.21.0.dev5, Python 3.12 # rasa-version-ignore: historical assessment

The pinned regression tests also require unmarked stale assessments to fail. Markers cannot exempt a stale manifest or lock. Keep the original command, source revision, and output beside a dated result when possible.

Keep the exception narrower than the check

Ignoring every line with a history marker would create another bug. Someone could attach the marker to an outdated install instruction and hide real drift.

The shared classification function therefore gives the recognized installation field precedence:

def is_historical_version_line(line: str) -> bool:
    """Explicit history can be preserved; a current Installation field cannot."""
    return VERSION_IGNORE_MARKER in line and INSTALLATION_VERSION_RE.search(line) is None

The README reader, migration, and lint use that same classification. The reader looks for the current installation field before falling back to an unmarked verification field. The migration leaves explicitly historical lines unchanged.

The fix uses one installation grammar for both classification and version extraction. The negative control supplies indentation, a bullet, and a block quote with a misleading history marker. It checks that drift is reported, migration updates the version, and the repaired document passes. Sharing the grammar keeps the exception from silently recognizing fewer fields than the reader.

This parser recognizes the repository’s documented fields. Other version claims in prose need a human to classify them. Use a structured, immutable result file for new test evidence when you can.

Keep the observation and check the setup

Keep the historical assessment unchanged. Update and check the current installation requirement and lock. A history marker does not exempt either current dependency claim. Qualify the upgraded agent separately before recording a new assessment.

Run the pinned offline checks

Use Python 3.12 and Git. Check out the exact source below; the command runs the repository’s maintenance tests, not its agent examples.

git clone https://github.com/RasaHQ/rasa-community-resources.git
cd rasa-community-resources
git checkout 0285d2b86cc3d1fde13881ad68c5288429bce8e4

The recorded rerun on November 12, 2026 ended with:

----------------------------------------------------------------------
Ran 11 tests in 0.131s

OK

The suite creates temporary test projects. It checks migration ordering and failure handling as well as historical README fields. Its dependency resolver calls are mocked. It installs no Rasa version, trains no model, and makes no provider call. The elapsed time describes this one local run.

Before your next dependency update, give the maintainer one test for preserving a dated result and another for rejecting a stale setup command. Ask the release owner which new runtime evidence is required. The release-decision guide helps turn those results into a decision; the evaluation tutorial covers the agent checks themselves.