Unit Testing and CI/CD

What is Unit Testing and CI/CD

Unit testing means running automated checks that verify individual pieces of your code behave correctly. Each test calls a specific function or endpoint with known inputs and asserts the output matches what you expect. Tests catch regressions — bugs introduced by a change that accidentally breaks something that was previously working.

This project currently has unit tests covering:

  • Service layer (tests/services/) — grading logic, XML parsing, prompt construction, authentication, and API routes
  • MCP server (tests/mcp/) — course XML authoring tools, validation, and example browsing
  • Scripts (tests/scripts/) — CLI utilities for building solution packages and question files

More tests will be added in the future.

There is also a separate UI test suite (tests/ui/) that drives a real browser using Playwright. These tests are slower and require additional setup, so they are excluded from the standard test run.

And there are two suites for grading tests — the instructor-authored cases described in Testing your grading. They are split by what they cost, and both sit on the same module (llmgrader/services/gradetests.py), so a case that passes in one passes in the other for the same reason:

  • tests/services/test_gradetests_static.py runs in the standard test run. It makes no API calls and needs no key: it checks each course test file against the unit it targets — qtags that no longer exist, stale part labels and rubric ids, bands that can never be satisfied, assertions in the wrong form for the question’s grading mode, rubric items no case covers. This is the CI gate for course content.
  • tests/live/test_course_cases.py grades every case for real, one named test per case. It is gated like the rest of tests/live (see below) and costs money.

CI/CD (Continuous Integration / Continuous Deployment) means every time you push a commit or open a pull request, GitHub automatically runs your tests on a clean machine and reports whether they pass. “Continuous Integration” refers specifically to this automatic checking step. The “CD” (Continuous Deployment) part — automatically publishing a passing build — is not currently configured.

Running Unit Tests Locally

Prerequisites

Follow the developer instructions for creating a virtual environment with llmgrader as an editable package.

If you don’t already have the test packages, install them:

pip install pytest pytest-flask pytest-mock

Running the tests

Activate the virtual environment, and run all unit tests (excluding the browser-based UI suite):

pytest --ignore=tests/ui -v

Run a single test file:

pytest tests/services/test_unit_parser.py -v

Run a single test by name:

pytest tests/services/test_unit_parser.py::test_validate_unit_file_accepts_demo_unit

Simulating CI locally

CI runs in a clean environment with only the dependencies in pyproject.toml. You can replicate this exactly before pushing to catch any missing dependencies early:

python -m venv ci-test-env
ci-test-env\Scripts\activate        # Windows
# source ci-test-env/bin/activate   # macOS / Linux

pip install -e .
pip install pytest pytest-flask pytest-mock
pytest --ignore=tests/ui -v

deactivate

Remove the environment when done:

rmdir /s /q ci-test-env   # Windows
# rm -rf ci-test-env       # macOS / Linux

UI Tests

The UI test suite (tests/ui/) drives a real Chromium browser using Playwright and is excluded from the standard test run for speed. CI does not run these tests, but developers are encouraged to run them locally before merging changes that touch the frontend.

Install the browser once:

playwright install chromium

Then run the suite:

pytest tests/ui/ -v

Live model tests

tests/live/ puts real requests on the OpenAI API and costs real money, so it is gated twice over and deselected from a bare pytest run by -m 'not live' in pyproject.toml. Running it is explicit:

LLMGRADER_RUN_LIVE_TESTS=1 OPENAI_API_KEY=... pytest tests/live -m live

Both conditions are required, and a missing one skips rather than fails, so a developer with no key can run the whole suite and see it skip cleanly.

Two things live there:

  • test_models_live.py grades a fixture unit on every model in the registry. This is what catches a retired or renamed model id, and it produces the cost/latency table the next slate refresh is argued from.
  • test_course_cases.py runs the course’s own grading tests, one named test per <case>. By default each question is graded with its own preferred_model, so the run tests exactly what students hit. To run it cheaply, force a tier:
LLMGRADER_RUN_LIVE_TESTS=1 LLMGRADER_GRADETEST_MODEL=simple pytest tests/live/test_course_cases.py -m live

Both write into a throwaway SQLite database rather than your local_data/, so a run leaves no fake submissions behind to show up in the dashboard.

CI/CD in GitHub

How it works

When you push a commit or open a pull request targeting main, GitHub automatically:

  1. Spins up a fresh Ubuntu virtual machine
  2. Checks out your code
  3. Installs Python 3.12 and the project dependencies
  4. Runs pytest --ignore=tests/ui -v
  5. Reports the result back to GitHub

This is defined in .github/workflows/ci.yml. You don’t need to configure anything on the GitHub website — GitHub detects the file automatically.

Pip packages are cached between runs (keyed to pyproject.toml), so subsequent runs skip re-downloading dependencies and complete faster.

What you see on a check-in

After pushing, go to your commit on GitHub. You will see a small icon next to the commit hash:

  • Yellow circle — tests are running
  • Green checkmark — all tests passed
  • Red X — one or more tests failed

On a pull request, the same status appears at the bottom of the PR page under “Checks”. You can click through to see the full test output, including which test failed and the error message.

What happens if tests fail

The red X is informational — it does not block you from merging by default. However, it is a signal that something is broken and should be investigated before merging.

To see what failed:

  1. Click the red X on the commit or the “Details” link on the PR checks
  2. Open the unit-tests job
  3. Expand the “Run unit tests” step to see the full pytest output

Fix the failing test locally, push the fix, and a new check run starts automatically.

If you want to enforce that tests must pass before merging, enable branch protection rules under Settings → Branches → Add rule and check “Require status checks to pass before merging”.

This site uses Just the Docs, a documentation theme for Jekyll.