Now that cargo init has given us raw materials, it’s time to fill in some initial proof-of-concept content. In effect we have been given an deflated balloon – all that we need, but no space inside. It’s time to blow up the balloon with enough infrastructure that we can “just write code.”

Tests

There are a lot of kinds of tests, but nearly all of them fall into three categories:

  • Unit tests test that a single module’s behavior agrees with its documentation and does not emit surprising errors.
    • A special case of unit test is a “smoke test” – a trivial test that a function can be called or an object created and destroyed (also called a “lifecycle test”). Smoke tests are more pedantically categorized as integration tests (e.g. constructability of an object is higher-level than the object’s Liskov-specification) but in the same ways that olives and cucumbers are “fruits” rather than “vegetables”.
    • Unit tests deliberately introduce API rigidity; you cannot change the API without changing the tests, and rightly so! So if your API embodies a larger principle that would be true of all possible APIs, you might want an integration test instead.
  • Integration tests test that large sections of the code, like entire libraries, work together to accomplish caller-relevant tasks.
    • Integration tests are the tests that stay mostly the same when you refactor a module: Refactoring often changes lower-level APIs, but the same overall narrative purpose remains.
    • A common informal distinction is that unit tests complete in less than a second on most platforms; anything bigger than that needs a higher level test because you’re likely going to end up refactoring it for performance later.
  • System tests invoke user-level affordances like binaries under realistic or simulated conditions as though a user were carrying out a supported action.
    • A common informal rule is that system tests may be too long to run in pre-merge CI, or may run only on specific platforms; this should not be true of integration tests which should remain a part of developers' daily workflows.

I know my tests, but I’m less familiar with Rust’s conventions around them. Here’s my best first pass at how Rust does this, and how we’ll use it.

Unit tests and integration tests

Cargo’s idiosyncratic convention is that unit tests live near code, while integration tests live parallel to code. That is, you can put your module unit tests right in the same file as your module, or nextdoor. Tests that test multiple modules or entire libraries or binaries, though, live in a tests directory sibling to src.

(Test helper tools conventionally live in modules beneath a tests directory to prevent them being autodiscovered as tests; this is a rather dubious layout that doesn’t help with unit test helpers, so we’ll ignore this for now.)

System tests

System tests are a different beast; because they simulate user behavior rather than going through your module APIs, they exist at a fundamentally different level than our source code. This is why many large low-level codebases have system tests written in python, shell scripting, or even perl for their binary artifacts.

Higher-level languages like python are particularly good for system testing because of their fluent syntax for text manipulation.

Cargo doesn’t provide hooks for system testing, because that isn’t its job – system tests exist above built artifacts. There are several cargo modules to allow higher-level scripting (cargo-run-script is notable here) but we will follow Rico’s Law here and use a scripting language rather than embed scripting in a config language.

Special case: Testing main

main is a function, and therefore an API subject to testing. It by definition has behavior that changes during development (at some level of generality all feature development exists principally to change the behavior of main), so we’ll call that an integration test.

An understandable gap in Rust’s test framework is that main.rs is the one file that cannot be tested from within it.

As such we have an obligation to make main vacuous of all nontrivial logic. That is done by moving the interesting content into a lib.rs file.

This PR

Cargo’s hello-world setup only gave us a main.rs, no modules, so there’s no place for unit tests so far. So we’ll start off with an integration test, just to pave the ground a little bit. Since main.rs currently just does a “Hello, world!”, that’s what we’ll test for.

So this PR adds:

  • Indirection of main to make it unit-testable,
  • An example unit test of the main function,
  • An example integration test of the main function, and
  • An example system test of the main binary target.

After this PR:

  • cargo check passes (our build configuration is sound)
  • cargo clippy passes (we are lint-free)
  • cargo test passes (our units and integration points are sound)
  • ./tests/system_smoke_test.py passes (our sole system test passes)

None of this is of any real significance; it’s just laying out some working infrastructure. We’re still a couple of PRs away from “real” code, but a sound foundation pays for itself.

This blog post corresponds to repository state post_03

Lunar metadata: This is a contraction phase; the density of the codebase grows.