The client part of the testing environment. This program runs on the host and connects over a virtual serial port to the test-runner.

The tests themselves are defined in this package, using the interface provided by test-runner.

The server part of the testing environment. This program runs in guest VMs and provides the test-manager with the building blocks (RPCs) needed to create tests.

A support library for the other two packages. Defines an RPC interface, transports, shared types, etc.

For macOS, the host machine must be macOS. All other platforms assume that the host is Linux.

• Get the latest stable Rust from https://rustup.rs/.

Normally, you would use Tart here. It can be installed with Homebrew. You'll also need wireguard-tools, a protobuf compiler, and OpenSSL:

brew install cirruslabs/cli/tart wireguard-tools pkg-config openssl protobuf

Wireshark is also required. More specifically, you'll need wireshark-chmodbpf, which can be found in the Wireshark installer here: https://www.wireshark.org/download.html

You also need to add the current user to the access_bpf group:

dseditgroup -o edit -a THISUSER -t user access_bpf

This lets us monitor traffic on network interfaces without root access.

For running tests on Linux and Windows guests, you will need these tools and libraries:

dnf install git gcc protobuf-devel libpcap-devel qemu \
    podman mingw64-gcc mingw64-winpthreads-static mtools \
    golang-github-rootless-containers-rootlesskit slirp4netns dnsmasq \
    dbus-devel pkgconf-pkg-config swtpm edk2-ovmf \
    wireguard-tools

rustup target add x86_64-pc-windows-gnu

Building the test-runner binary is done with the build.sh script. Currently, only x86_64 platforms are supported for Windows/Linux and ARM64 (Apple Silicon) for macOS.

The build.sh requires the $TARGET environment variable to be set. For example, building test-runner for Linux would look like this:

TARGET=x86_64-unknown-linux-gnu ./build.sh

For a Linux target podman is required to build the test-runner. See the Linux section under Prerequisities for more details.

TARGET=x86_64-unknown-linux-gnu ./build.sh

TARGET=aarch64-apple-darwin ./build.sh

The test-runner binary for Windows may be cross-compiled from a Linux host.

TARGET=x86_64-pc-windows-gnu ./build.sh

See BUILD_OS_IMAGE.md for how to build images for running tests on.

See cargo run --bin test-manager for details.

Here is an example of how to create a new OS configuration and then run all tests:

# Create or edit configuration
# The image is assumed to contain a test runner service set up as described in ./docs/BUILD_OS_IMAGE.md
cargo run --bin test-manager set debian11 qemu ./os-images/debian11.qcow2 linux \
    --package-type deb --architecture x64 \
    --provisioner ssh --ssh-user test --ssh-password test

# Try it out to see if it works
cargo run --bin test-manager run-vm debian11

# Run all tests
cargo run --bin test-manager run-tests debian11 \
    --display \
    --account 0123456789 \
    --current-app <git hash or tag> \
    --previous-app 2023.2

Here is an example of how to create a new OS configuration (on Apple Silicon) and then run all tests:

# Download some VM image
tart clone ghcr.io/cirruslabs/macos-ventura-base:latest ventura-base

# Create or edit configuration
# Use SSH to deploy the test runner since the image doesn't contain a runner
cargo run --bin test-manager set macos-ventura tart ventura-base macos \
    --architecture aarch64 \
    --provisioner ssh --ssh-user admin --ssh-password admin

# Try it out to see if it works
#cargo run -p test-manager run-vm macos-ventura

# Run all tests
cargo run --bin test-manager run-tests macos-ventura \
    --display \
    --account 0123456789 \
    --current-app <git hash or tag> \
    --previous-app 2023.2

Account tokens are read (newline-delimited) from the path specified by the environment variable ACCOUNT_TOKENS. Round robin is used to select an account for each VM.