Downloading and testing tig algorithms

Downloading and testing tig algorithms

This guide is for anyone who wants to download and run an existing TIG algorithm locally using test_algorithm. You do not need to write, compile, or submit any code.

Before you begin, you will need the challenge name and algorithm name. This guide assumes that these have already been provided to you.

Install Docker

Install Docker and make sure that the Docker service is running.

For a GPU challenge, you will also need an NVIDIA GPU configured for use with Docker. TIG’s GPU challenges are vector_search, hypergraph, and neuralnet_optimizer.

Create a working directory

Create a directory for the downloaded algorithm, then change into it:

mkdir tig-algorithm-testing
cd tig-algorithm-testing

The Docker commands in the next step mount this directory at /app in the container. This means that the downloaded algorithm remains available after you exit the container.

Start the runtime container

TIG provides a Docker image for each challenge. Replace <CHALLENGE> with the challenge name you were given, such as satisfiability or knapsack.

docker run --rm -it -v "$(pwd):/app" ghcr.io/tig-foundation/tig-monorepo/<CHALLENGE>/runtime:latest

The remaining commands in this guide should be run inside the Docker container.

Download the algorithm

Replace <ALGORITHM> with the algorithm name you were given:

download_algorithm <ALGORITHM>

This downloads the algorithm from mainnet into the working directory.

Read the algorithm’s help

The algorithm’s author may have provided recommended hyperparameters or other usage notes. Display them with:

help_algorithm <ALGORITHM>

Test the algorithm

Run the algorithm with:

test_algorithm <ALGORITHM> <TRACK> <HYPERPARAMETERS>

For an initial test, use an active track for the challenge and pass null to use the algorithm’s default hyperparameters:

test_algorithm <ALGORITHM> <TRACK> null

A track identifies the type of challenge instances used for the test. Track IDs are comma-separated parameters, such as n_items=1000,budget=5 for a Knapsack track. Copy an active track ID from Available Challenges.

Hyperparameters are algorithm-specific settings. To see the available hyperparameters, inspect the algorithm’s source code. Pass null to use the defaults, or pass a JSON string when the algorithm’s help provides other settings. For example:

test_algorithm <ALGORITHM> <TRACK> '{"example_setting":123}'

Understanding the results

By default, test_algorithm runs 100 nonces and continually reports how many are processing, finished, or invalid. A nonce produces a distinct challenge instance.

avg_quality is the average quality of the valid solutions produced by the tested nonces. Quality is displayed as an integer scaled by 1,000,000. For example:

100,000 -> 0.100000
⚠️

An invalid solution causes the test to stop by default. The output will identify it as invalid; use --verbose to see more detail.

Useful test options

Use --help to see every option supported by test_algorithm:

test_algorithm --help

Common options include:

  • --verbose shows full output from the algorithm and prints the individual tig-runtime and tig-verifier commands. This is useful for diagnosing errors.
  • --nonces <NUMBER> changes the number of nonces tested from the default of 100.
  • --workers <NUMBER> runs multiple nonces concurrently.
  • --fuel <FUEL> sets the maximum fuel available to each nonce. The default is 100 billion fuel.
  • --seed <SEED> and --start <NONCE> let you repeat a particular set of test instances.
  • --ignore-invalid continues the test after an invalid solution. Invalid solutions will still cause real benchmarks to fail.

For example, this runs 10 nonces with verbose output:

test_algorithm <ALGORITHM> <TRACK> null --nonces 10 --verbose

When you are finished, run exit to leave the Docker container. The downloaded algorithm remains in your tig-algorithm-testing directory and will be available the next time you start the same challenge container from that directory.

Collecting per-nonce results

The standard test_algorithm command does not retain its per-nonce data. Download test_algorithm_results.py into your tig-algorithm-testing directory if you want to save the raw results for later analysis.

Run it inside the same runtime container after downloading the algorithm:

python3 test_algorithm_results.py <ALGORITHM> <TRACK> null

The script saves test_algorithm_results.json in the current directory. It contains a record for every nonce with:

  • Quality, when the solution was successfully verified
  • Runtime, verifier, and total elapsed time in milliseconds
  • Fuel consumed, when an output was produced
  • Runtime and verifier exit codes
  • Any runtime, verification, output, or script errors

Runtime time covers the complete tig-runtime process, including process startup, instance generation, algorithm execution, and output writing. Verifier time covers the complete tig-verifier process, including instance regeneration and solution evaluation. These are not pure algorithm execution timings.

The JSON contains raw per-nonce values rather than aggregate statistics, allowing you to choose how to filter and analyse the results. Use --output <PATH> to change the output filename or location.

The script uses one worker by default. You can run more nonces concurrently with --workers, but resource contention makes the timing results less accurate:

python3 test_algorithm_results.py <ALGORITHM> <TRACK> null --workers 4
Challenge OwnersBenchmarkers