Performance

Sprite measures itself with the benchmark harness under tests/lib/benchmarks on the programs under tests/data/curry/benchmarks. Section 9 of tests/README describes the suites, the records, and the comparison of two record files. The records of the performance program are under tests/data/curry/benchmarks/results, one JSON Lines file per suite and run, named by the label of the run.

The harness

The driver is tests/run_benchmarks. A run names a suite, the backends, the repetitions and the programs, and appends one record per program and backend to a file:

cd tests
./run_benchmarks -s throughput -b cxx -r 3 --label NAME -o NAME.jsonl
./run_benchmarks -s throughput --nightly -r 3 --timeout 300 -o t.jsonl
./run_benchmarks list -s throughput

The suites are:

  • throughput: the evaluation of main of each program in a new process.

  • compile: each program from source with the ICurry cache cold and warm, and the expression 1+2.

  • import: the start of the interpreter and the first imports.

  • memory: the peak memory with the collector on and off.

  • split: a search program whole and in parts.

Every run is a child process under a cap on its address space and a timeout, in step mode of the rotation, so the counters of a record reproduce. A record holds the medians of the repetitions: wall and CPU seconds, the peak resident set, the seconds of the evaluation alone, the compile seconds, the counters steps, forks and collections, and the instructions retired when perf is available.

Two record files are compared with:

./run_benchmarks compare OLD NEW
./run_benchmarks compare --deterministic OLD NEW

The counters must be equal: a change in them is a change in the program or the compiler, not in the machine. --deterministic decides by the instructions, with the seconds as advisory ratios; CPU seconds decide only on a quiet machine, and never across machines.

The records

baseline-2026-10-03

The baseline before the performance program, at commit 1780ae09, on a quiet 12-core x86-64 workstation: the four suites, the C++ backend with five repetitions, the reference backend of the developer notes with one, and PAKCS on nine programs. On the C++ backend the 30 programs of the dissertation took between 0.43 s (PaliFunPats) and 42 s (Psort) of CPU. The two Poker programs failed on every backend for lack of the Curry preprocessor.

phase2-2026-10-04

The record of the C++ backend after Phase 2 of the program: the in-place rewrite, the block heap, the weak free-variable table, the interpreter. It was taken at commit 00f5472a with the changes in the working tree, on the same machine with a load average near 2 from other work. It holds the nightly set of ten programs, three repetitions, and the memory suite. The table gives the throughput record, CPU seconds of the whole process, the seconds of the evaluation alone, and the peak resident set.

Program

CPU s

eval s

steps

forks

collections

peak MB

Tak1

1.302

1.131

33629915

0

30

67.9

Fib

0.347

0.179

3500006

0

2

120.5

Reverse

0.607

0.427

8407963

0

8

63.2

Primes

2.299

2.125

61364444

0

64

64.9

Queens10

1.736

1.558

54505073

0

54

63.4

PermSort

5.001

4.786

12531154

1588167

9

327.5

SearchQueens

4.977

4.768

41373250

1084883

20

316.2

QueensSet

0.772

0.599

2152979

511583

3

110.2

QueensSet9

5.330

4.990

12143480

2979048

11

337.0

Last

0.432

0.263

3100017

300002

4

66.5

The memory record of the same label runs each program with the collector on and off. Tak1 peaks at 67.8 MB with it and 1009 MB without. Primes peaks at 65 MB and 1863 MB, and Queens10 at 63 MB and 1512 MB. Both files are of schema 1, without the instructions column; a comparison by instructions needs a newer baseline. The later entries of the TODO (the saturation of apply chains, the repair of the set functions, tiered execution) moved the counters of some of these programs; a compare against this record reports them.

The step cost

The runtime counts a rewrite step at every return of a step function. The dated entries of the TODO give these costs per completed step on Tak1 (33.6 million steps):

  • The rotation check of the scheduler: 3 instructions per step in step mode and 5 in time mode, over the build before the ticker.

  • The write counters of the collector (make GC_WRITE_COUNTERS=1): 36 instructions per step, which is why the default build leaves them out.

  • The scheduler counters (make COUNTERS=1): one to three percent of the instructions and a word per node.

sprite-exec --stats prints the steps of a run, and the column instr_m of the harness the instructions in millions, so the quotient of a run is at hand.

Nightly history

A scheduled GitHub Actions job (.github/workflows/perf.yml) runs the harness every night on the C++ backend. It runs the throughput suite on a fixed set of ten programs with three repetitions, the compile suite on the same programs and the expression item, the import suite, and the applications suite: two programs of the examples against the tools of their trade (the dependency solver of example 15 against resolvelib, the resolver of pip, and the overload resolution of example 21 against a plain-Python implementation of the same ranking). The same measurement runs by hand with tests/run_benchmarks --nightly. The job appends the records to the branch perf-history of the repository with the command run_benchmarks history DIR FILE... and compares each record with the previous one of its item in the summary of the run. The numbers come from shared runners and are advisory: the job never fails on a timing, only on a run that failed. Read the CPU seconds with the exact counters, the rewrite steps and the forks: a change in them is a change in the program or the compiler, not in the machine.

The page Performance history draws one figure per suite from that branch: one small panel per program with the metric over time, the last value, a hollow marker where the steps changed, a cross where a run failed, a readout on hover with the commit and the counters, and a table view of the last run of every item against the previous one. The metric and the time range are selectable. The page is plain HTML and JavaScript; it reads points.json from the history branch.

Other systems

The baseline record holds the CPU seconds of PAKCS on nine programs beside those of Sprite; run_benchmarks -b pakcs measures it. This page makes no claim against other compilers.