.. highlight:: bash =============== Developer Setup =============== This page sets up a machine for work on Sprite itself. One script does the work: ``scripts/setup-dev-machine.sh``. It puts every bulky product under one scratch prefix, builds the pinned toolchain, configures and stages Sprite, and runs a smoke test. The script never runs ``sudo``. When an apt package is missing, it prints the apt command and stops. Quick start =========== Clone the repository, print the plan, then run it:: git clone https://github.com/andyjost/Sprite.git cd Sprite scripts/setup-dev-machine.sh --prefix /path/to/scratch --dry-run scripts/setup-dev-machine.sh --prefix /path/to/scratch The dry run prints every command and runs none. The real run stops at the first problem. Fix the cause, then run the same command again: a finished step is skipped, and configure runs again only when its flags changed or ``--reconfigure`` asks for it. The whole run takes under an hour on a current machine. Most of that time goes to the PAKCS build, the conda solve, and the compilation of the Curry library for the C++ backend. The steps ========= The script runs nine steps in this order. 1. Machine. Prints the OS, the cores, the memory, the swap, and the tools it found. ``--jobs auto`` stays ``auto``: ``make`` counts the processors when it starts, so no count of this machine enters ``Make.config``. 2. apt packages. Checks ``git swi-prolog-nox g++ make curl ccache time`` and the ``linux-tools`` packages of the running kernel, which provide ``perf``. SWI-Prolog serves PAKCS alone: the build of its saved states in step 3 and the test oracle ``pakcs``. Sprite itself runs no Prolog, and its C++ runtime needs no Boost. ``time`` is GNU time, which the benchmark harness uses for the peak memory of a run. A missing package stops the script. The printed command installs the missing ones. ``--skip-apt`` skips the check on a system without dpkg. 3. PAKCS 3.4.1. Downloads the binary archive from curry-lang.org into ``PREFIX/downloads`` and verifies its SHA-256. The hash is the one of ``conda/curry-frontend/meta.yaml``. The archive unpacks to ``PREFIX/pakcs-3.4.1``. The script gives SWI-Prolog 9 the stack limit that the distribution sets for version 8, then runs ``make`` there. The build takes about five minutes. Sprite takes the Curry front end and the test oracle from this tree. 4. Checkout. Clones the repository when ``--repo`` names a missing directory. Initializes the pybind11 submodule only. Links ``install`` and ``object-root`` of the checkout to ``PREFIX/install`` and ``PREFIX/object-root``. A link that points elsewhere stops the script. The script deletes nothing. 5. Conda environment. Creates ``PREFIX/conda/env`` from ``conda/dev-environment.yml`` of the checkout with micromamba, mamba, or conda, whichever is found first. The package cache goes to ``PREFIX/conda/pkgs``. The environment holds Python 3.14, numpy, Sphinx, and the Sphinx theme, and nothing else. 6. configure. Runs ``configure`` in the clean environment with the system compilers, the PAKCS of step 3, and the Python of step 5. The script probes ``configure --help`` and adds ``--jobs`` and ``--with-ccache`` when configure supports them. configure empties ``install`` and ``object-root``, so the script records the flags in ``PREFIX/configure-args`` and skips the step while ``Make.config`` exists and the flags are the same. ``--reconfigure`` forces it. 7. ``make stage``. Builds the runtime and compiles the Curry library for both backends. Then ``make overlay`` extracts the archived products of the test programs, after the stage so that they are newer than the installed library interfaces. 8. Smoke test. Runs one fast unit test file on each backend under the address-space cap and the time limit of the test drivers. 9. Next steps. Prints the paths for the personal rules file and the runs to make next. With ``--icurry`` the script also installs ``icurry`` 3.1.0 through the Curry Package Manager, after step 3. That is the optional second route from Curry to ICurry. The step follows the CI script ``.github/scripts/install-curry-toolchain.sh``. It writes ``~/.cpmrc`` only when the file is missing, with the CPM home under ``PREFIX/cpm``. The clean environment ===================== A conda toolchain in the login shell exports ``CC``, ``CXX``, ``CFLAGS``, ``CXXFLAGS``, ``CPPFLAGS``, and ``LDFLAGS``. The Makefiles of Sprite append those flags, so a build in that shell mixes two toolchains. The script runs configure, make, and the smoke test under ``env -i`` with five variables: ``PATH=/usr/local/bin:/usr/bin:/bin``, ``HOME``, ``LC_ALL=C.UTF-8``, ``TMPDIR``, and ``CCACHE_DIR=PREFIX/ccache``. Use the same command for every later build:: env -i PATH=/usr/local/bin:/usr/bin:/bin HOME=$HOME LC_ALL=C.UTF-8 CCACHE_DIR=PREFIX/ccache make stage ``CCACHE_DIR`` keeps the compiler cache under the prefix, not under ``HOME``. With ``configure --with-ccache`` the compiler of the installation runs ccache too, so every test and every program that compiles a module for the C++ backend reads the variable. For a shell outside the clean environment, say once ``ccache --set-config cache_dir=PREFIX/ccache``. ``SPRITE_SETUP_CLEAN_PATH`` replaces the PATH of the clean environment. Layout of the prefix ==================== ================================ ============================================== Path Content ================================ ============================================== ``PREFIX/downloads`` the PAKCS archive ``PREFIX/pakcs-3.4.1`` PAKCS: ``bin/pakcs``, ``bin/pakcs-frontend``, ``bin/cypm`` ``PREFIX/conda/env`` the conda environment; ``bin/python`` is the Python of Sprite ``PREFIX/conda/pkgs`` the conda package cache ``PREFIX/install`` the staged install, linked from the checkout ``PREFIX/object-root`` the object tree, linked from the checkout ``PREFIX/cpm`` the CPM home, with ``--icurry`` ``PREFIX/ccache`` the compiler cache (``CCACHE_DIR`` of the clean environment) ``PREFIX/configure-args`` the flags of the last configure run ================================ ============================================== Options ======= =================== ========================================================= Option Meaning =================== ========================================================= ``--prefix DIR`` the scratch root; required ``--repo DIR`` the checkout; cloned when missing; the default is the checkout that holds the script ``--repo-url URL`` the clone URL ``--branch NAME`` the branch to clone ``--jobs N|auto`` the job count for configure, when it supports ``--jobs``; auto is one job per processor, counted by ``make`` when it starts ``--reconfigure`` run configure although ``Make.config`` exists and the flags are the ones of the last run; ``make stage`` then rebuilds everything ``--dry-run`` print every command, run none ``--skip-apt`` do not check the apt packages ``--icurry`` also install icurry 3.1.0 through cypm ``--smoke-file F`` the unit test file of the smoke test; the default is ``unit_expr.py`` ``--cap-kb N`` the address-space cap of each smoke run in KiB, or ``unlimited``; the default is the cap of ``tests/run_tests`` ``--timeout SEC`` the time limit of each smoke run ``--cc``, ``--cxx`` the system compilers =================== ========================================================= ``PAKCS_VERSION`` and ``ICURRY_VERSION`` in the environment change the pinned versions. A PAKCS version without a pinned hash gets its size and hash printed instead of a check. Every budget is a parameter: the job count, the memory cap, and the time limit. The script detects the cores and the memory and prints them. Do not copy a number from one machine to the next. After the setup =============== The last step prints the same list with the paths of the machine. 1. Calibrate the test manifest on the new machine. Run every test file once per backend on a quiet machine with the manifest update of the test runner:: cd tests ./run_tests -j 1 --backend both --update-manifest The durations and the peak memory of every test file must be measured there, then committed in ``tests/manifest.json``. See ``./run_tests -h`` and section 10 of ``tests/README``. 2. Record a baseline with the benchmark harness on a quiet machine. The records carry the deterministic columns steps, forks, and collections, which the compare command checks:: cd tests ./run_benchmarks -b cxx -b py --label baseline -o PREFIX/baseline.jsonl ./run_benchmarks counters PREFIX/baseline.jsonl 3. Update the personal rules file with the new paths: the Python, the PAKCS tree, icurry, the install and object trees, the conda package cache, the compiler cache, and the clean environment. Record the cores and the memory. 4. Copy ``tests/.cache/icurry.db`` from the old machine. The cache is keyed by the source text, so the first suite run then compiles nothing. Notes ===== - PAKCS, cypm, and icurry write under ``/tmp``. A sandbox that mounts ``/tmp`` read-only breaks them. - The Kiel mirrors of the Curry Package Manager do not answer. The script writes a ``~/.cpmrc`` that names ``cpm.curry-lang.org``. - The script was written for Ubuntu 24.04. On another distribution, install the equivalents of the apt packages and pass ``--skip-apt``. - ``tests/unit_setup_script.py`` runs the script with ``--dry-run`` and checks the plan. It needs no network and installs nothing.