curry.backends.cxx.toolchain module

The build steps of the C++ backend: ICurry-JSON to C++, and C++ to a shared object.

The C++ compiler is the slow step. Every generated module includes cyrt/cyrt.hpp, and parsing that header costs about two thirds of the time g++ needs for a small module. So the toolchain precompiles the header once, and g++ loads the result. See PrecompiledHeader.

Generated code comes in two flavors, as the runtime does (make DEBUG=1). The release flavor drops the assertions, the stack protector, and the procedure linkage table. The debug flavor keeps the assertions and compiles for a debugger. A module follows the flavor of the installed runtime (config.cxx_flavor) unless the interpreter flag debug is set. See FLAVOR_FLAGS and Cpp2So.flavor.

A compiled module stays valid as long as the runtime headers it was compiled against, the flags of its flavor and the installation it links against do not change. The headers are those the generated code includes: the include closure of cyrt/cyrt.hpp (runtime_headers). A header of the runtime outside that closure, one the runtime library alone uses, is not part of the stamp, so an edit to it rebuilds the library and no module. Cpp2So records the digest of the closure and of the flags, and the installation as text, beside each shared object (the ABI stamp, <module>.so.abi) and compiles the module again when the installation gives another digest or another path. See runtime_digest, object_digest, and Cpp2So.is_stale. A generated .cpp file carries a format stamp; Json2Cpp, which writes the file, refuses one of another format. See Json2Cpp.is_stale.

An object names no path, so it serves any tree (format 13 of the generated code). Each object carries a SONAME, the full name of its module and the format version, “sprite-<module>.so.<format>” (soname); the linker records the SONAMEs of the objects of the imports as the NEEDED entries of an object, and the loader imports those modules before it opens the object, so the dynamic linker finds each name mapped (loader.load_module). The record of a module names its source relative to the installation or to the object (compiler._source_file_name). See Cpp2So for the link line.

A compile stores its products in the product cache, and a plan places the cached products of a module instead of generating and compiling it when the cache holds them. See Cpp2So.restore, Cpp2So.store and curry.toolchain._productcache.

class curry.backends.cxx.toolchain.Cpp2So(interp)

Bases: object

Compiles a generated .cpp file into a shared object, in the flavor of the installed runtime or, under the interpreter flag debug, in the debug flavor (see flavor). Each object gets an ABI stamp, a text file beside it (<module>.so.abi) of two lines:

<digest>
<installation>

The first line is the digest of the runtime headers the generated code includes and of the flags the object was compiled with (object_digest), 16 hex digits. The second line is the real path of the installation the object was compiled under (installation_path). An object is current when the digest is accepted (accepted_digests) and the path is the installation of the process (is_stale). So a copy of an installation made by hand keeps objects whose stamps name the original and compiles them again, and a package whose manager rewrites the path at install time (conda lists the stamps in info/has_prefix as text files) keeps its objects. A stamp of one line, written before the path joined it, is stale.

The link line names no path that the object keeps. The object gets the SONAME of its module (soname; the module name comes from the header of the generated file, “// MODULE: N”), the objects of its imports stand on the line by their files and enter the object as their SONAMEs (the NEEDED entries; every import, in the order of the IMPORTS line, see LINK_FLAGS), and the runtime library enters as “libcyrt.so”, which the extension module of the backend has loaded. So readelf -d shows no absolute path, and an object compiled in one tree loads in a copy of the tree at another path, or from the product cache in another tree. A program that opens such an object with dlopen and without the loader must open the objects of its imports first, in dependency order, with the runtime library loaded or on the search path; or it must name, on LD_LIBRARY_PATH, a directory that holds each import under its SONAME.

A compile stores its products in the product cache (store), and the plan places the cached products of a module instead of a compile (restore).

IMPORT_PAT = re.compile('// IMPORTS: (.*)')

MODULE_PAT = re.compile('// MODULE: (\\S+)')

PRODUCT_SUFFIXES = ('.cpp', '.so')

STAMP_SUFFIX = '.abi'

accepted_digests()

The stamps of the objects this step keeps. An object of the installed flavor is always kept. Under the interpreter flag debug an object of the debug flavor is kept as well. So a debug session keeps the objects of the installation and its own, and a session without the flag compiles the debug objects again, once. Empty when the installation holds no headers.

digest()

The stamp this step writes: the object_digest of its flavor.

ends_plan(filename)

Tells whether the plan ends at filename, a generated .cpp file this step would compile. Under the interpreter flag interpret set to ‘tiered’, a module with a generated file and no object (a background compile that was cancelled) is interpreted from the JSON beside the file, which loadcurry reads, and the background compile starts from the generated file. Without the JSON the plan goes on.

property flavor

‘debug’ under the interpreter flag debug or in a debug installation (make DEBUG=1), else ‘release’.

Type:

The flavor this step compiles in

format_version(file_in)

The format stamp of a generated file (format_version).

import_lacks_an_object(sofile, currypath=None)

Tells whether an import of the module of sofile runs without a compiled object. The imports are read from the generated file beside the object, or from the NEEDED entries of the object when the file is gone (sprite-make –tidy removes it; loader.needed_modules), and imported first, on the search path currypath (by default the path of the interpreter). The two lists agree: the linker records every import (LINK_FLAGS).

The object names the objects of its imports as needed libraries, by SONAME (see _dependencies), and the dynamic linker satisfies each name with the object of that name the process has mapped. When an import was edited, its object is stale: under the interpreter flag interpret the import is interpreted from its ICurry, no object of its name is mapped, and the load of the importer would fail. So an object is loaded only when every import was loaded from its object; a module whose import is interpreted is interpreted too (or compiled again, without the flag). An object whose imports cannot be read is trusted; the loader reads them once more before it opens the object (loader.load_module).

is_stale(filename, currypath=None)

Tells whether a cached file of this step is unusable. The plan then starts again from the file before it (Plan.prune_stale). currypath is the search path of the imports of the module (_findcurry.imports_path).

A .cpp file is stale when its format stamp is not the emitter’s (source_is_stale; a file without a stamp is format 1).

A .so file is stale when its ABI stamp is missing, holds a digest this step does not accept (accepted_digests), or names another installation (stamp_is_current): the object was compiled against other headers, with the flags of another flavor, or under another installation. The check reads the headers, not time stamps, so a new copy of the same runtime keeps every object, and a relocated package keeps its objects. An installation without headers (runtime_digest gives None) cannot compile anything, so its objects are trusted as they are. A .so file is stale as well when an import of its module runs without an object (import_lacks_an_object).

prepare_header()

Builds the precompiled header when it is missing or stale. Returns True when g++ can use it, False when the header is disabled, no compiler is installed, or the build failed (then the objects compile without it). sprite-make –jobs calls this once before its children start, so that the children do not each build the header.

product_facts(curryfile)

The facts, as strings, that decide the generated code and the object of a module beyond its source chain: the key of the product cache digests them (see _productcache.product_key). In order: the format of the generated code (compiler.FORMAT_VERSION) and the digest of the sources of the code generator (generator_digest); the keys of the optimizer passes and the inline budget of the interpreter, which shape the code; and the intermediate directory and the route from Curry to ICurry (cache.frontend_digest), which decide the ICurry of a source. No fact names a tree: the products name no path since format 13 (the SONAMEs and the record), so an entry serves every tree of one runtime, and a second worktree compiles nothing the first compiled. Before format 13 the real paths of the installation and of the directory of the source were facts, because the object named both.

product_key(filename, currypath)

The key of the product cache for the module of filename, a file of its chain (see _productcache.product_key), or None when the module has no text to digest.

classmethod read_stamp(sofile)

The digest recorded in the ABI stamp of sofile, or None.

classmethod read_stamp_lines(sofile)

The two lines of the ABI stamp of sofile: the digest and the real path of the installation. A missing line is None; so is each of a missing stamp.

classmethod remove_stamp(sofile)

Removes the ABI stamp of sofile, if there is one.

restore(file_in, currypath, **kwds)

Places the cached products of the module of file_in, a current JSON or generated file of the module, beside it: the generated file, the object, and a stamp for this installation. Returns the object, or None when the cache is off or holds no entry, when the module is excluded (_productcache.excluded), when a recompile is forced (SPRITE_FORCE_RECOMPILE_CXX), when the directory cannot be written, or when an import of the module runs without an object (import_lacks_an_object): the object stays in place for a later plan, which finds it current once the import has its object, but the module is not loaded from it now. Nothing is placed under the interpreter flag interpret set to ‘new’ either: that mode interprets a module without a current object and never compiles it (Json2Cpp.ends_plan), and a cached object would make the module compiled after all. The placement counts on the compile clock. currypath is the search path of the module; kwds are the keywords of the plan, whose is_sourcefile tells whether the imports are searched on it (_findcurry.imports_path).

stamp_is_current(sofile)

Tells whether the ABI stamp of sofile holds a digest this step accepts and names this installation. The paths are compared as real paths, so a stamp that spells the installation through a link (the prefix a package manager wrote) names it all the same. True without a digest to compare (an installation without headers).

classmethod stampfile(sofile)

The ABI stamp of a shared object.

store(cppfile, sofile, currypath)

Stores the products of a compile in the product cache: the generated file, the object and its stamp. Nothing when the cache is off. A cache that cannot be written is logged once and left alone.

write_stamp(sofile)

Records the stamp of this step beside sofile: its digest and the real path of this installation (see the class). Without headers there is no digest, and no stamp is written.

class curry.backends.cxx.toolchain.Json2Cpp(interp)

Bases: Json2TargetSource

NAME = 'json2cpp'

SUFFIX = '.cpp'

ends_plan(filename)

Tells whether the plan ends at filename, a JSON file this step would compile. Under the interpreter flag interpret set to ‘new’, a module that reached the JSON stage without a compiled object is interpreted by the runtime (cyrt/icurry.hpp; see materialize.py), so this step and the C++ compiler do not run for it. Under ‘tiered’ the same holds, and the module is compiled in the background after its load (see tiered.py). A module with a current object never reaches this step: the object is the newest file.

is_stale(filename, currypath=None)

Tells whether a .cpp file this step wrote is unusable (source_is_stale). The plan asks this step when the file ends the plan (sprite-make –cxx); with a compile step after it, Cpp2So asks the same question. The plan asks about the JSON input of this step as well, which is never refused.

class curry.backends.cxx.toolchain.PrecompiledHeader(root, cxx, cxxflags)

Bases: object

The precompiled form of cyrt/cyrt.hpp.

The files live in <root>/cyrt/cyrt.hpp.gch/. The root is the installed include directory unless SPRITE_CXX_PCH_ROOT says otherwise (see config.cxx_pch_root). g++ accepts a directory of that name in place of one file and uses the member that matches its options. So each flavor (the optimized build, the debug build, custom CXXFLAGS) gets its own member, named after the optimization flags and a hash of the compiler and the full flag list.

A member is stale when it is older than any header, its sources. The comparison uses modification times: make stage links the installed headers to the sources, and make install copies a header only when the source is newer, so a build that changes no header keeps the member. A stale member of another flavor is removed when a flavor prepares its own (see remove_stale_members): g++ never compares a member with the headers, and it takes the first member of the directory whose options agree with the compilation. When the directory cannot be written or the build fails, the toolchain logs one warning per root and compiles without the header. The generated code is the same either way.

HEADER = 'cyrt/cyrt.hpp'

build()

Compiles the header into a temporary file, then moves it into place.

property flavor

The member name for this compiler and these flags, without suffix.

static header_files()

The headers the precompiled header depends on: the include closure of cyrt/cyrt.hpp (runtime_headers). A header outside it never makes a member stale.

is_current()

True when the member exists and no header is newer. False when a header of the closure cannot be read (PrerequisiteError, an OSError): the build then fails on the same header, and prepare() logs its one warning.

prepare()

Builds the member if it is missing or stale. Returns True when g++ can use it, and False when the toolchain must compile without it.

remove_stale_members()

Removes the other members of the directory that are older than a header. g++ tries the members in the order of the directory and uses the first one whose options agree with the compilation; it never compares a member with the headers. So when a compilation defines a macro that the current member was built without (the macro of the write counters, or of the Memory Pool System), g++ rejects that member and may take a stale one of an earlier runtime, and the module is compiled against old headers. A flavor whose member is removed builds it again. A member that another process reads stays readable after the unlink.

curry.backends.cxx.toolchain.extend_plan_skeleton(interp, skeleton)

curry.backends.cxx.toolchain.flavor_flags(flavor)

The compiler flags of a flavor of generated code: ‘release’ or ‘debug’.

curry.backends.cxx.toolchain.format_version(file_in)

The format stamp of a generated file. A file without one is format 1.

curry.backends.cxx.toolchain.gc_flags(gc=None, write_counters=None)

The compiler flags of a collector of the runtime, by default the collector of the installed runtime (config.cxx_gc), with the flag of the write counters when write_counters is true, by default when the installed runtime has them (config.cxx_gc_write_counters). The counters belong to the block heap: ‘mps’ never gets the flag.

curry.backends.cxx.toolchain.generator_digest()

A digest of the files of GENERATOR_SOURCES, cached for the life of the process.

curry.backends.cxx.toolchain.generator_files()

The files of GENERATOR_SOURCES, relative to the package, sorted.

curry.backends.cxx.toolchain.installation_path()

The real path of the installation: the text of the second line of an ABI stamp. The stamp names the installation the object was compiled under, and an object whose stamp names another is stale (Cpp2So.is_stale): a copy of an installation made by hand compiles its objects again, and a package whose manager rewrote the text keeps them (the decision of issue #100). The object itself names no path since format 13 of the generated code: it names the objects of its imports and the runtime library by SONAME (see soname), so the line is the one tie between an object and an installation.

curry.backends.cxx.toolchain.module_of_soname(name)

The full name of the module whose object has the SONAME name (see soname), or None when name is not of that form: the name of a system library, or of the runtime library. The format version is not checked.

curry.backends.cxx.toolchain.object_digest(flavor=None, include_dir=None, gc=None, write_counters=None)

The digest of the ABI stamp of an object compiled now: a digest of the runtime headers the generated code includes (runtime_digest), of the flags of flavor, by default the flavor of the installed runtime (config.cxx_flavor), of the flags of the collector gc and of its write counters, by default the installed ones (config.cxx_gc, config.cxx_gc_write_counters), of the link flags, of the compiler the runtime was built with (config.cxx_compiler: its version and target, as the build recorded them in sysconfig/cxx_compiler) and of the format of the generated code (compiler.FORMAT_VERSION). So a change to a header of the closure, to the flags of a flavor, to the collector, to the write counters, to the compiler of the build or to the format compiles every object again, once; a change to a header outside the closure compiles none. The digest covers what decides whether an object fits a runtime, and nothing of where the runtime is installed: two installations of one runtime give one digest, so the product cache can serve both, and a package relocated by its manager keeps its objects. The installation is the second line of the stamp, as text (see Cpp2So). The compiler enters through the record of the build, not through a probe of the compiler at hand: an installation without a compiler computes the digest of its shipped objects all the same. The flags of the environment (CXXFLAGS) are not part of the digest.

Returns None when the tree holds no header (see runtime_digest).

curry.backends.cxx.toolchain.runtime_digest(include_dir=None)

A digest of the runtime headers the generated code includes (runtime_headers) under include_dir, by default the installed ones: the ABI that generated code is compiled against. The digest covers the names and the contents of the headers of the closure, not their time stamps and not the other headers of the runtime. So a new copy of the same headers, as make stage installs, gives the same digest, a change to any header of the closure gives another, and a change to a header outside it (the equality of graphs, the walks, the bytecode of the interpreter) gives the same. The result is cached for the life of the process.

Returns None when the tree holds no header. Such an installation cannot compile, so its objects are trusted as they are (see Cpp2So.is_stale).

Raises:

• PrerequisiteError – a header cannot be read. make stage links the

• installed headers to the source tree, so this happens when the source –

• tree is gone. The message names the installation. –

curry.backends.cxx.toolchain.runtime_headers(include_dir=None)

The runtime headers the generated code includes, in a fixed order. Every generated file includes cyrt/cyrt.hpp (compiler.RUNTIME_INCLUDES), and the list is the include closure of that header under include_dir, by default the installed include directory: the header, every header it includes with a quoted directive, and so on, a conditional include (#ifdef) counted. A header of the runtime that no include of the closure reaches, one the runtime library alone uses, is not in the list, so an edit to it leaves the digest of the ABI stamp as it is (runtime_digest) and compiles no module again. The list is sorted by the path of each header relative to the include directory, so the digest does not depend on the order of the directives. Empty when the root does not exist (an installation without headers).

Raises:

• PrerequisiteError – a header of the closure exists and cannot be read

• (see runtime_digest). –

curry.backends.cxx.toolchain.soname(fullname, format_version=None)

The SONAME of the object of the module fullname: the full name of the module and the format of the generated code (compiler.FORMAT_VERSION by default), in the form “sprite-<module>.so.<format>”, for instance “sprite-Data.List.so.13”. The compile step gives every object its SONAME, and the linker records the SONAME of each object on the link line as a NEEDED entry of the importer. The dynamic linker satisfies such an entry with an object of that name that the process has mapped already, whatever its path, so the loader imports the modules an object needs before it opens the object (loader.needed_modules). The format version is part of the name, so an object of another format never satisfies the entry; the ABI stamp keeps such an object out of a process in any case. A dot of the module name is a dot of the SONAME; the prefix and the suffix “.so.” bound the name (module_of_soname).

curry.backends.cxx.toolchain.source_is_stale(file_in)

Tells whether a generated .cpp file is unusable: its format stamp is not the emitter’s (compiler.FORMAT_VERSION). It was written for another runtime, and the runtime would read its static data with the wrong layout.