3 Zuo as a make Replacement🔗ℹ

The zuo/build library is modeled on make and Shake for tracking dependencies and build steps. The library has two layers:

• The core target datatype and build engine, as reflected by functions like target and build.

• A makefile-like, declarative form for dependencies as implemented by the make-targets function.

A target represents either an input to a build (such as a source file) or a generated output, and a target can depend on any number of other targets. A target’s output is represented by a string that is normally an SHA-256 hash: more precisely, it is represented by a value satisfying the predicate sha256?. The build procedure records hashes and dependencies in a database located alongside non-input targets, so it can avoid rebuilding targets when nothing has changed since the last build. Unlike make, timestamps are used only as a shortcut to avoiding computing the SHA-256 of a file (i.e., if the timestamp has not changed, the SHA-256 result is assumed to be unchanged).

“Recursive make” is encouraged in the sense that a target’s build rule can call build to start a nested build, or it can call build/dep to build or register a dependency that is discovered in the process of building.

Here’s an example of a Zuo script to build "demo" by compiling and linking "main.c" and "helper.c":

#lang zuo

(provide-targets targets-at)

(define (targets-at at-dir vars)

(define demo (at-dir (.exe "demo")))

(define main.c (at-source "main.c"))

(define main.o (at-dir (.c->.o "main.c")))

(define helper.c (at-source "helper.c"))

(define helper.o (at-dir (.c->.o "helper.c")))

(make-targets

`([:target ,demo (,main.o ,helper.o)

,(lambda (dest token)

(c-link dest (list main.o helper.o) vars))]

[:target ,main.o (,main.c)

,(lambda (dest token)

(c-compile dest main.c vars))]

[:target ,helper.o (,helper.c)

,(lambda (dest token)

(c-compile dest helper.c vars))]

[:target clean ()

,(lambda (token)

(for-each rm* (list main.o helper.o demo)))])))

Although the make-targets function takes a makefile-like description of targets and dependencies, this script is still much more verbose than a Unix-specific makefile that performs the same task. Zuo is designed to support the kind of syntactic abstraction that could make this script compact, but the current implementation is aimed at build tasks that are larger and more complex. In those cases, it’s not just a matter of dispatching to external tools like a C compiler, and most Zuo code ends up in helper functions and libraries outside the make-targets form.

3.1 Creating Targets🔗ℹ

Construct a target with either input-file-target (given a filename), input-data-target (given a value whose ~s form is hashed), or target (given a filename for a real target or a symbol for a phony target).

Only a target created with target can have dependencies, but they are not specified when target is called, because computing dependencies for a target may involve work that can be skipped if the target isn’t needed. Instead, target takes a get-rule procedure that will be called if the dependencies are needed. The get-rule procedure returns up to three results in a rule record: a list of dependencies; the hash of an already-built version of the target, if one exists, where file-sha256 is used by default; and a rebuild procedure that is called if the returned hash, the hash of dependencies (rebuilt if needed), and recorded results from a previous build together determine that a rebuild is needed.

When a target’s rebuild function is called, it optionally returns a hash for the result of the build if the target’s rule has one, otherwise file-sha256 is used to get a result hash. Either way, it’s possible that the result hash is the same as the one returned by get-rule; that is, maybe a dependency of the target changed, but the change turned out not to affect the built result. In that case, rebuilding for other targets that depend on this one can be short-circuited.

Finally, in the process of building a target, a rebuild procedure may discover additional dependencies. A discovered dependency sent to build/dep is recorded as a dependency of the target in addition to the ones that were reported by get-deps. Any changes in these additional targets trigger a rebuild of the target in the future. Meanwhile, the build system assumes that if none of the dependencies change, then the set of additional dependencies discovered by rebuild would be the same; that assumption allows the build system to skip rebuild and its discoveries if none of the dependencies have changed.

A phony target is like a regular target, but one that always needs to be rebuilt. A typical use of a phony target is to give a name to a set of “top-level” targets or to implement an action along the lines of make install. Create a phony target with target and a symbol name.

A target can declare multiple outputs by specifying additional outputs in a 'co-outputs option. The target’s rebuild procedure will be called if any of the additional outputs are missing or not consistent with the result of an earlier build.

In many cases, a plain path string can be used as a target as a shorthand for applying input-file-target to the path string.

3.2 Building Targets🔗ℹ

There is no global list of targets that build draws from. Instead, build starts with a given target, and it learns about other targets as get-dep procedures return them and as rebuild procedures expose them via build/dep. If build discovers multiple non-input targets with the same filename, then it reports an error.

The build/command-line function is a convenience to implement make-like command-line handling for building targets. The build/command-line procedure takes a list of targets, and it calls build on one or more of them based on command-line arguments (with help from find-target).

All relative paths are considered relative to the start-time current directory. This convention works well for running a Zuo script that’s in a source directory while the current directory is the build directory, as long as the script references source files with at-source to make them relative to the script. For multi-directory builds, a good convention is for each directory to have a script that exports a targets-at procedure, where targets-at takes an at-dir procedure (supplied as just build-path by default) to apply to each target path when building a list of targets, and a hash table of variables (analogous to variables that a makefile might provide to another makefile via make arguments).

As a further convenience following the targets-at model, the provide-targets form takes an identifier for such a targets-at procedure, and it both exports targets-at and creates a main submodule that calls build/command-line* on with the targets-at procedure.

As a naming convention, consider using "main.zuo" in a directory where build results are intended to be written, but use "build.zuo" in a source directory that is intended to be (potentially) separate from the build directory. In other words, use "main.zuo" as a replacement for "Makefile" and "build.zuo" as a replacement for "Makefile.in" in a configure-style build. You may even have a configure script that generates a "main.zuo" script in a build directory so that zuo . is a replacement for make. The generated "main.zuo" could import the source directory’s "build.zuo" and calls build/command-line* on with the imported targets-at procedure plus at-source:

#lang zuo

(require "‹srcdir›/build.zuo")

(build/command-line* targets-at at-source)

However, correctly encoding ‹srcdir› can be tricky when working from something like a shell configure script or batch file to generate "main.zuo". You may find it easier to write the path to a separate file using a shell-variable assignment syntax, and then have the generated "main.zuo" read from that file. The bounce-to-targets form implements that pattern. For example, if "Mf-config" is written in the same directory with a srcdir= line to specify the source directory (where no escapes are needed for the path after =), then a "main.zuo" of the form

#lang zuo

(bounce-to-targets "Mf-config" 'srcdir "build.zuo")

reads "Mf-config" to find and dispatch to "build.zuo" in the same way as the earlier example module.

3.3 Recording Results🔗ℹ

Build results are stored in a "_zuo.db" file in the same directory as a target (by default). Cached SHA-256 results with associated file timestamps are stored in a "_zuo_tc.db" in the same directory (i.e., the cached value for dependency is kept with the target, which is in a writable build space, while an input-file target might be in a read-only source space). A target’s options can specify an alternative directory to use for "_zuo.db" and "_zuo_tc.db". Timestamp recording in "_zuo_tc.db" is disabled if the SOURCE_DATE_EPOCH environment variable is set.

In the unfortunate case that a "_zuo.db" or "_zuo_tc.db" file gets mangled, then it may trigger an error that halts the build system, but the "_zuo.db" or "_zuo_tc.db" file will be deleted in reaction to the error. Another attempt at the build should recover, while perhaps rebuilding more than it would have otherwise, since the result of previous builds might have been lost.

Specify a location for the "_zuo.db" and "_zuo_tc.db" files associated with a target via the 'db-dir target option. The make-targets function recognizes as :db-dir clause to set the option for all of the targets that it creates.

3.4 Parallelism🔗ℹ

A build runs in a threading context, so a target’s get-deps or rebuild procedure can use thread-process-wait to wait on a process. Doing so can enable parallelism among targets, depending on the 'jobs option provided to build or build/command-line, a --jobs command-line argument parsed by build/command-line, a jobserver configuration as provided by GNU make and communicated through the MAKEFLAGS environment variable, or the ZUO_JOBS environment variable.

When calling build for a nested build from a target’s get-deps or rebuild procedures, supply the build token that is passed to get-deps to the build call. That way, parallelism configured for the enclosing build will be extended to the nested build.

3.5 Build API🔗ℹ

The result of (symbol->string name) must be distinct among all the input-data dependencies of a particular target, but it does not need to be globally unique.

In the case of a file target, get-deps receives name back, because that’s often more convenient for constructing a target when applying an at-dir function to create name.

The build token argument to get-deps represents the target build in progress. It’s useful with file-sha256 to take advantage of caching, with build/dep to report discovered targets, and with build/no-dep or build.

The following keys are recognized in options:

• 'co-outputs mapped to a list of path strings: paths that are also generated by the target in addition to name when name is a path string; the target’s build function will be called if the combination of name and these files is out-of-date.

• 'precious? mapped to any value: if non-#f for a non-phony target, name is not deleted if the get-deps function or its result’s rebuild function fails.

• 'command? mapped to any value: if non-#f, when build/command-line runs the target as the first one named on the command line, all arguments from the command line after the target name are provided get-deps as additional arguments. When building a target directly instead of through build/command-line, use command-target->target to supply arguments.

• 'noisy? mapped to any value: if non-#f, then a message prints via alert whenever when the target is found to be already up to date.

• 'quiet? mapped to any value: if non-#f, then even when build runs the target directly or as the dependency of a phony target, it does not print a message via alert when the target is up to date, unless the target is also noisy. When a phony target is quiet, it builds its dependencies as quiet.

• 'eager? mapped to any value: if non-#f, then the target’s rule is not run in a separate thread, which has the effect of ordering the rule before others that do run in a separate thread.

• 'recur? mapped to any value: if non-#f, then the target’s rule is run in dry-run modes of build the same as non-dry-run modes. This option is analogous to prefixing a command with + in a makefile.

• 'db-dir mapped to a path or #f: if non-#f, build information for the target is stored in "_zuo.db" and "_zuo_tc.db" files in the specified directory, instead of the directory of name.

Changed in version 1.8: Added 'recur? for options.

A path string can be reported as a dependency in dependencies, in which case it is coerced to a target using input-file-target. If sha256 is #f, file-sha256 is used to compute the target’s current hash, and rebuild is not expected to return a hash. If sha256 is not #f, then if rebuild is called, it must return a new hash.

If target is a path, then it is coerced to target via input-file-target, but the only effect will be to compute the file’s SHA-256 or error if the file does not exist.

The options argument supplies build options, and the following keys are recognized:

• 'jobs mapped to a positive integer: controls the maximum build steps that are allowed to proceed concurrently, and this concurrency turns into parallelism when a task uses a process and thread-process-wait; if 'jobs is not mapped, a jobserver is used if found via maybe-jobserver-client; otherwise, the default is the value of the ZUO_JOBS environment variable if it is set, 1 if not

• 'log? mapped to any value: enables logging of rebuild reasons via alert when the value is not #f; logging also can be enabled by setting the ZUO_BUILD_LOG environment variable

• 'dry-run-mode mapped to #f, 'question, or 'dry-run: enables “dry run” mode when non-#f; when the value is 'dry-run, build prints targets whose rules would be run (without running them); when the value is 'question, build does not rules, but exits with 1 when some target’s rule would be run; a target can be made immune to dry-run mode through a 'recur? option; when 'dry-run is not set in options, the mode is determined by calling maybe-dry-run-mode

If token is not #f, it must be a build token that was passed to a target’s get-deps to represent a build in progress (but paused to run this one). The new build process uses parallelism available within the in-progress build for the new build process.

Whether or not token is #f, the new build is independent of other builds in the sense that target results for others build are not reused for this one. That is, other builds and this one might check the states of some of the same files, but any triggered actions are separate, and phony targets are similarly triggered independently. Use build/dep or build/no-dep, instead, to recursively trigger targets within the same build.

Changed in version 1.1: Use maybe-jobserver-client if 'jobs is not set in options.
Changed in version 1.8: Added support for 'dry-run-mode in options.

If options has a mapping for 'args, the value is used as the command-line arguments to parse instead of (hash-ref (runtime-env) 'args). If options has a mapping for 'usage, the value is used as the usage options string.

The targets-at procedure is applied to at-dir and a hash table of variables, where each variable name is converted to a symbol and the value is left exactly as after =.

The sha256? predicate recognizes no-sha256 and strings for which string-length returns either sha256-length or a multiple of sha256-length. The later case is used for multi-file targets, which concatenate the constituent SHA-256 strings.

See also string-sha256.

Changed in version 1.7: Removed build-path as a second argument to build/command-line* so that the default (make-at-dir ".") is used, instead.

The path produced by config-file-expr is interpreted relative to the enclosing module. If the path in that file for key-symbol-expr is relative, it is treated relative to the config-file-expr path.

See Building Targets for an explanation of how bounce-to-targets is useful. The expansion of bounce-to-targets is roughly as follows:

(define config (config-file->hash (at-source config-file-expr)))

(define at-config-dir (make-at-dir (or (car (split-path config-file)) ".")))

(define script-file (at-config-dir (hash-ref config key-symbol-expr)

script-file-expr))

(build/command-line* (dynamic-require script-file 'targets-at)

at-source)

Although it might seem natural for this make-like specification to be provided as a syntactic form, typical makefiles use patterns and variables to generate sets of rules. In Zuo, map and similar are available for generating sets of rules. So, make-targets takes an S-expression representation of the declaration as specs, and plain old quasiquote and splicing can be used to construct specs.

The specs argument is a list of lines, where each line has one of the following shapes:

`[:target ,path (,dep-path-or-target ...) ,build-proc ,option ...]

`[:depend ,path (,dep-path-or-target ...)]

`[:target (,path ...) (,dep-path-or-target ...) ,build-proc ,option ...]

`[:depend (,path ...) (,dep-path-or-target ...)]

`[:db-dir ,path]

A ':target line defines a build rule that is implemented by build-proc, while a ':depend line adds extra dependencies for a path that also has a ':target line. A ':depend line with multiple paths is the same as a sequence of ':depend lines with the same dep-path-or-target list, but a ':target line with multiple paths creates a single target that builds all of the paths.

In ':target and ':depend lines, a path is normally a path string, but it can be a symbol for a phony target. When a ':target has multiple paths, they must all be path strings.

A build-proc accepts a path (if not phony) and a build token, just like a get-deps procedure for target, but build-proc should build the target like the rebuild procedure for rule (or phony-rule). When a ':target line has multiple paths, only the first one is passed to the build-proc.

A dep-path-or-target is normally a path string. If it is the same path as the path of a ':target line, then a dependency is established on that target. If dep-path-or-target is any other path string, it is coerced to an input-file target. A dep-path-or-target can also be a target that is created outside the make-targets call.

An option can be ':precious, ':command, ':noisy, ':quiet, ':eager, or ':recur to set the corresponding option (see target) in a target.

A ':db-dir line (appearing at most once) specifies where build information should be recorded for all targets. Otherwise, the build result for each target is stored in the target’s directory.

Changed in version 1.8: Added ':recur for option.