16 Digs🔗ℹ

The digsite metaphor says that one uses a shovel to dig for artifacts. Despite putting the same amount of effort into each scoop, what the shovel encounters may change. This metaphor helps explain non-deterministic attempts to find a resource.

In Racket, a shovel is a shovel/c procedure and a dig is underway when such a procedure has program control.

The digsite metaphor implies some level of repeated effort (digging) towards uncertain results. For example, the built-in shovels for denxi/dig/filesystem resolve symbolic links as a way to “scoop” towards a possibly non-existent file. In a similar scenario, a server hosting data may respond differently to the same request, or issue redirects a shovel may follow.

1. The path of a symbolic link to create with respect to (current-directory).

2. A string S, such that (current-shovel S) returns an artifact.

If hint is already an artifact, then it will be returned as-is. Otherwise, the result depends on the implementation of the shovel.

The default value is broken-shovel.

16.1 Memory Shovels🔗ℹ

(define dig (make-memory-shovel (hash 'a (artifact (text-source "hello")))))

(dig 'a) ; OK

(dig 'b) ; Fails

contents is a nested hash table keyed first by provider names, then package names, then edition names, then both revision names and numbers as shown.

(define providers contents)

(define packages (hash-ref providers "example.com"))

(define editions (hash-ref packages "rpg"))

(define revisions (hash-ref editions "directors-cut"))

(artifact? (hash-ref revisions 0))

(artifact? (hash-ref revisions 28))

(artifact? (hash-ref revisions (hash-ref revisions "re-release")))

As indicated by the last line, each hash table may map keys to other keys in the same table.

e.g.

(hash-ref providers (hash-ref providers "alias.example.com"))

The shovel will recursively resolve keys in this way until it encounters a value of a type not used as a key, in observance of the digsite metaphor. If the search yields cyclic keys, this function will not terminate.

Package queries are autocompleted using defaults, and converted to canonical form with respect to a canon based on the hash table’s shape.

Example:

(define dig

(make-memory-shovel/pkgdef

(hash "default" "jon"

"jon" (hash "calculator"

(hash "scientific"

(hash 0 (artifact #"...")

8 (artifact #"...")

"initial" 0

"beta" 8))))))

(dig ":calculator:scientific:initial:beta")

16.2 Filesystem Shovels🔗ℹ

You can bind shovels to filesystem directories. In observance of the digsite metaphor, the shovels defined herein always follow symbolic links.

Used as a conventional location for unencoded digests for whatever is located at path.

Use as a conventional location for unencoded signatures.

• relative-path complies with (and/c path-string? (not/c complete-path?))

• (build-path directory-path relative-path) must exist as a readable file, or as a link to a readable file.

For example, assume /tmp/example exists as a file in a Unix-like system for a Denxi process to read. From here you can bind a dig procedure to /tmp like so.

(define S (make-filesystem-shovel "/tmp" 'md5 public-key-source))

From here, (S "example") will produce an artifact based on the existing file.

(artifact (file-source (string->path "/tmp/example")) #f #f)

If a file does not exist, then S behaves like broken-shovel. Note that S is non-deterministic, and may begin returning artifacts depending on the state of the directory at the time.

We specified 'md5 as an example CHF. To add integrity information for /tmp/example, create a /tmp/example.md5. It must hold the unencoded bytes of the digest produced from the contents of /tmp/example. If you do this, then (dig "example") will produce this enhanced artifact.

(artifact

(file-source (string->path "/tmp/example"))

(integrity 'md5

(file-source (string->path "/tmp/example.md5")))

#f)

Formally, integrity information appears in the artifact when (build-path directory-path (~a relative-path "." chf)) exists as a readable file.

Singature information may appear in another adjacent file. The complete directory listing should now appear as follows:

/tmp/example.md5.sig must hold the unencoded bytes for a signature produced from the digest’s contents. That is, for /tmp/example.md5.

In this scenario, (dig "example") will produce a complete artifact.

(artifact

(file-source (string->path "/tmp/example"))

(integrity 'md5

(file-source (string->path "/tmp/example.md5")))

(signature public-key-source

(file-source (string->path "/tmp/example.md5.sig"))))

Formally, signature information appears in the artifact when (build-path directory-path (~a relative-path "." chf ".sig")) exists as a readable file.

For a given provider P, package name K, edition E, revision name A, and revision number N, the following invariants must hold for the files in directory-path.

• If (build-path directory-path P "public-key") exists, it must be a readable file containing the public key for P. Corollary: K must not be equal? to "public-key".

• file-name-string? returns #t for P, K, E, A, and N.

• If (build-path directory-path P K E) exists, it is a readable directory suitable for use with make-filesystem-shovel.

• If (build-path directory-path P K E N) exists, it is a readable file containing a package definition.

• If (build-path directory-path P K E A) exists, then it must be a readable link pointing to exactly N. The link’s name must be a revision name from the corresponding package definition file (Digest and signature links are not required).

In terms of the above, here is a valid directory structure for a catalog.

In this example, the providers are Alice, a video game developer, and John, an office application developer.

There is a gap in revision numbers in Alice’s raw-input package. This is fine, because it reflects the possibility of missing information. The catalog will return the latest available match for queries like alice:raw-input:::open-beta.

Finally, each directory may contain a symlink named after a string one could use in a package query. You can leverage this to support getting the latest available version using queries like alice:raw-input.

16.3 HTTP Client Shovels🔗ℹ

The shovel accepts a url-variant? value as an argument. If a different value type is provided, the shovel behaves like broken-shovel. Otherwise, the shovel will create an artifact with http-source components w.r.t. base-url. Path and query string elements from the argument are appended to base-url’s respective fields. This implies that duplicate query string keys are allowed.

Invariant: All http-sources in created artifacts must yield the raw, unencoded bytes of the requested resource. This implies that HTTP response headers must include Content-Type: application/octet-stream.

If optional-chf is #f, then an output artifact will always lack both integrity information and signature information. Otherwise, the artifact will contain integrity information using the given CHF, along with a corresponding (~a "." optional-chf) extension to the last path element of the URL used to fetch the digest.

If optional-pubkey is #f, then an output artifact will always lack signature information. Otherwise, the artifact will contain signature information with optional-pubkey as the public key’s source, along with a corresponding (~a "." optional-chf ".sig") extension to the last path element of the URL used to fetch the signature.

Examples follow. Arguments to http-source are shown as URL strings for brevity. The actual output will include instances of url.

Given a shovel with no additional expectations...

(define dig (make-http-shovel "https://example.com"))

...both (dig "/some-file") and (dig (string->url "/some-file")) will produce

(artifact (http-source "https://example.com/some-file") #f #f)

Given a shovel with all expectations...

(define dig

(make-http-shovel "https://example.com"

'sha3-384

(http-source "https://example.com/public.pem")))

...both (dig "/some-file") and (dig (string->url "/some-file")) will produce

(artifact (http-source "https://example.com/some-file")

(integrity 'sha3-384

(http-source "https://example.com/some-file.sha3-384"))

(signature (http-source "https://example.com/public.pem")

(http-source "https://example.com/some-file.sha3-384.sig")))

16.3.1 Caveats for HTTP Shovels🔗ℹ

Due to the volume of possible network conditions and API definitions for servers, denxi/dig/http provides no mechanisms for checking artifact availability, and no package query canon for package queries. Therefore, make-http-shovel will return artifacts that may produce no content when they are actually used. make-http-shovel only offers reasonable, minimal defaults for downloading compatible artifacts that follow the same naming conventions as make-filesystem-shovel.

If you want a client to check resource availability in advance of artifacts, you will need to implement your own HTTP shovel. If you want an HTTP server to work with make-http-shovel, you can simply serve the files from a directory suitable for make-filesystem-shovel. This will add an invariant where the shovel and the server must agree in advance about what files are available, which you can represent as arguments to make-http-shovel.

Those implementing services will also need to consider the relationship of package queries to artifacts representing package definitions, and implement a package query canon to allow that transition. All of the related machinery for distributing artifacts over the network can be delivered to clients in a launcher, which warrants additional scrutiny since launchers have all privileges of the process’ user.