4 Tokens of Syntax Object Tree Content🔗ℹ

Traditionally, a bracket appears as text in a text stream, and this token can be pulled out of the text stream and treated as a string. In Punctaffy, when we recognize taffy-notation? hyperbracket notations, we need to pull a token out of a program represented by a Racket syntax object. This results in a list of adjacent syntax objects, where each syntax object can have holes in it. Punctaffy supplies a data structure to represent this kind of data, which we call a token of syntax object tree content, or a token of syntax for short.

The holes in a token of syntax are given mutually unique labels, and the labels are part of the token’s representation. We call these labels the token’s free variables. They can be arbitrary values identified by equal?, but usually, they’re interned symbols.

In our representation, a token of syntax can also have nodes in it that represent assertions that they have only one subform once the holes are filled in. At the moment, these assertions are checked only when the token is converted into a list of syntax objects.

(In Punctaffy’s hypersnippet terminology, a token of syntax is a degree-2 hypersnippet of the textual code, as opposed to a string token, which is a degree-1 hypersnippet. We could represent tokens of syntax in a way that’s generalizable to arbitrarily high degrees by using hypernests, but instead we dedicate some unique attention to this data structure so we can manipulate it efficiently.)

Note that if the token has a single hole at its root, that hole could be filled with more than one (or fewer than one) syntax object. What singular-token-of-syntax? checks for has to do with representation details of the token of syntax itself, not what its shape is when it’s embedded in a syntax object tree.

The element list is a list of singular-token-of-syntax? values. This just prevents token-of-syntax-beginning-with-splice? values from being nested, which would otherwise be an unnecessary source of variation between token values.

This node type makes up for the fact that all compositions of tokens of syntax are splicing compositions in the sense of unquote-splicing or unsyntax-splicing. Using this, it’s possible to assert that a certain use site actually only admits a single element, as would be the case with a non-splicing unquote or unsyntax.

The hole is labeled with an arbitrary object to use as a free variable name, which will be identified using equal?.

When this token is converted to a list of syntax objects, the caller performing that conversion will specify some list of syntax objects to substitute for the variable, and that list will be the result. In the name of the node type, we specifically refer to this a "splicing" free variable occurrence, since the way it deals with a list of trees makes it more analogous to unquote-splicing or unsyntax-splicing than to unquote or unsyntax.

When this token is converted to a list of trees, it asserts that the given e token results in a single value, and then it invokes datum->syntax to wrap that value as a syntax object. The the lexical information, source location, and syntax properties of the wrapper are copied from the given stx-example.

When this token is converted to a list of trees, it asserts that the given element token results in a single value, and then it wraps that value in an immutable box.

When this token is converted to a list of trees, it takes the list of trees that result from the given elements token and wraps them as the elements of an immutable vector.

When this token is converted to a list of trees, it takes the list of trees that result from the given elements token and wraps them as the fields of an immutable prefab struct with the same prefab-struct-key as prefab-struct-example. If the prefab key isn’t consistent with the computed number of fields, this raises an error. For now, this error is reported in terms of an internal call to make-prefab-struct.

When this token is converted to a list of trees, it asserts that the given tail token results in a single value, and then it invokes list* to wrap that as the tail of a possibly improper list beginning with the elements obtained from the result of elements.

If tail results in a single tree that’s a proper list, the result will also be a single tree that’s a proper list.

If elements results in a list of zero trees, the result will just be the result of tail, even if that’s not a pair?.

Unlike with token-of-syntax-beginning-with-splice? nodes, we make no attempt to enforce that two token-of-syntax-beginning-with-list*? nodes aren’t nested in a way that could be combined into a single node. (TODO: Should we?)

When this token is converted to a list of trees, it results in just one tree, namely the given value.

The value must not be a syntax object, an immutable box, an immutable vector, an immutable prefab struct, or a pair. This just prevents token-of-syntax-beginning-with-other-value? values from being represented in alternative ways using the other token of syntax constructors, which would be an unnecessary source of variation between token values.

(TODO: What if Racket’s syntax supports more values in the future? In taffy-let and our other hyperbracketed operations, we defensively disallow certain values that people might want hypersnippets to reach into someday, like hash? and regexp? values. Maybe we should disallow those here too. Alternatively, maybe we should just allow every type of value here, a policy which might be good for performance so that we don’t traverse into certain data instances further than necessary.)

This operation may fail if assertions in the token are unmet, such as the assertions of single-element intermediate results made by nodes like token-of-syntax-beginning-with-assert-singular? and token-of-syntax-beginning-with-box?.

This is like token-of-syntax-beginning-with-other-value, but it also traverses into syntax objects, immutable boxes, immutable vectors, immutable prefab structs, and pairs to construct the appropriate token nodes.

We’ll call the resulting token quotation-token.

When token-of-syntax->syntax-list converts quotation-token to a list of trees, the result is a list containing some number of Racket expressions which, when run, result in lists of trees. When these lists of trees are appended, they form the result of a simulated invocation of token-of-syntax->syntax-list on token.

The substitutions this simulated call supplies for token’s free variables are obtained from the substitutions of quotation-token’s corresponding free variables. In particular, the substitution of each of quotation-token’s free variables should be a list of Racket expressions, each of which should return a list of trees when it’s run. The concatenation of these lists of trees is the substitution for token’s corresponding free variable.

The quote-expr argument should be a function that takes a Racket syntax object and returns another Racket syntax object that represents an expression which quotes it. This is applied to token-of-syntax-beginning-with-other-value? nodes of the token, and it effectively determines what kind of quoting is performed. For instance, quote-expr could be (lambda (expr) #`'#,expr) to do the usual kind of quoting of values, which preserves many values but converts several types of mutable data structures into immutable ones. (TODO: Test with multiple choices of quote-expr to be sure it works, and see if we can explain better.)

The datum->result--id argument should be an identifier bound to a syntax that can be used like (datum->result stx-example datum-expr) to transfer a syntax wrapper. For instance, the following syntax would create syntax wrappers that carry over the lexical information of the original syntax wrappers, but not their source locations or syntax properties:

(define-syntax-parse-rule (datum->result stx-example datum)

(syntax->datum (quote-syntax stx-example #:local) datum))

(TODO: Test with multiple choices of datum->result--id to be sure it works, and see if we can explain better.)