5 Defining Hyperbracket Notations for Racket🔗ℹ

In order to use higher-dimensional hypernests for syntax in Racket, we supply some new notations that augment Racket’s low-dimensional lexical structure. We do this not by replacing Racket’s existing reader and macro system, but by letting each Racket macro that cares about higher-dimensional structure do its own parsing of that structure when it needs to. Before this parsing occurs, we represent the structure in Racket syntax in the form of what we call hyperbrackets.

Punctaffy hasn’t been put to the test yet, and there may still be some uncertainty about what the best notation for hypernest-structured programs will be. We may augment Punctaffy with new ideas in the future, and not all possible paths will necessarily lead to the same feature set.

With this in mind, we define our most basic hyperbracket notation keywords in such a way that they have no innate functionality but can instead be assigned meanings (hopefully consistent enough ones) by multiple external DSL implementations.

This is a pretty typical scenario in Racket. Racket has identifiers like _, ..., else, and unquote that are only assigned meaning by some external DSL.

Typically these DSLs recognize these special identifiers using free-identifier=?. This approach allows new keywords to be added over time by exporting them from new (module path, symbol) pairs.

We take a different approach: We treat the identity as part of the functionality of the macro itself, not of the identifier. For each new piece of hyperbracket notation that should be identifiable, we provide some innate way to identify it, essentially treating it as a small macro DSL of its very own. As various experimental DSL implementations based on Punctaffy’s hyperbracket notation mature and stabilize their macro calling conventions, we expect hyperbrackets to be a patchwork of multiple DSLs anyway, so we embrace that from the start.

Note that the term "hyperbracket" could be generalized beyond our Racket-specific use here. Punctaffy’s own hypernest-bracket? values are an abstract syntax representation of 0-dimensional hyperbrackets with labels in certain places. The ones we’re talking about here are instead a concrete (indeed, unparsed) representation of 1-dimensional hyperbrackets with no labels. We’re using the generic term "hyperbracket" for the latter because it’s the one that’s most likely to be relevant most often in the context of Punctaffy’s use as a Racket library, since Racket already has 1-dimensional hypernest bumps (namely, paren matchings) to attach hyperbrackets to.

A structure type that implements this property should generally also implement prop:procedure to display an informative error message when the notation is used in a position where a Racket expression is expected.

A structure type that implements this property should generally also implement prop:taffy-notation to signify the fact that it’s a hyperbracket notation.

A structure type that implements this property should generally also implement prop:procedure to display an informative error message when the notation is used in a position where a Racket expression is expected.

The result is an equal?-based hash that represents several components parsed from the term:

'lexical-context

An identifier that may refer to certain information Punctaffy needs to keep track of about the hyperbracket-nesting depth that this hyperbracket interacts with. Currently, this identifier is not actually used (TODO), but we intend to use this identifier’s lexical information to prevent a macro call’s macro-introduced uses of hyperstacks from interacting unintentionally with the macro caller’s uses of hyperbrackets. A hyperbracket-nesting depth is more than a number and will likely be represented by a hyperstack, possibly together with information about variables that are in scope at the various depths.

Hyperbracket notations implementing this method may differ about how they determine the 'lexical-context identifier, but when the user doesn’t specify it explicitly in the hyperbracket’s syntax, the usual default will be an identifier whose name is the symbol '#%lexical-context and whose lexical information matches that of stx.

'direction

A representation of whether this call site of this notation represents an opening hyperbracket or a closing hyperbracket. If it’s an opening hyperbracket like ^<d or ^<, this is the symbol '<. Otherwise, it’s a closing hyperbracket like ^>d or ^>, and this is the symbol '>.

'degree

A syntax object containing a natural? number, representing the degree of the hyperbracket. This corresponds to the degree the user would specify explicitly when using ^<d or ^>d.

'contents

A list of syntax objects. This hyperbracket usage site only takes up some outermost part of stx, and some unconsumed region of syntax remains beyond that, which is what this represents.

If this use site is an opening hyperbracket, this remaining syntax begins inside it, but it may contain its own closing hyperbrackets that continue into syntax that’s outside again. If this use site is a closing bracket, this remaining syntax begins outside it, but it may contain closing hyperbrackets that close this one and continue into syntax that’s inside again. Calling it the "contents" is somewhat of a misnomer. (TODO: Rename 'contents to use a term like "tail," "rest," "beyond," or "remainder" instead, ideally referring to its role as a piece of the given stx rather than a piece of the hyperbracket we parsed out.)

'token-of-syntax

A token of syntax value representing the concrete syntax of the hyperbracket usage site itself, possibly separated from the specific values of 'lexical-context, 'degree, and 'contents so that these can be replaced with new values.

The token may have free variables named 'lexical-context, 'degree, and 'contents. If the token is converted to a list of trees using token-of-syntax->syntax-list, it should result in a single tree that closely resembles stx, particularly if the free variables are substituted with the corresponding information that this method call parsed out.

Specifically, since the substitution takes lists of syntax as input, the 'lexical-context and 'degree parts of the parsed hash need to be wrapped in singleton lists to perform the substitution that most faithfully reproduces stx. The 'contents part of the parsed hash is already a list and can be substituted in without wrapping it.

The reason this 'token-of-syntax entry exists in the parse result is to support usage scenarios that involve parsing a program, transforming parts of it, and reconstructing it with the new parts. For instance, taffy-quote uses this so that it can process interpolated expressions without disrupting the syntax of hyperbrackets that appear in the quoted region of code.