Zstandard - Fast real-time compression algorithm🔗ℹ

This collection provides Racket bindings to the reference C implementation of the RFC8878 Zstandard compression format, libzstd.

Binding names are provided by this module without prefixes. To prevent name clashes, it is recommended you require the library using prefix-in as follows:

(require (prefix-in zstd: zstd))

(zstd:compress #"data to compress" 1)

For the rest of this manual, names will be referred to without prefix.

1 About FFI🔗ℹ

Please note that this library is a binding to a native library via FFI. Native calls to FFI functions suspend all Racket threads and cannot be preempted.

You should use caution when using these functions on codepaths which are both latency- sensitive and accept very large inputs. Alternative solutions are to have a dedicated place responsible for running compression and decompression, or using the streaming API’s, which operate in chunks and thus allow for the Racket runtime to run between chunks of work.

2 Simple API🔗ℹ

Compresses src at level compression-level and returns the compressed result in a fresh byte string.

Decompresses src and returns the result in a fresh byte string.

If dest-capacity is not #f, it is used directly as an upper bound on the length of the result. Providing a value that is too small for the decompressed data will raise exn:fail:zstd.

If dest-capacity is #f, the decompressed size is estimated using ZSTD_getFrameContentSize. If the size still cannot be determined, for example if the ZStandard metadata does not include the decompressed size or src is malformed, exn:fail:zstd is raised.

Note that the size estimated by ZSTD_getFrameContentSize can be very large. If your application handles inputs of unknown size, you should provide an explicit dest-capacity limiting the output size, or use the streaming API, to avoid denial-of-service attacks.

3 Explicit Contexts🔗ℹ

An explicit context object allows the library to reuse some state, providing a speed boost. Context objects also hold parameters used to fine-tune the compression and decompression.

Compresses src at level compression-level and returns the compressed result in a fresh byte string.

Decompresses src and returns the result in a fresh byte string.

dest-capacity is handled as in decompress.

4 Advanced API🔗ℹ

The advaned API allows parameters to be set on compression and decompression contexts.

The list of valid parameter names is given by the C enum ZSTD_cParameter.

Only non-experimental parameters are supported, and the names are mapped from the C names as follows:

• Lower-case the name

• Replace all underscores with dashes

• Separate words with dashes

• Convert to symbol

So, for example, the C enum constant ZSTD_c_ldmHashRateLog is expressed as 'zstd-c-ldm-hash-rate-log.

The list of valid strategies is given by the C enum ZSTD_strategy.

The C names are mapped to Racket names in the same manner as valid-compression-parameter?.

Additionally, 'zstd-default is not defined in the C enum but is supported to mean "the default strategy".

Returns the minimum and maximum values for the given compression parameter param.

Set the parameter param to the value value. Values must be an exact integer, unless if param is 'zstd-c-strategy, in which case it must be one of the symbols accepted by valid-compression-strategy?.

Resets cctx to default values according to directive.

Compresses src according to the parameters set in cctx and returns the result in a fresh byte string.

The list of valid parameter names is given by the C enum ZSTD_dParameter.

Only non-experimental parameters are supported.

The C names are mapped to Racket names in the same manner as valid-compression-parameter?.

Returns the minimum and maximum values for the given decompression parameter param.

Set the parameter param to the value value.

Resets dctx to default values according to directive.

5 Streaming API🔗ℹ

Prepares cctx for streaming compression at compression-level.

Reads chunks of bytes from in and compresses them, calling f with each chunk of compressed data as a fresh mutable byte string. Stops when eof is first returned by in.

The return value of f is ignored.

Prepares dctx for streaming decompression.

Reads chunks of bytes from in and decompresses them, calling f with each chunk of decompressed data as a fresh mutable byte string. Stops when eof is first returned by in.

The return value of f is ignored.

6 Simple Dictionary API🔗ℹ

Compresses src using dict at compression-level, and returns the compressed result in a fresh byte string.

Decompresses src and returns the result in a fresh byte string.

dest-capacity is handled as in decompress.

7 Bulk Dictionary API🔗ℹ

Loads the dictionary represented by dict for compression at compression-level and returns it. Underlying native resources are automatically released when the returned value is garbage-collected.

Loads the dictionary represented by dict for decompression and returns it. Underlying native resources are automatically released when the returned value is garbage-collected.

Compresses src using cdict and returns the compressed result in a fresh byte string.

Decompresses src and returns the result in a fresh byte string.

dest-capacity is handled as in decompress.

8 Dictionary Helpers🔗ℹ

Returns the dictionary ID of the dictionary represented by dict, or 0 if it does not have a valid ZStandard dictionary header.

Returns the dictionary ID of cdict, or 0 if it was loaded from a buffer that did not have a valid ZStandard dictionary header.

Returns the dictionary ID of ddict, or 0 if it was loaded from a buffer that did not have a valid ZStandard dictionary header.

Returns the dictionary ID needed to decompress the ZStandard frame in src, or 0 in the following cases:

• The frame does not require a dictionary to be decoded (most common case).

• The frame was built with dictID intentionally removed. Whatever dictionary is necessary is hidden information.

• src is too small, and as a result, the frame header could not be decoded.

• src is not a Zstandard frame.

9 Advanced Dictionary API🔗ℹ

The ref* family of functions are not exposed, because they hold onto user-provided objects in native code, which is dangerous in the face of garbage collection.

Loads the dictionary dict into cctx.

Loads the dictionary dict into dctx.

10 Dictionary Training🔗ℹ

Trains a dictionary on the samples, returning a byte string of length at most dict-size. This dictionary can be saved and later reloaded for use in the various dictionary compression and decompression API’s.

11 Extra Helpers🔗ℹ

This section documents items not directly corresponding to an underlying libzstd API, but are provided for convenience to Racket programmers.

Streams bytes from in, compresses them at the default compression level, and writes them to out. Stops when eof is first returned by in.

Streams bytes from in, decompresses them, and writes them to out. Stops when eof is first returned by in.

12 License and Contributing🔗ℹ

This binding is licensed under the 3-Clause BSD license, same as libzstd.

Please see this page for information about contributing to my libraries.