⇋ backpack

MessagePack implementation with back-referencing extension.

Features

[!WARNING] This is mostly a learning project and—although it works—it is not ready to be used in production.

BackPack implements MessagePack specification and the following extensions:

• [x] Timestamp extension.
• [x] Map and Set extensions.
• [x] Back-referencing extension.
• [x] Does not depend on Node- or browser-specific APIs.

Back-referencing

It's a really simple extension that instructs serializer to replace short repeating strings and property names with numeric ids (references) and keep a map of references to strings in a message header. When deserializing, it does the opposite: reads a message header and then replaces references with associated strings.

Notes

• References implemented as extension formats, specifically fixext1 and fixext2.

• There is a maximum number of references that can be safely stored: $2^{16} = 65536$. There is no point in storing 32 bit long references, since the overhead makes the whole approach non-viable.

• Right now all strings that are 2-16 bytes characters long get turned into references. Turning into references only duplicated strings would make gains even more noticeable, but it's not an easy feat.

Usage

Doesn't get any simpler than this:

import { serialize, deserialize } from '@nrsk/backpack'

const data = {
  /* ... */
}

const se = serialize(data)
const de = deserialize(se)

Benchmarks

Benchmarks for both size and performance can be found here.

Todo

This is mostly about public API:

• [ ] Add options:
  • [ ] Disable built-ins (default: false). If true, disables all built-in extensions;
  • [ ] Use Timestamp extension (default: true);
  • [ ] Use Reference extension (default: true);
  • [ ] Use Map extension (default: true);
  • [ ] Use Set extension (default: true);
  • [ ] Ignore undefined (default: false). If false, it will be encoded as null. Setting to true should theoretically allow one to implement a custom extension to encode undefined as a distinct format/type.
  • [ ] Throw on unknown data type (default: true). Setting to false will just ignore unknown data types and formats.
• [ ] Rewrite extension system.

License

MIT.