Hyperschema

Hyperschema builds a registry of declarative compact-encoding schemas and generates the encoders and decoders from it. Every change to a schema bumps its version, and the builder rejects changes that would break data written by an earlier version. That matters in peer-to-peer systems, where peers run different schema versions and old data is never rewritten. HRPC, hyperdb and hyperdispatch all build on it. See RPC and schemas for where it fits. For upstream source and releases, see the Hyperschema repository.

Install

npm i hyperschema

Generated code imports hyperschema/runtime, so keep hyperschema installed wherever the generated code runs, not only where you build it.

Quickstart

Register a type in a namespace and write the schema to disk:

const Hyperschema = require('hyperschema')

const schema = Hyperschema.from('./spec/schema')
const example = schema.namespace('example')

example.register({
  name: 'message',
  fields: [
    { name: 'id', type: 'uint', required: true },
    { name: 'text', type: 'string' }
  ]
})

Hyperschema.toDisk(schema)

toDisk() writes ./spec/schema/schema.json, which records the schema and its version, and ./spec/schema/index.js, which holds the generated encodings. Load them with getStruct():

const c = require('compact-encoding')
const { getStruct } = require('./spec/schema')

const message = getStruct('@example/message')
const encoded = c.encode(message, { id: 1, text: 'hi' })

console.log(c.decode(message, encoded)) // { id: 1, text: 'hi' }

To evolve the type, add a field at the end and run the build again. Commit schema.json, because the next build compares against it:

example.register({
  name: 'message',
  fields: [
    { name: 'id', type: 'uint', required: true },
    { name: 'text', type: 'string' },
    { name: 'author', type: 'string' }
  ]
})

The build bumps the schema from version 1 to 2. Both versions stay readable and writable:

const v1 = getStruct('@example/message', 1)
const v2 = getStruct('@example/message', 2)

const value = { id: 1, text: 'hi', author: 'ana' }

c.decode(v1, c.encode(v1, value)) // { id: 1, text: 'hi', author: null }
c.decode(v2, c.encode(v2, value)) // { id: 1, text: 'hi', author: 'ana' }

API Reference

Building a schema

Hyperschema.from([source][, opts])

src

Create a schema. With a directory it loads schema.json from there when the file exists, so the build continues from the last version. Without one it starts empty.

• Returns: Hyperschema

new Hyperschema(json, opts) takes the same arguments but never reads a directory.

Hyperschema.toDisk(schema[, dir][, opts])

src

Write schema.json and the generated index.js to dir, creating the directory if needed. If the schema changed since it was loaded, the version has already been bumped by then. toDisk(schema, opts) also works and writes to the schema's own dir.

• Returns: void
• Throws: a TypeError if neither dir nor schema.dir is set.

Hyperschema.toDisk(schema, { esm: true })

schema.namespace(name)

src

Create a namespace. Types registered through it get the @name/ prefix, so example.register({ name: 'message' }) defines @example/message. Later types refer to it by that name.

• Returns: Namespace
• Throws: Namespace already exists if the name was already created in this build.

schema.resolve(fqn[, opts])

src

Look up a type by its fully qualified name. Built-in types resolve too.

• Returns: ResolvedType, or undefined if nothing has that name. With aliases: false an unknown name throws a TypeError instead.

schema.register(description)

src

Register a type without going through a namespace. The description takes the same keys as ns.register() plus namespace. Prefer ns.register(): a description without namespace is named @undefined/<name>, and namespace: null registers a bare name with no prefix.

• Returns: ResolvedType

schema.toJSON([opts])

src

The object that toDisk() writes as schema.json.

• Returns: SchemaJSON

schema.toCode([opts])

src

The source of the generated module, as a string.

• Returns: string
• Throws: Must provide filename if using external types when a schema with external types gets no filename.

Hyperschema.esm

src

The default output format. It's false for require('hyperschema') and true for import Hyperschema from 'hyperschema', so the entry point you load decides whether the generated code uses import or require. The esm option of toDisk() and toCode() overrides it. ES module output still goes to index.js, so Node needs "type": "module" in the nearest package.json to load it without a warning.

• Returns: boolean

schema.version / schema.versioned / schema.dir

The current schema version, whether versioning is on, and the output directory. A new schema starts at version 0 and reaches 1 with its first type. After that it goes up by one for each build that changes something.

• Returns: number / boolean / string|null

schema.linkAll() / schema.maybeBumpVersion()

Resolve type references, and bump the version once if this build has changed anything. toJSON(), toCode() and toDisk() call the first, and registering a change calls the second. You don't need to call either.

Namespaces

ns.register(description)

src

Register a type in the namespace. Registering a name that already exists in schema.json updates it, and the builder checks the update against the earlier definition. See type definitions for the keys and schema updates for what is allowed.

• Returns: ResolvedType

ns.require(filename)

src

Set the module that provides this namespace's external types and the map functions of its versioned types. The generated code requires it by a path relative to the generated file, and schema.json records that path.

example.require('./lib/encodings.js')

The generated module

src

toDisk() writes a module that exports the functions below. It works with any registered type, not only structs.

getStruct(name[, version])

An encoding for the type, in the shape compact-encoding uses (preencode, encode and decode), so you can pass it to c.encode() and c.decode() or embed it in another encoding. resolveStruct is the same function under its older name.

• Returns: object
• Throws: Encoder not found <name> if no type has that name.

version is a schema version. It isn't the version of a versioned type.

encode(name, value[, version]) / decode(name, buffer[, version])

Shortcuts for c.encode(getStruct(name, version), value) and c.decode(getStruct(name, version), buffer).

• Returns: Buffer / the decoded value

getEncoding(name)

The raw encoding for the type. It uses whichever schema version was set last, by setVersion() or by an encode(), decode() or getStruct() encoding that ran. Prefer getStruct().

• Returns: object
• Throws: Encoder not found <name>

getEnum(name)

The value map of an enum. A numeric enum maps each key to its number, such as { red: 1, green: 2 }. An enum with strings: true maps each key to itself.

• Returns: object
• Throws: Enum not found <name>

setVersion(version)

Set the schema version that getEncoding() uses.

version

The schema version the module was generated at. In CommonJS output it's that fixed number. In ES module output it's a live binding, so setVersion() and the version argument of the other functions change it.

Entry points

Type definitions

src

register() takes one description object, and its keys decide the kind of type. The builder checks them in this order: alias, enum, array, record, external, versions. With none of them the description is a struct. Every kind needs a name.

Struct

src

Optional fields share a flags bitfield, one bit each, and a field is encoded only when it has a value. A required bool or uint1–uint7 field lives in the flags too, so the builder treats it as optional.

Fields

src

A field can't be a record. Register the record as its own type and use that type.

example.register({
  name: 'profile',
  fields: [
    { name: 'name', type: 'string', required: true },
    { name: 'tags', type: 'string', array: true },
    { name: 'kind', type: 'string', constant: 'profile' }
  ]
})

Alias

src

example.register({ name: 'room-id', alias: 'fixed32' })

Enum

src

A numeric enum takes and returns numbers. Encoding a number past the last value throws Unknown enum. With strings: true the field takes and returns the key string, and decoding an unknown value returns null.

example.register({ name: 'role', enum: ['member', 'admin'] })

Array type

src

A field with array: true is usually enough. Register an array type when you want to name it.

Record type

src

A record is an object. Pass a plain object when you encode, and expect one without a prototype back. A Map encodes as an empty record.

example.register({ name: 'scores', record: true, key: 'string', value: 'uint' })

Versioned type

src

A type that wraps several struct layouts under version numbers you choose. Encoding writes value.version (the latest if it's missing) and then the struct for that version. Decoding reads the version, decodes that struct, sets version on the result and applies the map function if the entry has one. An unknown version throws Unsupported version.

Keep the previous schema.json when you regenerate. Without it every versioned type is framed, which differs from a spec generated with 1.24.0 or earlier.

example.register({
  name: 'item',
  versions: [
    { version: 0, type: '@example/item-v0', map: 'upgradeV0' },
    { version: 1, type: '@example/item-v1' }
  ]
})

External type

src

example.require('./lib/encodings.js')
example.register({ name: 'point', external: 'point' })

Built-in types

src

A field's type is a built-in type or a type you registered. Each built-in type is the compact-encoding encoder of the same name, except uint1–uint7.

Schema updates

src

schema.json is the schema's memory. It records each type, in order, and the schema version at which each field and enum value was added. Hyperschema.from(dir) reads it, and the builder compares every registration against it. Updates are append-only, so data written under any earlier version still decodes.

These mistakes fail at registration or when you build:

Types

Namespace

What schema.namespace() returns.

ResolvedType

What register() and resolve() return.

SchemaJSON

The content of schema.json.

See also

• RPC and schemas—why Pear apps generate their wire format from one schema.
• HRPC—typed RPC stubs generated from a Hyperschema.
• Add typed RPC to a Pear app—the worked how-to, from schema to generated client.
• compact-encoding—the encodings that generated code is built from.
• A production Autobase-backed chat room—a worked example with a generated spec/ directory.
• Upstream Hyperschema repository—source, releases and implementation details.