1. Quick Start
2. Introduction
3. Why
4. Goals
5. Data Format
6. Value Types
   1. JSON
   2. Universal Binary JSON
7. Container Types
   1. JSON
   2. Universal Binary JSON
8. Streaming Types
   1. No-Op Type
   2. Unknown-Length Containers
9. Size Requirements
10. Endianness
11. MIME Type
12. File Extension
13. History
14. Changelog
15. Requests for Enhancement (RFE)

Quick Start

You know what JSON is, you know what UBJSON is and you just want to dig in on your project with it, here are the quick bits:

1. Open up the Type Reference in a tab to keep handy (don’t forget the sub-sections with details on each type if you need them)
2. Go download the UBJSON library for your language of choice, and have fun!
3. Discuss questions about the Spec or Libraries in the Google Group. File bugs or issues in GitHub!

Introduction

JSON has become a ubiquitous text-based file format for data interchange. Its simplicity, ease of processing and (relatively) rich data typing made it a natural choice for many developers needing to store or shuffle data between systems quickly and easy.

Unfortunately, marshaling native programming language constructs in and out of a text-based representations does have a measurable processing cost associated with it.

In high-performance applications, avoiding the text-processing step of JSON can net big wins in both processing time and size reduction of stored information, which is where a binary JSON format becomes helpful.

Why

Attempts to make using JSON faster through binary specifications like BSON, BJSON or Smile exist, but have been rejected from mass-adoption for two reasons:

1. Custom (Binary-Only) Data Types: Inclusion of custom data types that have no ancillary in the original JSON spec, leaving room for incompatibilities to exist as different implementations of the spec handle the binary-only data types differently.
2. Complexity: Some specifications provide higher performance or smaller representations at the cost of a much more complex specification, making implementations more difficult which can slow or block adoption. One of the key reasons JSON became as popular as it did was because of its ease of use.

BSON, for example, defines types for binary data, regular expressions, JavaScript code blocks and other constructs that have no equivalent data type in JSON. BJSON defines a binary data type as well, again leaving the door wide open to interpretation that can potentially lead to incompatibilities between two implementations of the spec and Smile, while the closest, defines more complex data constructs and generation/parsing rules in the name of absolute space efficiency.

The existing binary JSON specifications all define incompatibilities or complexities that undo the singular tenant that made JSON so successful: simplicity.

JSON’s simplicity made it accessible to anyone, made implementations in every language available and made explaining it to anyone consuming your data immediate.

Any successful binary JSON specification must carry these properties forward for it to be genuinely helpful to the community at large.

This specification is defined around a singular construct used to build up and represent JSON values and objects. Reading and writing the format is trivial, designed with the goal of being understood in under 10 minutes (likely less if you are very comfortable with JSON already).

Fortunately, while the Universal Binary JSON specification carriers these tenants of simplicity forward, it is also able to take advantage of optimized binary data structures that are (on average) 30% smaller than compacted JSON and specified for ultimate read performance; bringing simplicity, size and performance all together into a single specification that is 100% compatible with JSON.

Why not JSON+gzip?

On the surface simply gzipping your compacted JSON may seem like a valid (and smaller) alternative to using the Universal Binary JSON specification, but there are two significant costs associated with this approach that you should be aware of:

1. At least a 50% performance overhead for processing the data.
2. Lack of data clarity and inability to inspect it directly.

While gzipping your JSON will give you great compression, about 75% on average, the overhead required to read/write the data becomes significantly higher. Additionally, because the binary data is now in a compressed format you can no longer open it directly in an editor and scan the human-readable portions of it easily; which can be important during debugging, testing or data verification and recovery.

Utilizing the Universal Binary JSON format will typically provide a 30% reduction in size and store your data in a read-optimized format offering you much higher performance than even compacted JSON.

If you had a usage scenario where your data is put into long-term cold storage and pulled out in large chunks for processing, you might even consider gzipping your Universal Binary JSON files, storing those, and when they are pulled out and unzipped, you can then process them with all the speed advantages of UBJ.

As always, deciding which approach is right for your project depends heavily on what you need.

Goals

The Universal Binary JSON specification has 3 goals:

1. Universal Compatibility

Meaning absolute compatibility with the JSON spec itself as well as only utilizing data types that are natively supported in all popular programming languages.

This allows 1:1 transforms between standard JSON and Universal Binary JSON as well as efficient representation in all popular programming languages without requiring parser developers to account for strange data types that their language may not support.

2. Ease of Use

The Universal Binary JSON specification is intentionally defined using a single core data structure to build up the entire specification.

This accomplishes two things: it allows the spec to be understood quickly and allows developers to write trivially simple code to take advantage of it or interchange data with another system utilizing it.

3. Speed / Efficiency

Typically the motivation for using a binary specification over a text-based one is speed and/or efficiency, so strict attention was paid to selecting data constructs and representations that are (roughly) 30% smaller than their compacted JSON counterparts and optimized for fast parsing.

Data Format

The Universal Binary JSON specification utilizes a single binary tuple to represent all JSON data types (both value and container types):

[type, 1-byte char]([length, 1 or 4-byte integer])([data])

Each element in the tuple is defined as:

• type
  • A 1-byte ASCII char used to indicate the type of the data following it.
  • A single ASCII char was chosen to make manually walking and debugging data stored in the Universal Binary JSON format as easy as possible (e.g. making the data relatively readable in a hex editor).
• length (OPTIONAL)
  • 1-byte or 4-byte length value based on the type specified. This allows for more aggressive compression and space-optimization when dealing with a lot of small values.
    • 1-byte: An unsigned byte value (0 to 254) used to indicate the length of the data payload following it. Useful for small items.
    • 4-byte: An unsigned integer value (0 to 2,147,483,647) used to indicate the length of the data payload following it. Useful for larger items.
• data (OPTIONAL)
  • A run of bytes representing the actual binary data for this type of value.

In the name of efficiency, the length and data fields are optional depending on the type of value being encoded. Some value are simple enough that just writing the 1-byte ASCII marker into the stream is enough to represent the value (e.g. null) while others have a type that is specific enough that no length is needed as the length is implied by the type (e.g. int32).

The specifics of each data type will be spelled out down below for more clarity.

The basic organization provided by this tuple (type-length-data) allows each JSON construct to be represented in a binary format that is simple to read and write without the need for complex/custom encodings or null-termating bytes anywhere in the stream that has to be scanned for or references resolved.

Value Types

This section describes the mapping between the 5 discrete value types from the JSON specification into the Universal Binary JSON format.

JSON

The JSON specification defines 7 value types:

• string
• number
• object (container)
• array (container)
• true
• false
• null

Of those 7 values, 2 of them are types describing containers that hold the 5 basic values. We have a separate section below for looking at the 2 container types specifically, so for the time being let’s only consider the following 5 discrete value types:

• string
• number
• true
• false
• null

Most of these types have a 1:1 mapping to a primitive type in most popular programming languages (Java, C, Python, PHP, Erlang, etc.) except for number. This makes defining the types for the 4 easy, but let’s take a closer look at how we might deconstruct number into its core representations.

Number Type

In JavaScript, the Number type can represent any numeric value where as many other languages define numbers using 3-6 discrete numeric types depending on the type and length of the value being stored. This allows the runtime to handle numeric operations more efficiently.

In order for the Universal Binary JSON specification to be a performant alternative to JSON, support for these most common numeric types had to be added to allow for more efficient reading and writing of numeric values.

number is deconstructed in the Universal Binary JSON specification and defined by the following signed numeric types:

• byte (8-bits, 1-byte)
• int16 (16-bits, 2-bytes)
• int32 (32-bits, 4-bytes)
• int64 (64-bits, 8-bytes)
• float (32-bits, 4-bytes)
• double (64-bits, 8-bytes)
• huge (arbitrarily long, UTF-8 string-encoded numeric value).

Trying to maintain a single number type represented in binary form would have lead to parsing complexity and slow-downs as the processing language would have to further inspect the value and map it to the most optimal type. By pre-defining these different numeric types directly in binary, in most languages the number can stay in their optimal form on disk and be deserialized back into their native representation with very little overhead.

When working on a platform like JavaScript that has a singular type for numbers, all of these data types (with the exception of huge) can simply be mapped back to the Number type with ease and no loss of precision.

When converting these formats back to JSON, all of the numeric types can simply be rendered as the singular number type defined by the JSON spec without issue; there is total compatibility!

Value Type Summary

Now that we have clearly defined all of our (signed) numeric types and mapped the 4 remaining simple types to Universal Binary JSON, we have our final list of discrete value types:

1. null
2. true
3. false
4. byte
5. int16
6. int32
7. int64
8. float
9. double
10. huge
11. string

Now that we have defined all the types we need, let’s see how these are actually represented in binary in the next section.

Universal Binary JSON

The Universal Binary JSON specification defines a total of 13 discrete value types (that we saw in the last section) all delimited in the binary file by a specific, 1-byte ASCII character (optionally) followed by a length and (optionally) a data payload containing the value data itself.

Some of the values (null, true and false) are specific enough that the single 1-byte ASCII character is enough to represent the value in the format and they will have no length or data section.

All of the numeric values (except huge) automatically imply a length by virtue of the type of number they are. For example, a 4-byte int32 always has a length of 4-bytes; an 8-byte double always requires 8 bytes of data.

In these cases the ASCII marker for these types are immediately followed by the data representing the number with no length value in between.

Because string and huge are potentially variable length, they contain all 3 elements of the tuple: type-length-data.

This table shows the official definitions of the discrete value types:

With each field of the table described as:

Type

• The binary value data type defined by the spec.

Size

• The byte-size of the construct, as stored in the binary format. This is not the value of the length field, just an indicator to you (the reader) of approximately how much space writing out a value of this type will take.

Marker

• The single ASCII character marker used to delimit the different types of values in the binary format. When reading in bytes from a file stored in this format, you can simply check the decimal value of the byte (e.g. ‘A’ = 65) and switch on that value for processing.

Length?

• Indicates if the data type provides a length value between the ASCII marker and the data payload.
• Many of the data types represented in the binary format either don’t have a length (null, true or false) or their types (e.g. the numeric values) are specific enough that the length is implied.
• When specifying the length for a string or huge value (UTF-8 encoded), the length must represent the number of bytes of the UTF-8 string and not the number of characters in the string.

Data?

• Indicates if the data type provides a data payload representing the value.
• Most types except for null, true and false provide a data payload indicating their value.
• Variable-length types like string and huge do not provide a data payload when they are empty (i.e. length of 0).More specifically, if you are writing a parser for the Universal Binary JSON format and you encounter a string of length 0, you know the very next byte is an ASCII marker for another value since the string has no data payload.

To further clarify the binary layout of these data types, below are some visual examples of what the bytes would look like inside of a binary JSON file.

NOTE: [ ]-block notation is used for readability, the [ ] characters are not actually written out in the binary format.

Now that we have seen how the JSON data value types map to the binary format, in the next section we will see how we can combine these values together into the two container types (objects and arrays) to create complex object hierarchies using the Universal Binary JSON format.

Container Types

In this section we will look at the 2 remaining JSON value types that we are referring to as “container types”, namely object and array.

JSON

The two JSON container types are described as follows:

object

• A construct containing 0 or more name-value pairings, where the name is always a string and the value can be any valid value type including container types themselves.

array

• A flat list of values only, where the values can by any valid value type including container types themselves.
• The JSON specification does not make it a requirement that the values in an array are all of the same type and neither does the Universal Binary JSON specification.

Given these two constructs, let’s see how they are modeled in the Universal Binary JSON format.

Universal Binary JSON

The two container types defined by JSON are modeled using the same tuple that defines all of our other data structures in this specification so far with a minor modification: the length value is considered a count of the child elements the container holds. It does not mean the byte length of the child elements.

More specifically, the tuple stays exactly the same, it is just the meaning of the center length element that changes:

[type, 1-byte char]([count, 1 or 4-byte integer])([data])

All the code used to process the constructs defined by this specification stays the same, but when an object or array construct are encountered, the code needs to be aware that the length value is the child element count so it can know when the scope of the container ends.

For example, if you have an object that contains 4 arrays of length 50, the length argument for the object is 4 (because it contains the four arrays) while the length argument for each array is 50 (because they each hold 50 elements).

Additionally, the only optional field in the tuple for container types is data, if the container is empty and contains no elements (i.e. the length is 0) then there is no data segment.

All together, the definitions for the object and array container types looks like this:

The details for each field are the same as described for the non-container values in the previous section with the one caveat that length is a count of child elements and not the number of bytes representing the contents of the container.

Let’s look at a quick example of encoding an object, again using the handy [ ]-notation we used before simply for readability (the [ ] chars are not written out or part of the file format).

Consider the following JSON (30-bytes compacted):

{
    "id": 1234567890,
    "name": "bob"
}

Storing that object in the Universal Binary JSON format would look like this (whitespace added for readability):

[o][2] 2 bytes

[s][2][id][I][1234567890] 4 + 5 = 9 bytes

[s][4][name][s][3][bob]   6 + 5 = 11 bytes

Walking through our example above, using a word-journey this is what a parser might see and do:

1. I see an “o”, so I know I am parsing an object and that the next byte is the length (or count) for this object.
2. I see a “2″, so I know the object contains 2 elements that I must account for to know when the object scope is closed (because we don’t use the { } brackets like JSON).
3. I see an “s”, knowing how the name-value pairings inside of an object work, I know this is the name portion of some upcoming value.
4. I see an “I”, I know this is an int32 value and that it belongs to the name I parsed in the previous step.
5. I see another “s”, I know this is a new name-value pair and this is the name portion.
6. I see another “s” and know this is the value belonging to the name I just processed.
7. I have just parsed 2 values, so now I know the object scope is closed.

Encoding objects containing other objects would work identically except we would have encountered another ‘o’ or ‘O’ marker and descended a level further into a new object.

Let’s look at another example, this time a simple JSON array construct (remember, they only contain values and not name-value pairs like objects).

This array is 48-bytes in compacted JSON:

[
    null,
    true,
    false,
    4782345193,
    153.132417549,
    "ham"
]

Storing the array in the Universal Binary JSON format would look like this (whitespace added for readability):

[a][6] - 2 bytes

[Z] - 1 byte

[T] - 1 byte

[F] - 1 byte

[I][4782345193] - 5 bytes

[D][153.132417549] - 9 bytes

[s][3][ham] - 5 bytes

Because the container types specify their total child element count, it is easier and faster for parsers to know when the scope of a container has closed or is still open waiting for more children (e.g. in the case of streaming over the network). This is not unlike the high-performance Redis protocol.

This also has the added benefit of not needing any terminating values in the binary that need to be scanned for to know when a container-scope is closed. This way data can be read in chunks and not read-and-scanned byte-by-byte.

As was mentioned previously though, there are some cases where having an unbounded container are important (for example, streaming content from a server as it generates it on-the-fly).

In the next section we will take a look at the Universal Binary JSON constructs that are optimized for streaming. Fortunately, there are only 3 and they are just as easy as the constructs we have covered so far!

Streaming Types

The Universal Binary JSON specification is optimized for fast read-speed by prefixing the byte-length of every construct to the front of it, this allows parsers to digest entire chunks of the data stream at a time without scanning for terminating byte values.

Unfortunately, this model of data becomes very expensive (and sometimes impossible) to adhere to in a streaming-friendly environment where a server may be generating UBJ formatted data on-the-fly and streaming it back in real time to the client.

If the server had to adhere to the prefixed-length requirement of this specification up until now, it would have to generate, buffer and count all the elements in its reply before writing out the Universal Binary JSON so it could correctly prefix the lengths to all the containers.

In this section of the specification we look at 1 new additional type to the Universal Binary JSON specification that compliments our streaming scenario and then two minor changes to the existing container types to enable easy and efficient streaming with unknown-length support for our array and object containers.

No-Op Type

The noop value stands for “No Op” or “No Operation”, it is a specific value (like ‘Z’ for null, ‘T’ for true and ‘F’ for false) that is useful in streaming scenarios where an acknowledge of life needs to be sent between two end points, but the confirmation being sent cannot change the meaning of the data it is sent within.

The most common use for such a value type is as a keep-alive signal from a server to the client; letting the client know the server is possibly operating on a long-running job and is still alive, but just isn’t ready to send more data yet.

The noop type is defined as follows:

Any parser code written to load the Universal Binary Spec needs to be aware that encountering the ‘N’ marker in files of any kind is valid and is merely useful as a signal mechanism from producer to consumer to say “Hey, I am still alive.”, the marker is intended to be safely ignored if the server or client doesn’t need the acknowledgement.

In order for this keep-alive-esque construct to work, the specification had to define a single byte value that had no meaning for the server and client to exchange if needed, but caused no modification to the meaning of the data that they are exchanging.

In code that handles streaming from a server, your handler for the noop type might just reset a disconnect timer. In code that handles UBJ files, you would simply ignore the noop marker when you encountered it in the file because it would mean nothing.

If your interaction with the Universal Binary JSON format is primarily as a file format, it is likely that you may never need to use the noop type; its value becomes more apparent in long-lived, client-server, data-streaming scenarios.

Unknown-Length Containers

The Universal Binary JSON specification supports containers (array and object) of unknown length to be specified when the producer of the binary data cannot (efficiently) know in advance how many elements it is going to write out.

In these cases, the lowercased, 1-byte-length versions of array or object must be used (‘a’ or ‘o’ markers) with a length value of 0xFF (255) as well as specifying an ‘E’ terminator character after the last element in the container.

The ‘E’ type used to delimit the end of unknown-length containers is defined as follows:

An example would look like this:

[a][255]

[S][3][bob]

[I][1024]

[T]

[F]

[S][4][ham!]

[E]

The three key elements being the lowercased ‘a’ marker, the length of 0xFF (255) and the ‘E’ marker at the end of the container.

Another example might look like this:

[o][255]

[B][4]

[D][21.786]

[N]

[Z]

[h][27][131.098412283059e2371293452]

[E]

You might notice in the example above we injected a noop instruction right in the middle, before the null. As mentioned in the No-Op Type section, this is valid and can occur at any time while parsing the contents of an unknown-length container.

If your parser has no need for recognizing the noop code (e.g. listening for a keep-alive) then it can just be safely ignored and parsing continued (hence the name “no-op”). It is up to the implementation to decide what to do with the noop type.

You might be wondering how using a 1-byte ‘E’ as a terminator to an unbounded container can work and not get confused with say another ‘E’ inside of a string, the reason this works is because none of the discrete value types (numeric, string, etc.) are of unknown length.

The lengths of all the values contained inside of the container are known and must be read completely, doing so will guarantee that the ‘E’ is only ever encountered by itself as an element marker which is easily handled by parsing code to know the scope of the container has been closed.

Size Requirements

The Universal Binary JSON specification tries to strike the perfect balance between space savings, simplicity and performance.

Data stored using the Universal Binary JSON format are on average 30% smaller as a rule of thumb. As you can see from some of the examples in this document though, it is not uncommon to see the binary representation of some data lead to a 50% or 60% reduction in size.

The size reduction of your data depends heavily on the type of data you are storing. It is best to do your own benchmarking with a comprehensive sampling of your own data.

Size Reduction Tips

The amount of storage size reduction you’ll experience with the Universal Binary JSON format will depend heavily on the type of data you are encoding.

Some data shrinks considerably, some mildly and some not at all, but in every case your data will be stored in a much more efficient format that is faster to read and write.

Below are pointers to give you an idea of how certain data may shrink in this format:

• null, true and false values will compress 75% (80% in the case of false)
• large numeric values (> 5 digits < 20 digits) will compress on average 50%.
• string values
  • of length <= 254 stay the same size.
  • of length > 254 are 3-bytes bigger per string.
• object and array values compress 1-byte-per-element.

One of the great things about the Universal Binary JSON format is that even though most all your data will be represented in a smaller footprint, you still get two big wins:

1. A smaller data format means faster writes and smaller reads. It also means less data to process when parsing.
2. Binary format means no encoding/decoding primitive values to text and no parsing primitive values from text.

Endianness

The Universal Binary JSON specification requires that all numeric values be written in Big-Endian order.

MIME Type

The Universal Binary JSON specification is a binary format and recommends using the following mime type:

This was added directly to the specification in hopes of avoiding similar confusion with JSON.

File Extension

“ubj” is the recommended file extension when writing out files using the Universal Binary JSON format (e.g. “user.ubj“).

The extension stands for “Universal Binary JSON” and has no known conflicting mappings to other file formats.

History

The original specification for Universal Binary JSON was written in September of 2011 in a Google Doc available here. The specification was formalized after Draft 4 and moved to its own domain (http://ubjson.org) to ease community review, feedback and interaction.

Changelog (MM-DD-YY)

• 05-25-13: Removed ‘Draft 9′ section. Changes are finalized.
• 05-03-13: Finished revised Type Reference page for Draft 9.
• 04-14-13: Added CHAR and UINT8 types to spec. Finalized Draft 9 Type Reference.
• 04-01-13: Continued to rewrite all the Type Reference sub pages, examples, definitions.
• 03-25-13: Refine detailed information about numeric types. Updated Draft 9 Progress section.
• 03-24-13: Rewrote of Type Reference page and Value Type subpages.
• 02-01-13: Began rewriting the Type Reference to integrate the Draft 9 changes.
• 07-23-12: Further clarified the breaking numeric changes coming in Draft 9.
• 05-29-12: Began working on integrating the Draft 9 corrections and changes.
• 02-11-12: Added clarifying information about infinite numeric types needing to be encoded as null per the JSON spec recommendation.
• 02-09-12: Fixed typo in the JSON numeric type section and the number of discrete UBJSON types in table.
• 01-21-12: Added numeric type signage information to specification. Restructured Value Type section (UBJ implementation details were listed a section too early). Refined the Type Reference page to include more detailed information.
• 11-10-11: Added clarification surround the valid positions of the noop type within a stream.
• 10-04-11: Added “Why not JSON+gzip?” section. Added int16 and float types.
• 10-03-11: Added .NET implementation to the Libraries page.
• 10-02-11: Clarified huge usage. Full rewrite pass on the entire specification. Rewrote the Type Reference page. Added MIME Type and File Extension pages. Updated Links page. Moved Examples and Best Practices to sub pages of Specification to make navigation easier to grok.
• 10-01-11: Wrote up Streaming section. Edited the Type Reference page and updated it with the new information.
• 09-29-11: Added Examples page. Modified specification to add new short-types of variable length items AND variable length container support.
  • Significant edits to Value Types and Container Types to include the support of the new shorter types and examples.
  • Rewrote the Size Requirements section to factor in the new types and add clarification down to the byte level of data representation in the binary format.
• 09-28-11: Added a slew of ideas in Under Consideration after collecting feedback from the CouchDB dev team.
• 09-27-11: Added Java Library implementation link. Added JSON Structures section.
• 09-26-11: Added clarification to numeric encoding (huge and double). Updated Draft version as a result. Added Thanks. Removed Performance, there is no way to provide general-case performance metrics as it depends on the implementation. That will be a property of the individual implementations to communicate.
• 09-25-11: Added Value Types, edited Size Requirements. Added formatting. Edited Best Practices. Added Endianness and Status. Added Container Types.
• 09-24-11: Added Goals, Data Format and Size Requirements sections. Added Type Reference and Best Practices pages.
• 09-23-11: Added ToC, Introduction, Why and Recommendations sections.

Requests for Enhancement (RFE)

All (proposed) changes to the specification are being tracked in GitHub.