What JSON Costs, and What Binary Buys

JSON won for good reasons: every language can parse it, you can read it in a terminal, and you can paste it into a bug report. Those are real advantages and they are why it is the correct default for most APIs.

It also has specific, well-known problems that matter more the further the data travels from a human.

The failure modes

Numbers are ambiguous. JSON has one number type and no way to say whether 1 is an integer or a float. Worse, the JSON specification puts no limit on precision, but JavaScript parses every number as a 64-bit float — so integers above 2⁵³ lose precision silently. A Twitter-scale id or a Snowflake id pasted into JavaScript comes back subtly wrong, which is why large-id APIs return them as strings.

There are no bytes. Binary data must be base64-encoded, which inflates it by 33% and hides the fact that it was ever binary.

There are no dates. Everyone agrees to use ISO 8601 strings, and everyone implements the agreement slightly differently. Time zone handling is by convention, and conventions are not enforced.

Every record carries its own field names. This one is the big one for size.

{"userId":1001,"eventType":"click","timestamp":1735689600}

Fifty-seven bytes, of which userId, eventType and timestamp are 26 — and those same 26 bytes are repeated in every one of the ten billion records you store. You are paying to say the same thing over and over to a machine that already knows it.

Binary formats that keep the field names

The obvious first move is to keep JSON’s data model but encode it in binary — MessagePack, BSON, CBOR, and several others do exactly this.

The result is disappointing, and it is worth understanding why. Encoding the example above in MessagePack gives roughly 44 bytes against JSON’s 57 — about 23% saved. The field names are still in there, just with a length prefix instead of quotes and colons.

The change that actually matters

To remove the field names you need somewhere else to put them — a schema that both sides know, so the encoded record can refer to fields by number instead of by name.

JSON         {"userId":1001,"eventType":"click","timestamp":1735689600}    57 bytes
MessagePack  binary, names retained                                        ~44 bytes
Tagged       field 1 varint 1001 | field 2 str click | field 3 varint ...  ~17 bytes

Now the record carries tags rather than names, and integers use a variable-length encoding so small numbers occupy one or two bytes rather than eight. The saving is a factor of three or more, and it grows with how repetitive your records are.

The cost is that the bytes are meaningless without the schema. You cannot read them in a terminal, and you have to distribute schema definitions to everyone who reads or writes the data. That is the trade, and the next lesson is about the machinery for managing it.

Varints, briefly

The other half of the saving is worth knowing because it explains a surprise.

A variable-length integer uses the top bit of each byte as a continuation flag and the other seven bits for data. Numbers under 128 fit in one byte, under 16,384 in two, and so on. Most real numbers are small, so most fields shrink.

The surprise: negative numbers are the worst case. In two’s complement, -1 has every high bit set, so a naive varint encodes it in ten bytes. Formats solve this with zigzag encoding, which maps signed integers onto unsigned ones so that small magnitudes stay small in both directions: 0, -1, 1, -2, 2 become 0, 1, 2, 3, 4.

If you ever see a field that is unexpectedly large on the wire, a negative number in a non-zigzag varint is a strong first guess.

What to take away

JSON’s real cost is repeating field names in every record and forcing readers to scan text. Binary formats that keep the names fix the small half of that. Removing the names requires a schema — which buys size and speed, and costs you the ability to read your own data without tooling.