The Tag Is the Identity — Learn Data Systems

The last lesson ended with a problem: to drop field names from the wire, both sides need a schema. Here is what that looks like.

message Order {
  int64  id       = 1;
  string customer = 2;
  int32  total    = 3;
}

Those numbers are the point. On the wire, a field is written as its tag, its type, and then its value:

[tag 1][varint][1001]  [tag 2][length 3]["ada"]  [tag 3][varint][4200]

The name customer appears nowhere in the encoded bytes. It exists only in the schema file, for humans.

The idea

The tag number is the field’s identity. The name is a label for people reading the schema. Change the name and nothing on the wire changes; change the number and you have created a different field.

That single decision is what makes safe evolution possible, and it produces rules that are easy to remember:

Renaming a field is free. The tag is unchanged, so every reader still finds the value. Rename customer to customerName and deploy; nobody notices.
Reusing a tag number is catastrophic. Delete field 2 and later add a new field with tag 2, and old data will be parsed as though the old values belong to the new field. This is why Protobuf has a reserved keyword — you retire a number permanently rather than recycling it.

Skipping what you do not understand

The other half of the mechanism is the length prefix.

When a reader meets tag 7 and its schema has no field 7, it does not have to guess where that field ends. The encoding tells it — either by an explicit length for variable-size types, or by a wire type that implies a fixed size. The reader skips exactly that many bytes and carries on with the next field.

Why this is the whole trick

Forward compatibility requires old code to survive data containing things it has never heard of. That is only possible if unknown data is skippable, and it is only skippable if its length is knowable without understanding it. Length prefixes are what make forward compatibility mechanical rather than hopeful.

JSON gets a weaker version of the same property for free, because a parser can find the end of a value structurally. That is genuinely why “ignore unknown fields” works in JSON APIs — and why strict validators that reject unknown fields quietly destroy forward compatibility for a small security benefit.

Required is a trap

Protobuf 2 had required fields. Protobuf 3 removed the keyword entirely, and the reasoning is instructive.

A required field can never be removed, because every existing reader will reject data without it — and you may not control every reader. There is no migration path that does not involve upgrading everything simultaneously, which is the one thing you cannot do.

So in practice: every field is optional, and absence is handled by the application. It feels weaker than it is. You have moved the check from the parser to your code, where you can decide what a missing value means, rather than having the parser fail before you get a chance.

Work the matrix

The change

The encoding

Fields identified by a numeric tag and length-prefixed, so unknown fields can be skipped byte-exactly.

Add a required field

New field with no default; readers must find a value.

Backward compatible

safe

new writer → old reader

The old reader never knew to require it, so an extra field is harmless in this direction.

Forward compatible

breaks

old writer → new reader

Old writers never sent it and there is no default, so the new reader has nothing to fall back on. This is the classic broken deploy.

Needs a multi-step rollout

One direction fails, so during a rolling deploy some instance will meet data it cannot handle. Split it: ship a version that tolerates both shapes, migrate the data, then ship the version that assumes the new shape.

Switch between JSON and tagged binary on the rename case — that is the clearest demonstration of what tags buy you. Then try adding a required field and note that no encoding saves you: it is a schema-design problem, not an encoding one.

The other approach: ship the writer’s schema

Avro takes a different route, and it is worth knowing because it suits a different problem.

Avro records contain no tags at all — just values, concatenated. Parsing is only possible with the exact schema that wrote them. So the reader is given two schemas: the writer’s schema (how these bytes were laid out) and the reader’s schema (what this code wants), and the library resolves between them by field name, filling in defaults for anything the reader expects but the writer did not have.

That sounds like more work, and it buys two things:

The smallest encoding of the three, since neither names nor tags are on the wire.
Dynamically generated schemas. With no tag numbers to keep stable, a schema can be derived from something else — a database table, say — and regenerated whenever that changes. Tag-based formats need a human to assign and preserve numbers, which is exactly what you cannot do if the schema is generated.

The catch is that the writer’s schema must travel with the data. For a large file, write it once in the header and amortise it over millions of records — which is why Avro is the usual choice for archival files and Hadoop-style storage. For individual messages, use a schema registry and put a small schema id in each record.

Check yourself

A team deletes a deprecated Protobuf field with tag 4, then a year later adds a new field and gives it tag 4. What happens?

What to take away

Tags decouple a field’s identity from its name, and length prefixes make unknown fields skippable. Between them you get renames for free and forward compatibility by construction — provided you never reuse a tag number, and treat every field as optional.