Model the absence of values in Protobuf

Protocol Buffers version 3 (often shortened to Protobuf) implicitly treat every field as optional: whenever the producer of a message does not provide a value for a given field, consumers reading that field will get its default value.

Depending on your use case it might be relevant for your consumer to be able to make the difference between:

• “the producer didn’t set any value” and
• “the producer explicitly set its default value”

If that’s not the case, you can stop reading here, since the rest of the post will not be relevant to you :)

Note: I’ll use Go examples in the rest of the post, but the behaviour described below applies to all languages supported by Protobuf (with language-specific differences).

The initial model #

Let’s take a concrete example: you are creating a protocol to describe incremental changes to an entity.

message Thing {

	// Required.
	int32 id = 1;

	// Optional. Only provided when updated.
	bool enabled = 2;
 
	// Optional. Only provided when updated.
	Status status = 3;

	// Optional. Only provided when updated.
	Piece piece = 4;
}

enum Status {
	STATUS_INVALID = 0;
	// other values ...
}

message Piece {
	// its fields are not relevant
}

One way to model this behaviour is to adopt the following conventions:

• id is a required field and will be provided in every message
• the other (“optional”) fields will only be provided whenever their value has changed; if not, they will be omitted and consumers can assume that they are unchanged.

If you use the model as it is and the producer does not set a value for the “optional” fields, the code generated by Protobuf (= the consumer) would return the following values:

enabled: false
status: STATUS_INVALID
piece: <absent> // the actual value is language-dependent: for example, in Go this would be nil.

This is because consumers will implicitly return the default value of any missing field, which depends on the field type.

Absence of message field values #

The default value of message fields is language-dependent: in Go, it is nil; in Java, it is null, and so on. This means that consumers can always understand if a message field is absent.

Absence of scalar field values #

The default value of scalar fields (booleans, bytes, numbers, strings) depends on the actual type, but doesn’t allow you to understand if a field is absent (e.g. the default value of numbers is 0, of booleans is false). As an example, this is what the Go enabled field would look like:

type Thing struct {

	// ...
	
	Enabled bool `protobuf:"varint,2,opt,name=enabled,proto3" json:"enabled,omitempty"`

	// ...
}

Since it has been mapped to a bool, if you get false after deserialization there’s no way to tell whether the consumer implicitly used the default value or the producer explicitly set the field to false.

If you want to explicitly mark the absence of a scalar type, you can use a wrapper type such as BoolValue as in the following, updated model.

// NOTE only showing differences with the previous model, for brevity's sake

// NEW
import "google/protobuf/wrappers.proto";

message Thing {

	// CHANGED
	google.protobuf.BoolValue enabled = 2;

	// ...
}

Using the BoolValue type, the generated field now looks like this:

type Thing struct {  

	// ...

	Enabled *wrapperspb.BoolValue `protobuf:"bytes,2,opt,name=enabled,proto3" json:"enabled,omitempty"`

	// ...
}

The struct now has a pointer field, so you can now distinguish the absence of value (nil) from an explicitly-set value.

Protobuf provides wrapper types for all scalar types, so this approach works for any of them.

Absence of enum field values #

There is no wrapper type for enums, so a possible solution is to wrap it inside a message field so that it will have a nil default value.

// NOTE only showing differences with the previous model, for brevity's sake

message Thing {

	// ...
 
	// CHANGED
	StatusValue status = 3;
}

message StatusValue {
	Status value = 1;
}

Final model #

The final model would look like this:

import "google/protobuf/wrappers.proto";

message Thing {

	// Required.
	int32 id = 1;
 
	google.protobuf.BoolValue enabled = 2;
 
	StatusValue status = 3;
 
	Piece piece = 4;
}

message StatusValue {
	Status value = 1;
}

enum Status {
	// enum values ...
}

message Piece {
	// its fields are not relevant
}