Inspecting binary Protobuf data

Table of Contents

Note: This is a short post, just to quickly capture something I’ve recently learned while troubleshooting an issue with Protobuf messages.

Binary Protobuf payloads can be inspected using Protoscope. This can be useful, for example, when troubleshooting unexpected results in the (un)marshalling of Protobuf messages.

Here is a complete example. Given the following Protobuf message:

syntax = "proto3";

import "google/protobuf/timestamp.proto";

package v1;

message Metadata {
  bytes hash = 1;
  string name = 2;
  int64 size = 3;

  google.protobuf.Timestamp created_at = 10;
}

Imagine that you have saved the binary payload to a file, say hello.bin (the code used to generate this file is in the Appendix, for reference). You can now use protoscope to see its unmarshaled contents:

$ protoscope hello.bin 
1: {"hello world"}              # field number 1: 'hash'
2: {"hello.txt"}                # field number 2: 'name' 
3: 123                          # field number 3: 'size'
10: {                           # field number 10: 'created_at' (itself a message containing two fields: Seconds and Nanos)
  1: 1701983104
  2: 221713000
}

protoscope provides several CLI flags to show additional information about the Protobuf structure; the following one, for example, shows the wire types:

$ protoscope -explicit-wire-types hello.bin
1:LEN {"hello world"}
2:LEN {"hello.txt"}
3:VARINT 123
10:LEN {
  1:VARINT 1701983104
  2:VARINT 221713000
}

Wire types are defined in the Protobuf wire format.

Should you be interested in how Protobuf marshals optional fields, I’ve previously written about it here.

Appendix #

package main

import (
	"os"

	"google.golang.org/protobuf/proto"
	"google.golang.org/protobuf/types/known/timestamppb"

	v1 "protoscope-example/gen/example/v1"
)

func main() {
	msg := v1.Metadata{
		Hash:      []byte("hello world"),
		Name:      "hello.txt",
		Size:      123,
		CreatedAt: timestamppb.Now(),
	}

	data, err := proto.Marshal(&msg)
	if err != nil {
		panic(err)
	}

	file, err := os.OpenFile("hello.bin", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
	if err != nil {
		panic(err)
	}
	defer file.Close()

	_, err = file.Write(data)
	if err != nil {
		panic(err)
	}
}