JSON to Proto Definition Converter

JSON to Proto Definition: Buf CLI, Field Numbers, Well-Known Types, and Proto3 Optionality

Converting JSON to a Protobuf .proto file requires understanding decisions JSON doesn't force you to make: which integer type (int32, int64, uint32, uint64, sint64), whether to use optional vs implicit presence, which field numbers to reserve for deleted fields, and how to represent dates (google.protobuf.Timestamp), arbitrary JSON (google.protobuf.Struct), and nullable primitives (google.protobuf.StringValue wrappers). The modern toolchain uses Buf CLI instead of raw protoc, because Buf handles module management, breaking change detection, and linting out of the box.

Live Example: Complete Proto3 Message from JSON

// Input JSON (order event)
{
  "order_id": "ord_abc123",
  "user_id": 10042,
  "status": "PENDING",
  "total_cents": 4998,
  "discount_code": null,
  "items": [
    { "sku": "SKU-001", "qty": 2, "price_cents": 1999 }
  ],
  "shipping": {
    "address": "123 Main St",
    "city": "Springfield",
    "country": "US"
  },
  "metadata": { "source": "mobile", "utm_campaign": "summer24" },
  "created_at": "2024-03-15T10:00:00Z",
  "updated_at": null
}

// Generated .proto file
syntax = "proto3";

package orders.v1;

option go_package = "github.com/example/orders/v1;ordersv1";
option java_package = "com.example.orders.v1";

import "google/protobuf/timestamp.proto";
import "google/protobuf/wrappers.proto";

message OrderItem {
  string sku         = 1;
  int32  qty         = 2;
  int64  price_cents = 3;  // use int64 for money — avoids overflow at high volumes
}

message ShippingAddress {
  string address = 1;
  string city    = 2;
  string country = 3;
}

enum OrderStatus {
  ORDER_STATUS_UNSPECIFIED = 0;  // proto3 enum must have 0 default value
  ORDER_STATUS_PENDING     = 1;
  ORDER_STATUS_CONFIRMED   = 2;
  ORDER_STATUS_SHIPPED     = 3;
  ORDER_STATUS_DELIVERED   = 4;
  ORDER_STATUS_CANCELLED   = 5;
}

message Order {
  string      order_id      = 1;
  int64       user_id       = 2;
  OrderStatus status        = 3;
  int64       total_cents   = 4;

  // Nullable string — use StringValue wrapper for explicit null distinction
  // (without wrapper: empty string and "not set" are indistinguishable in proto3)
  google.protobuf.StringValue discount_code = 5;

  repeated OrderItem items    = 6;
  ShippingAddress    shipping = 7;

  // Dynamic key-value pairs — map for string values
  map<string, string> metadata = 8;

  google.protobuf.Timestamp created_at = 9;

  // Optional timestamp — using proto3 optional for explicit presence
  optional google.protobuf.Timestamp updated_at = 10;

  // Reserved field numbers for deleted fields — prevents accidental reuse
  reserved 11, 12, 15 to 20;
  reserved "legacy_customer_id", "deprecated_status";
}

Integer Type Selection

// Proto3 integer types — choose based on expected range and encoding
//
// int32:    varint encoding, -2B to +2B. Good for: qty, page numbers, counts.
// int64:    varint encoding, -9×10^18 to +9×10^18. Good for: IDs, money, timestamps.
// uint32:   unsigned, 0 to 4B. Good for: positive counts, ports, millisecond durations.
// uint64:   unsigned, 0 to 1.8×10^19. Good for: byte sizes, large unsigned IDs.
// sint32:   ZigZag encoding — efficient for negative values. Good for: coordinates.
// sint64:   ZigZag encoding, large negative. Good for: balance deltas.
// fixed32:  4 bytes always. Good for: values always near 2^32.
// fixed64:  8 bytes always. Good for: 64-bit hashes, values always near 2^64.

// JSON number inference rules:
// JSON integer, small range  → int32
// JSON integer, large values → int64  (use for ALL IDs — JS number max is 2^53)
// JSON float                → double  (use float only for ML model weights)
// Financial amounts          → int64 (store as cents/smallest unit, not decimal)

// WRONG — float loses precision for money:
// amount: float = 4;   // 15.92 becomes 15.9200001...

// CORRECT — store as integer cents:
// price_cents: int64 = 4;   // 1592 = $15.92 exactly

// Well-known types for semantic clarity:
// import "google/protobuf/timestamp.proto"
// google.protobuf.Timestamp created_at = 9;
// → seconds + nanos since Unix epoch (UTC), maps to JS Date, Python datetime, etc.

// import "google/protobuf/duration.proto"
// google.protobuf.Duration timeout = 10;
// → seconds + nanos duration

// import "google/protobuf/struct.proto"
// google.protobuf.Struct extra_data = 11;  // arbitrary JSON object
// google.protobuf.Value  any_value  = 12;  // any JSON value (string, number, etc.)
// google.protobuf.ListValue array    = 13;  // JSON array

Buf CLI: Modern Protobuf Toolchain

// Buf CLI replaces raw protoc for most use cases
// npm install -g @bufbuild/buf  OR  brew install bufbuild/buf/buf

// buf.yaml — workspace configuration
version: v2
modules:
  - path: proto
deps:
  - buf.build/googleapis/googleapis     # for google.protobuf.Timestamp etc.
  - buf.build/bufbuild/protovalidate    # validation annotations
lint:
  use:
    - STANDARD   # enforces naming conventions, package structure, etc.
breaking:
  use:
    - FILE       # detect breaking changes compared to previous version

// buf.gen.yaml — code generation config
version: v2
plugins:
  - remote:  buf.build/protocolbuffers/ts
    out:     src/generated/ts
    opt:
      - target=ts
  - remote:  buf.build/connectrpc/es
    out:     src/generated/connect

// Commands:
// buf lint                      — check naming/style
// buf build                     — compile and validate
// buf generate                  — run code generators
// buf breaking --against .git#branch=main  — detect breaking changes in CI
// buf dep update                — update dependencies to latest versions

// Generated TypeScript with @bufbuild/protoc-gen-es:
// src/generated/ts/orders/v1/orders_pb.ts
import { create, fromJson } from "@bufbuild/protobuf";
import { OrderSchema } from "./orders/v1/orders_pb";

const order = fromJson(OrderSchema, {
  orderId:    "ord_abc123",
  userId:     10042n,        // BigInt for int64
  status:     "ORDER_STATUS_PENDING",
  totalCents: 4998n,
  items:      [{ sku: "SKU-001", qty: 2, priceCents: 1999n }],
  createdAt:  "2024-03-15T10:00:00Z",
});

console.log(order.orderId);  // "ord_abc123" — fully typed

Service and RPC Definition

// Full service definition with streaming variants
syntax = "proto3";

package orders.v1;

import "orders/v1/orders.proto";

service OrderService {
  // Unary RPC
  rpc CreateOrder(CreateOrderRequest) returns (Order);
  rpc GetOrder(GetOrderRequest)       returns (Order);

  // Server streaming — returns multiple orders (pagination alternative)
  rpc ListOrders(ListOrdersRequest)   returns (stream Order);

  // Client streaming — bulk create
  rpc BulkCreateOrders(stream CreateOrderRequest) returns (BulkCreateResponse);

  // Bidirectional streaming — live order updates
  rpc WatchOrders(WatchOrdersRequest) returns (stream OrderEvent);
}

message CreateOrderRequest {
  string user_id = 1;
  repeated OrderItemInput items = 2;
  ShippingAddress shipping = 3;
  optional string coupon_code = 4;
}

message GetOrderRequest {
  string order_id = 1;
}

message ListOrdersRequest {
  string user_id   = 1;
  int32  page      = 2;
  int32  page_size = 3;
}

message BulkCreateResponse {
  int32          created_count = 1;
  repeated string failed_ids   = 2;
}

Best Practices for Production

• Never reuse or change field numbers once a message is in production: Field numbers are the actual wire representation — they identify fields in binary serialized data, not the field names. Changing user_id = 2 to something else, or reusing number 2 for a new field, silently corrupts all existing serialized data. Use reserved 2; to permanently prevent reuse after a field is deleted.
• Use Buf CLI's buf breaking in CI to catch breaking changes: Breaking changes in proto files are subtle — adding a required field, removing a field, changing a type, reordering enum values. Buf detects all of these automatically. Run buf breaking --against .git#branch=main on every PR that touches .proto files.
• Use google.protobuf.Timestamp for all timestamps, never int64 milliseconds: Raw int64 milliseconds are ambiguous (seconds? milliseconds? microseconds?). Timestamp carries the unit and timezone (UTC) unambiguously, and all Protobuf runtimes provide direct conversion to Date, datetime, time.Time, etc.
• Prefix enum values with the enum name: ORDER_STATUS_PENDING not just PENDING. Proto3 enum values are scoped to the package, not the enum — two enums in the same package can't both have a PENDING value. The prefix convention prevents collisions and satisfies Buf's ENUM_VALUE_PREFIX lint rule.

FAQ

Q: When should I use proto3 optional vs omitting it?
A: Without optional, a proto3 field with its default value (0, empty string, false) is indistinguishable from an unset field — both serialize to nothing on the wire. With optional, the field has explicit presence tracking via a has_* getter. Use optional when you need to distinguish "not set" from "set to default value" — common for nullable database columns, patch/update operations, and configuration settings where 0 is a valid meaningful value.

Q: Protobuf vs Avro for Kafka — which is better?
A: Both work well. Protobuf has better language support (20+ generated clients), easier schema evolution with field numbers, and is required for gRPC. Avro has simpler schema definition, native Confluent Schema Registry integration (Avro was designed for it), and slightly smaller payload sizes for nullable fields. If you're already using gRPC, use Protobuf for consistency. If you're Kafka-only, Avro is the Confluent-native choice.

Q: How do I handle polymorphic JSON (objects with different shapes based on a type field)?
A: Use oneof for mutually exclusive field variants: oneof payload { OrderCreatedPayload order_created = 1; PaymentProcessedPayload payment_processed = 2; }. The oneof ensures only one variant is set at a time, and generated code provides type-safe accessors. Alternatively, use google.protobuf.Any for fully dynamic messages, but this loses type safety.