go - Decoding "one-of" protobuf messages - Stack Overflow

I am in the process of implementing a protobuf extension to work with our existing in-house protobuf framework. We already have this implemented for C, Python, and a couple of other languages across multiple services, so changing the message format is not an option. What I am trying to do is to find the most go-idiomatic (and least painful) way of doing so.

What we have done is created a small header that we prepend onto all of our protobuf messages, and one of those fields indicates the message type. Encoding messages in this framework is really easy:

const (
    MessageEmpty = iota
    Message1
    Message2
)

func Encode(msgType byte, msg proto.Message) ([]byte, error) {
    buf, err := proto.Marshal(msg)
    if err != nil {
        return nil, err
    }

    hdr := Header {
        msgtype: msgType,
    }

    out := new(bytes.Buffer)
    err = binary.Write(out, binary.BigEndian, hdr)
    if err != nil {
        return nil, err
    }
    
    out.Grow(len(buf))
    _, err = out.Write(buf)
    if err != nil {
        return nil, err
    }

    return out.Bytes(), nil
}

I haven't thoroughly tested this yet, but it appears correct at first glance. However, my issue I'm running across is trying to write the Decode() function. What I'm trying to do is something along the following:

func Decode(pkt []byte) (proto.Message, error) {
    hdr := Header{}
    buf := bytes.NewBuffer(pkt)
    ret := proto.Message()

    err := binary.Read(buf, binary.BigEndian, &hdr)
    if err != nil {
        return nil, err
    }
    
    msg := buf.Bytes()
    switch hdr.msgType {
    case Message1:
        pbmsg := myproto.FirstMessage{}
        err = proto.Unmarshal(msg, &pbmsg)
        ret = &pbmsg
    case Message2:
    }

    if err != nil {
        return nil, err
    }

    return ret, nil
}

However, this fails to compile. And the next problem I will have is once I have a message, how will I determine what the type of that message is at runtime? (I suppose I could also return the msgType field as one of my return values).

Any suggestions and/or guidance would be welcome here.

I am in the process of implementing a protobuf extension to work with our existing in-house protobuf framework. We already have this implemented for C, Python, and a couple of other languages across multiple services, so changing the message format is not an option. What I am trying to do is to find the most go-idiomatic (and least painful) way of doing so.

What we have done is created a small header that we prepend onto all of our protobuf messages, and one of those fields indicates the message type. Encoding messages in this framework is really easy:

const (
    MessageEmpty = iota
    Message1
    Message2
)

func Encode(msgType byte, msg proto.Message) ([]byte, error) {
    buf, err := proto.Marshal(msg)
    if err != nil {
        return nil, err
    }

    hdr := Header {
        msgtype: msgType,
    }

    out := new(bytes.Buffer)
    err = binary.Write(out, binary.BigEndian, hdr)
    if err != nil {
        return nil, err
    }
    
    out.Grow(len(buf))
    _, err = out.Write(buf)
    if err != nil {
        return nil, err
    }

    return out.Bytes(), nil
}

I haven't thoroughly tested this yet, but it appears correct at first glance. However, my issue I'm running across is trying to write the Decode() function. What I'm trying to do is something along the following:

func Decode(pkt []byte) (proto.Message, error) {
    hdr := Header{}
    buf := bytes.NewBuffer(pkt)
    ret := proto.Message()

    err := binary.Read(buf, binary.BigEndian, &hdr)
    if err != nil {
        return nil, err
    }
    
    msg := buf.Bytes()
    switch hdr.msgType {
    case Message1:
        pbmsg := myproto.FirstMessage{}
        err = proto.Unmarshal(msg, &pbmsg)
        ret = &pbmsg
    case Message2:
    }

    if err != nil {
        return nil, err
    }

    return ret, nil
}

However, this fails to compile. And the next problem I will have is once I have a message, how will I determine what the type of that message is at runtime? (I suppose I could also return the msgType field as one of my return values).

Any suggestions and/or guidance would be welcome here.

• go
• protocol-buffers

Share Improve this question Follow asked Nov 20, 2024 at 0:52 Ken PKen P 58033 silver badges1111 bronze badges 7

• 1 General question best practice: fails to compile with what error? – Ry- ♦ Commented Nov 20, 2024 at 1:31
• Your Decode function will work if you change ret := proto.Message() (which you can't as its a synonym for protoreflect.Message which is an interface) into var ret proto.Message – DazWilkin Commented Nov 20, 2024 at 4:46
• 1 Protocol Buffers Messages are strongly-typed and it becomes challenging to envelope them as your in-house solution does. I assume Go isn't the only language where this becomes challenging. There's a general-purpose Any message that's functionally similar and it may be useful for you to reference this in your solution. Additionally you can use protoreflect to dynamically interact with messages though this is gnarly. – DazWilkin Commented Nov 20, 2024 at 4:50
• 1 Your code has various areas of potential improvement but this is difficult to convey on Stack overflow and generally discouraged. Rather than Encode|Decode functions, you should consider defining a type for your wrapped messages and then use Encode|Decode methods. There are better approaches to enum handling too (ironically see Enum). You can simplify Decode's return by dropping the if err!=nil, if a returned value's err!=nil, the first value is generally considered unusable – DazWilkin Commented Nov 20, 2024 at 5:02
• Yes, you'll need to return the message type (cheap or reflect, more expensive) in order to select appropriate handling. – DazWilkin Commented Nov 20, 2024 at 5:04

 |  Show 2 more comments

1 Answer 1

Sorted by: Reset to default 1

You shouldn't probably not roll your own mechanism. For dynamic messages there is google/protobuf/any.proto which could be used like:

import (
    "google.golang./protobuf/proto"
    "google.golang./protobuf/types/known/anypb"
)

func Encode(msg proto.Message) ([]byte, error) {
    anyMsg, err := anypb.New(msg)
    if err != nil {
        return nil, err
    }

    return proto.Marshal(anyMsg)
}

func Decode(buf []byte) (proto.Message, error) {
    var anyMsg anypb.Any
    if err := proto.Unmarshal(buf, &anyMsg); err != nil {
        return nil, err
    }

    return anyMsg.UnmarshalNew()
}

The returned value you could be used with a switch:

    msg, err := Decode(buf)
    if err != nil {
        // ...
    }

    switch msg := msg.(type) {
    case *myproto.FirstMessage:
        // Here msg is of type *myproto.FirstMessage
        // ...
    }

which also answers your second question.

As an alternative, when you just need to consider a couple of messages, define a containing message with a OneOf:

message ContainerMessage {
  oneof container_oneof {
    FirstMessage first_message = 1;
  }
}

Assuming for compatibility reasons you need to support your format, the following will compile:

func Decode(pkt []byte) (proto.Message, error) {
    buf := bytes.NewBuffer(pkt)

    var hdr Header
    err := binary.Read(buf, binary.BigEndian, &hdr)
    if err != nil {
        return nil, err
    }

    msg := buf.Bytes()

    var ret proto.Message
    switch hdr.MsgType {
    case Message1:
        var pbmsg myproto.FirstMessage
        err = proto.Unmarshal(msg, &pbmsg)
        ret = &pbmsg
    case Message2:
        // ...
    }

    return ret, err
}