Java & Golang: Best way for dealing with binary data?

[meyerdan]

In our messages we are definiting a fields which contian binary data. For this we are using the varDataEncoding composite defined as follows.

    <composite name="varDataEncoding">
      <type name="length" primitiveType="uint16" />
      <type name="varData" primitiveType="uint8" length="0" characterEncoding="UTF-8" />
    </composite>

My questions go to the "characterEncoding" attribute:

• When we use this with the Java Code generator we get methods generated which allow us to access the data both as a String but also as a binary buffer / array ... This is fine, we do not need the String representation so we simply do not call that method.

• In the Golang generator, however, when enabling the range check, the decoder throws an error, complaining that the bytes we put into the field are not utf8 valid.

We need some guidance on how to deal with this:

• should we define the protocol differently if we have binary content?
• or, is this a bug in the golang generator?

[mjpt777]

I find it best to separate strings from binary data doing something like the following as appropriate.

    <composite name="varDataEncoding">
      <type name="length" primitiveType="uint32" maxValue="1073741824"/>
      <type name="varData" primitiveType="uint8" length="0"/>
    </composite>
    <composite name="varAsciiEncoding">
      <type name="length" primitiveType="uint32" maxValue="1073741824"/>
      <type name="varData" primitiveType="uint8" length="0" characterEncoding="US-ASCII"/>
    </composite>
    <composite name="varStringEncoding">
      <type name="length" primitiveType="uint32" maxValue="1073741824"/>
      <type name="varData" primitiveType="uint8" length="0" characterEncoding="UTF-8"/>
    </composite>

[billsegall]

Dan, It is entirely possible that the utf8 validator has a bug so if you think the data you are passing is valid utf8 then could you send me a sample so I can fix it. Alternately, as Martin suggests, you shouldn't specify a character encoding that isn't valid for your data and it can be omitted. Bill.

…

On Thu, Aug 3, 2017 at 9:02 PM, Martin Thompson ***@***.***> wrote: I find it best to separate strings from binary data doing something like the following as appropriate. <composite name="varDataEncoding"> <type name="length" primitiveType="uint32" maxValue="1073741824"/> <type name="varData" primitiveType="uint8" length="0"/> </composite> <composite name="varAsciiEncoding"> <type name="length" primitiveType="uint32" maxValue="1073741824"/> <type name="varData" primitiveType="uint8" length="0" characterEncoding="US-ASCII"/> </composite> <composite name="varStringEncoding"> <type name="length" primitiveType="uint32" maxValue="1073741824"/> <type name="varData" primitiveType="uint8" length="0" characterEncoding="UTF-8"/> </composite> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#488 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFI1pGrmpOx0Ygs0lBR1j2dvEOFv3gifks5sUaisgaJpZM4OsFSR> .

[meyerdan]

Hi Bill and Martin,

thanks for your answers! Defining a separate composite for string data and binary data makes a lot of sense and this is the kind of guidance I was hoping to get.

When I define the composite without the characterEncoding as suggested by Martin, the I get the following exception in the golang generator:

java.lang.NullPointerException
    at uk.co.real_logic.sbe.generation.golang.GolangGenerator.generateCharacterEncodingRangeCheck(GolangGenerator.java:213)
    at uk.co.real_logic.sbe.generation.golang.GolangGenerator.generateVarDataEncodeDecode(GolangGenerator.java:1137)
    at uk.co.real_logic.sbe.generation.golang.GolangGenerator.generateEncodeDecode(GolangGenerator.java:766)
    at uk.co.real_logic.sbe.generation.golang.GolangGenerator.generateMessageCode(GolangGenerator.java:2042)
    at uk.co.real_logic.sbe.generation.golang.GolangGenerator.generate(GolangGenerator.java:154)
    at uk.co.real_logic.sbe.SbeTool.generate(SbeTool.java:292)
    at uk.co.real_logic.sbe.SbeTool.main(SbeTool.java:217)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
    at java.lang.Thread.run(Thread.java:748)

this is using the latest snapshot 1.7.1-SNAPSHOT of the sbe-tool.

[billsegall]

This is a bug and I'll work on a fix.

Bill.

[billsegall]

#491 is a workaround.

There should be some subsequent enhancements to the range checking but that's probably out of scope of this issue,

[meyerdan]

Thanks for the super fast fix. We'll try it out.