google/flatbuffers

FlatBuffers chat (also on Discord: discord.gg/6qgKs3R)

jay logan

@jnexthop:matrix.org

@mikkelfj: thanks. The data that is exchanged is essentially 1) simple control messages (bring up something, bring down something etc). There are some minor substructs here. 2) network PDUs that will bypass flatbuffers since they are already in wire format and version controlled (ietf). I do need the flexibility of a schema since we anticipate the daemons involved to be upgraded independently. i essentially have a union of messages (tables) each with some minor structs, vectors (of ip addresses). I was earlier wondering if flatbuffers is an overkill for the application.

my current plan is to use flatbuffer schema to define and exchange these messages. use flatbuffer offset api to get at the fields of the received messages, create local collections of various objects. The application cannot work directly off of flatbuffer offset traversed fields anyway.

It also keeps the application code loosely coupled with the encoding/decoding details

MikkelFJ

@mikkelfj

Sounds like you have reasonable use case for FlatBuffers to gain flexibility, but you will also have a lot of work duplicating data at app level. I personally use FlatBuffers a lot for MQTT messaging between systems.

I also use FlatBuffers for the sake of having a schema where JSON would otherwise suffice.

Wouter van Oortmerssen

@aardappel

https://github.com/google/flatbuffers/wiki/Why-FlatBuffers-vs-other-options%3F

jay logan

@jnexthop:matrix.org

[m]

@aardappel: thanks. i think the C api and the difficulty in figuring out the functions to use is the only drawback i am seeing so far. (yes i did use gcc -E equivalent to check out the actual functions the preprocessor generated). It is more of a language issue, i guess.

Wouter van Oortmerssen

@aardappel

API design definitely favors speed over convenience when the two are at odds, which is part of the value of FlatBuffers

which is also why C++ has the object API.. it is the optional "convenience over speed" layer on top of the base API :)

that wouldn't be so easy in C as it doesn't have the concept of standard resizable containers like string/vector

jay logan

@jnexthop:matrix.org

[m]

Another option i have is to use C++ for the serdes functions and use C from outside. So i am good at this point. thx

=Bill.Barnhill

@BillBarnhill

Good morning everyone. I have an immediate pressing problem I hope you can help with. I am attempting to use Flatbuffers over JeroMQ in a Java 8 only environment. I have code that works in a Java 9+ environment, but in Java 8 I get the exception below. Does anyone know how to fix this (other than obvious use Java 9 or 10)? For example, is there a flatc setting to generate Java 8 compatible classes?

Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at com.google.flatbuffers.FlatBufferBuilder.createString(FlatBufferBuilder.java:605)
at jeromq_and_flatbuffers.io.MonsterIO.writeTo(MonsterIO.java:54)
at jeromq_and_flatbuffers.io.MonsterIO.writeTo(MonsterIO.java:1)
at jeromq_and_flatbuffers.core.MonsterPublisher.sendMessage(MonsterPublisher.java:80)
at jeromq_and_flatbuffers.core.MonsterPublisher.run(MonsterPublisher.java:104)
at jeromq_and_flatbuffers.core.Main.lambda$0(Main.java:46)
at java.util.Optional.ifPresent(Optional.java:159)
at jeromq_and_flatbuffers.core.Main.main(Main.java:46)

I realize there were a number of NIO improvements in Java 9, but my network IT has constrained us to Java 8,despite all reasoning.

=Bill.Barnhill

@BillBarnhill

Discovered that it's not generated classes, but a FlatBuffers java class, this is the offending line: ((Buffer) bb).position(space -= length);

Wouter van Oortmerssen

@aardappel

@BillBarnhill I believe we fixed that in latest master

which version are you using? are you getting this thru maven?

make sure you are on 2.0.2 https://mvnrepository.com/artifact/com.google.flatbuffers/flatbuffers-java

=Bill.Barnhill

@BillBarnhill

You did, the buffer cast fixes it. I discovered my problem was that the Maven Central flatbuffers jar was built with Java 9 or later. I downloaded the Flatbuffers source for 2.0.2, built the jar with maven package, and used that jar rather than the one from Central Repo. That got me past my error. Thank you for your help.

Wouter van Oortmerssen

@aardappel

PRs where this was addressed: google/flatbuffers#6658 google/flatbuffers#6671

=Bill.Barnhill

@BillBarnhill

Thanks

Wouter van Oortmerssen

@aardappel

so 2.0.2 from maven still doesn't work with Java 8?

can you talk to @krojew on those PRs about it?

=Bill.Barnhill

@BillBarnhill

I'm not being clear. It works with Java 8, if Flatbuffers jar is built with Java 8. If Flatbuffers jar is built with 9 or later, it does not seem to work with Java 8

Wouter van Oortmerssen

@aardappel

so Java 8 users can't use 2.0.2 from maven?

=Bill.Barnhill

@BillBarnhill

That's the case for me, yes

OS is Mac, Catalina, Java is OpenJDK v 52.0 (Java 8).

Correction...my Ivy deps were set to 2.0.0 from Maven central

Wouter van Oortmerssen

@aardappel

ok, then that was the actual problem

batwicket

@batwicket

hi all..

I would like to efficiently store a bag of c++ objects but am uncertain how to efficiently store it - question posted on stackoverflow: https://stackoverflow.com/questions/68399757/what-is-a-good-way-to-efficiently-store-a-list-of-c-objects-from-different-cla

will update question with whatever i can discover..

batwicket

@batwicket

received an answer on Discord.. will update stackoverflow..

jdees

@jdees

Hi - I have a usage questions:
I have a flatbuffers structure representing a tree (SQL Plan).

I would like to add some methods to the generated classes, e.g.:
table Join { size:int; }
would like to add a method:
bool Join::isLargerThan100(); // returns true if size>100

I am not sure if my request makes sense or how to implement that.
Is there some support from flatbuffers itself or some recommended way of doing this?
Is it allowed to derive from class Join (or rather use composition) with my custom class and add functions and add a special desieralizetion pass of the flatbuffers buffer?

jdees

@jdees

also a motivation: I want to write a visitor, which can iterative over all serialized objects (bottom-up/top-down/...) - here it would also be great to be able to inherit from the actual autogenerated classes - what do you suggest here?

(in order to have a virtual "accept()/visit()" method)

MikkelFJ

@mikkelfj

It will be difficult to add new methods to generated classes, but you can traverse the binary flatbuffers schema and generate new classes and inline functions to add new functionality.

=Bill.Barnhill

@BillBarnhill

Hi all. I am using Flatbuffers JAVA 2.0.2 and I've run into a problem where I need to generate C++ from fbs schema files that are compatible with the 2.0.2 JAVA version, but the latest C++ source code release (or tag in git) is 2.0.0. Is there a C++ 2.0.2 version?

=Bill.Barnhill

@BillBarnhill

Ok, I may have answered my own question. I built master branch and ./flatc --version reports 2.0.0. I take it Java version and flatc are different then, and 2.0.0 should be fine?

=Bill.Barnhill

@BillBarnhill

I also tried to find documentation on the various build commands to use, specifically how to enabled generating a dynamic lib. Wound up delving into CMakeFile and finding option(BUILD_SHARED_LIB), build with the following commands that seemed to do the trick. Please let me know if there's a better approach. 1) mkdir build && cd build 2) cmake3 .. -G "Unix Makefiles" -DFLATBUFFERS_BUILD_SHAREDLIB=ON -DCMAKE_BUILD_TYPE=Release 3) make

Wouter van Oortmerssen

@aardappel

@jdees we have no mechanism to add custom methods to the generated code.. typically you don't want to mix handwritten and generated code, and your functionality would likely be better off in an external function

the only way to add generic new generated code features is to add them to our code generator

@BillBarnhill C++ 2.0.0 is compatible with Java 2.0.2.. Java needed emergency fixes, C++ did not

generally, ALL version of ALL languages are compatible with eachother when it comes to exchanging binary FlatBuffer files, they are always forwards/backwards compatible

Max Burke

@maxburke

one of the things I was wondering is that with the deprecated attribute it prevents the generation of both accessors and builders; is there anything like a write_deprecated attribute to prevent only the generation of builders? in other words, so that data structures created previous to the deprecation can still have the field read but not created for new instances?

Wouter van Oortmerssen

@aardappel

@maxburke hah, I'd think you'd almost want it the other way around: if a field is deprecated you want to stop relying on it as soon as possible (read), but any writes are harmless (just take up space)

but no we don't have such options

if you want to gradually deprecate something best would be to first change the code before adding the attribute, I guess?

Max Burke

@maxburke

I guess it's a bit of a unique scenario in our case... the issue is we have an old type that has two possible fields (ie: explicitly named field1 and field2) , and we're replacing it with a new type that has n-possible fields (ie: a list). the old data is still valid as the new ones as on load we deserialize field1 and field2 into a 2-element list, but we want to not write the old type anymore.

the current solution is to name the old field "deprecated_old_field_do_not_use" which I think kind of gets the point across ( :) ), and I think I can live with, but I was thinking it would be good to make sure that no builders were available for it.

Wouter van Oortmerssen

@aardappel

ah that makes sense yeah.. the renaming should make it easy to audit all cases though

Where communities thrive

People

Repo info

Activity