google/flatbuffers

FlatBuffers chat (also on Discord: discord.gg/6qgKs3R)

MikkelFJ

@mikkelfj

You can do that, but if you only have simple structs without strings or arrays, and it is process local, you are probably better off just using structs. If you do have strings and arrays, you may consider keeping data in flatbuffers to avoid the extra copying and allocation of data, even if it is not a performance concern. For what it is worth, the C interface allows you to create flatbuffers that have structs as root which can be accessed directly on little endian systems. Other languages require tables as root which are more expensive to create, but they can contain structs. Since you only have a small number of messages, this is hardly a critical point.

Edit: not process local, but system local, meaning same endian, same type of compiler, word size etc.

jay logan

@jnexthop:matrix.org

[m]

@aardappel: thanks, i am constrained to use C. I am assuming object API means the flatbuffers are unpacked into a local struct to give access like foo->bar->bas ? or do i use the flatbuffer offset api (generated) to get to the fields?

MikkelFJ

@mikkelfj

The object API is for C++ only working much like a DOM I suppose. Structs are directly accessible in the C Flatbuffer interface, but they cannot contain pointers like strings or arrays. You normally use functions to access struct fields because they provide endian translation when needed, and next to no overhead otherwise. You can access the structs natively in C on little endian systems.

If the data is simple enough to be contained in a C struct without pointers, you don’t really need FlatBuffers for your purposes, and otherwise you also need tables which requires accessor functions to navigate. You would not want to copy data out of FlatBuffers, that is not normally worthwhile. It is simpler to access the buffers directly with accessor funcitons. But depends.

To be clear: FlatBuffers has a concept of structs that is very similar to C structs, but C structs can contain pointers while FlatBuffer structs cannot. FlatBuffer tables can have offsets that resemble pointers.

jay logan

@jnexthop:matrix.org

[m]

@mikkelfj: thanks. The data that is exchanged is essentially 1) simple control messages (bring up something, bring down something etc). There are some minor substructs here. 2) network PDUs that will bypass flatbuffers since they are already in wire format and version controlled (ietf). I do need the flexibility of a schema since we anticipate the daemons involved to be upgraded independently. i essentially have a union of messages (tables) each with some minor structs, vectors (of ip addresses). I was earlier wondering if flatbuffers is an overkill for the application.

my current plan is to use flatbuffer schema to define and exchange these messages. use flatbuffer offset api to get at the fields of the received messages, create local collections of various objects. The application cannot work directly off of flatbuffer offset traversed fields anyway.

It also keeps the application code loosely coupled with the encoding/decoding details

MikkelFJ

@mikkelfj

Sounds like you have reasonable use case for FlatBuffers to gain flexibility, but you will also have a lot of work duplicating data at app level. I personally use FlatBuffers a lot for MQTT messaging between systems.

I also use FlatBuffers for the sake of having a schema where JSON would otherwise suffice.

Wouter van Oortmerssen

@aardappel

https://github.com/google/flatbuffers/wiki/Why-FlatBuffers-vs-other-options%3F

jay logan

@jnexthop:matrix.org

[m]

@aardappel: thanks. i think the C api and the difficulty in figuring out the functions to use is the only drawback i am seeing so far. (yes i did use gcc -E equivalent to check out the actual functions the preprocessor generated). It is more of a language issue, i guess.

Wouter van Oortmerssen

@aardappel

API design definitely favors speed over convenience when the two are at odds, which is part of the value of FlatBuffers

which is also why C++ has the object API.. it is the optional "convenience over speed" layer on top of the base API :)

that wouldn't be so easy in C as it doesn't have the concept of standard resizable containers like string/vector

jay logan

@jnexthop:matrix.org

[m]

Another option i have is to use C++ for the serdes functions and use C from outside. So i am good at this point. thx

=Bill.Barnhill

@BillBarnhill

Good morning everyone. I have an immediate pressing problem I hope you can help with. I am attempting to use Flatbuffers over JeroMQ in a Java 8 only environment. I have code that works in a Java 9+ environment, but in Java 8 I get the exception below. Does anyone know how to fix this (other than obvious use Java 9 or 10)? For example, is there a flatc setting to generate Java 8 compatible classes?

Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at com.google.flatbuffers.FlatBufferBuilder.createString(FlatBufferBuilder.java:605)
at jeromq_and_flatbuffers.io.MonsterIO.writeTo(MonsterIO.java:54)
at jeromq_and_flatbuffers.io.MonsterIO.writeTo(MonsterIO.java:1)
at jeromq_and_flatbuffers.core.MonsterPublisher.sendMessage(MonsterPublisher.java:80)
at jeromq_and_flatbuffers.core.MonsterPublisher.run(MonsterPublisher.java:104)
at jeromq_and_flatbuffers.core.Main.lambda$0(Main.java:46)
at java.util.Optional.ifPresent(Optional.java:159)
at jeromq_and_flatbuffers.core.Main.main(Main.java:46)

I realize there were a number of NIO improvements in Java 9, but my network IT has constrained us to Java 8,despite all reasoning.

=Bill.Barnhill

@BillBarnhill

Discovered that it's not generated classes, but a FlatBuffers java class, this is the offending line: ((Buffer) bb).position(space -= length);

Wouter van Oortmerssen

@aardappel

@BillBarnhill I believe we fixed that in latest master

which version are you using? are you getting this thru maven?

make sure you are on 2.0.2 https://mvnrepository.com/artifact/com.google.flatbuffers/flatbuffers-java

=Bill.Barnhill

@BillBarnhill

You did, the buffer cast fixes it. I discovered my problem was that the Maven Central flatbuffers jar was built with Java 9 or later. I downloaded the Flatbuffers source for 2.0.2, built the jar with maven package, and used that jar rather than the one from Central Repo. That got me past my error. Thank you for your help.

Wouter van Oortmerssen

@aardappel

PRs where this was addressed: google/flatbuffers#6658 google/flatbuffers#6671

=Bill.Barnhill

@BillBarnhill

Thanks

Wouter van Oortmerssen

@aardappel

so 2.0.2 from maven still doesn't work with Java 8?

can you talk to @krojew on those PRs about it?

=Bill.Barnhill

@BillBarnhill

I'm not being clear. It works with Java 8, if Flatbuffers jar is built with Java 8. If Flatbuffers jar is built with 9 or later, it does not seem to work with Java 8

Wouter van Oortmerssen

@aardappel

so Java 8 users can't use 2.0.2 from maven?

=Bill.Barnhill

@BillBarnhill

That's the case for me, yes

OS is Mac, Catalina, Java is OpenJDK v 52.0 (Java 8).

Correction...my Ivy deps were set to 2.0.0 from Maven central

Wouter van Oortmerssen

@aardappel

ok, then that was the actual problem

batwicket

@batwicket

hi all..

I would like to efficiently store a bag of c++ objects but am uncertain how to efficiently store it - question posted on stackoverflow: https://stackoverflow.com/questions/68399757/what-is-a-good-way-to-efficiently-store-a-list-of-c-objects-from-different-cla

will update question with whatever i can discover..

batwicket

@batwicket

received an answer on Discord.. will update stackoverflow..

jdees

@jdees

Hi - I have a usage questions:
I have a flatbuffers structure representing a tree (SQL Plan).

I would like to add some methods to the generated classes, e.g.:
table Join { size:int; }
would like to add a method:
bool Join::isLargerThan100(); // returns true if size>100

I am not sure if my request makes sense or how to implement that.
Is there some support from flatbuffers itself or some recommended way of doing this?
Is it allowed to derive from class Join (or rather use composition) with my custom class and add functions and add a special desieralizetion pass of the flatbuffers buffer?

jdees

@jdees

also a motivation: I want to write a visitor, which can iterative over all serialized objects (bottom-up/top-down/...) - here it would also be great to be able to inherit from the actual autogenerated classes - what do you suggest here?

(in order to have a virtual "accept()/visit()" method)

MikkelFJ

@mikkelfj

It will be difficult to add new methods to generated classes, but you can traverse the binary flatbuffers schema and generate new classes and inline functions to add new functionality.

=Bill.Barnhill

@BillBarnhill

Hi all. I am using Flatbuffers JAVA 2.0.2 and I've run into a problem where I need to generate C++ from fbs schema files that are compatible with the 2.0.2 JAVA version, but the latest C++ source code release (or tag in git) is 2.0.0. Is there a C++ 2.0.2 version?

=Bill.Barnhill

@BillBarnhill

Ok, I may have answered my own question. I built master branch and ./flatc --version reports 2.0.0. I take it Java version and flatc are different then, and 2.0.0 should be fine?

=Bill.Barnhill

@BillBarnhill

I also tried to find documentation on the various build commands to use, specifically how to enabled generating a dynamic lib. Wound up delving into CMakeFile and finding option(BUILD_SHARED_LIB), build with the following commands that seemed to do the trick. Please let me know if there's a better approach. 1) mkdir build && cd build 2) cmake3 .. -G "Unix Makefiles" -DFLATBUFFERS_BUILD_SHAREDLIB=ON -DCMAKE_BUILD_TYPE=Release 3) make

Wouter van Oortmerssen

@aardappel

@jdees we have no mechanism to add custom methods to the generated code.. typically you don't want to mix handwritten and generated code, and your functionality would likely be better off in an external function

the only way to add generic new generated code features is to add them to our code generator

@BillBarnhill C++ 2.0.0 is compatible with Java 2.0.2.. Java needed emergency fixes, C++ did not

generally, ALL version of ALL languages are compatible with eachother when it comes to exchanging binary FlatBuffer files, they are always forwards/backwards compatible

Max Burke

@maxburke

one of the things I was wondering is that with the deprecated attribute it prevents the generation of both accessors and builders; is there anything like a write_deprecated attribute to prevent only the generation of builders? in other words, so that data structures created previous to the deprecation can still have the field read but not created for new instances?

Where communities thrive

People

Repo info

Activity