Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 22:45
    eolivelli commented #4914
  • Jan 31 2019 22:07
    samsartor starred google/flatbuffers
  • Jan 31 2019 21:28
    marang starred google/flatbuffers
  • Jan 31 2019 20:51
    thyrlian starred google/flatbuffers
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 18:56
    aardappel commented #4914
  • Jan 31 2019 18:54
    aardappel commented #5144
  • Jan 31 2019 18:51
    aardappel commented #5141
  • Jan 31 2019 18:51
    aardappel commented #5145
  • Jan 31 2019 18:51
    krojew commented #5142
  • Jan 31 2019 18:49
    krojew commented #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:47
    gabyx commented #5142
  • Jan 31 2019 18:47
    aardappel commented #5002
  • Jan 31 2019 18:43
    gabyx commented #5142
  • Jan 31 2019 18:43
    krojew commented #5142
  • Jan 31 2019 18:43
    aardappel commented #5143
  • Jan 31 2019 18:42
    gabyx commented #5142
MikkelFJ
@mikkelfj
If the data is simple enough to be contained in a C struct without pointers, you don’t really need FlatBuffers for your purposes, and otherwise you also need tables which requires accessor functions to navigate. You would not want to copy data out of FlatBuffers, that is not normally worthwhile. It is simpler to access the buffers directly with accessor funcitons. But depends.
To be clear: FlatBuffers has a concept of structs that is very similar to C structs, but C structs can contain pointers while FlatBuffer structs cannot. FlatBuffer tables can have offsets that resemble pointers.
jay logan
@jnexthop:matrix.org
[m]
@mikkelfj: thanks. The data that is exchanged is essentially 1) simple control messages (bring up something, bring down something etc). There are some minor substructs here. 2) network PDUs that will bypass flatbuffers since they are already in wire format and version controlled (ietf). I do need the flexibility of a schema since we anticipate the daemons involved to be upgraded independently. i essentially have a union of messages (tables) each with some minor structs, vectors (of ip addresses). I was earlier wondering if flatbuffers is an overkill for the application.
my current plan is to use flatbuffer schema to define and exchange these messages. use flatbuffer offset api to get at the fields of the received messages, create local collections of various objects. The application cannot work directly off of flatbuffer offset traversed fields anyway.
It also keeps the application code loosely coupled with the encoding/decoding details
MikkelFJ
@mikkelfj
Sounds like you have reasonable use case for FlatBuffers to gain flexibility, but you will also have a lot of work duplicating data at app level. I personally use FlatBuffers a lot for MQTT messaging between systems.
I also use FlatBuffers for the sake of having a schema where JSON would otherwise suffice.
jay logan
@jnexthop:matrix.org
[m]
@aardappel: thanks. i think the C api and the difficulty in figuring out the functions to use is the only drawback i am seeing so far. (yes i did use gcc -E equivalent to check out the actual functions the preprocessor generated). It is more of a language issue, i guess.
Wouter van Oortmerssen
@aardappel
API design definitely favors speed over convenience when the two are at odds, which is part of the value of FlatBuffers
which is also why C++ has the object API.. it is the optional "convenience over speed" layer on top of the base API :)
that wouldn't be so easy in C as it doesn't have the concept of standard resizable containers like string/vector
jay logan
@jnexthop:matrix.org
[m]
Another option i have is to use C++ for the serdes functions and use C from outside. So i am good at this point. thx
=Bill.Barnhill
@BillBarnhill

Good morning everyone. I have an immediate pressing problem I hope you can help with. I am attempting to use Flatbuffers over JeroMQ in a Java 8 only environment. I have code that works in a Java 9+ environment, but in Java 8 I get the exception below. Does anyone know how to fix this (other than obvious use Java 9 or 10)? For example, is there a flatc setting to generate Java 8 compatible classes?

Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at com.google.flatbuffers.FlatBufferBuilder.createString(FlatBufferBuilder.java:605)
at jeromq_and_flatbuffers.io.MonsterIO.writeTo(MonsterIO.java:54)
at jeromq_and_flatbuffers.io.MonsterIO.writeTo(MonsterIO.java:1)
at jeromq_and_flatbuffers.core.MonsterPublisher.sendMessage(MonsterPublisher.java:80)
at jeromq_and_flatbuffers.core.MonsterPublisher.run(MonsterPublisher.java:104)
at jeromq_and_flatbuffers.core.Main.lambda$0(Main.java:46)
at java.util.Optional.ifPresent(Optional.java:159)
at jeromq_and_flatbuffers.core.Main.main(Main.java:46)

I realize there were a number of NIO improvements in Java 9, but my network IT has constrained us to Java 8,despite all reasoning.
=Bill.Barnhill
@BillBarnhill
Discovered that it's not generated classes, but a FlatBuffers java class, this is the offending line: ((Buffer) bb).position(space -= length);
Wouter van Oortmerssen
@aardappel
@BillBarnhill I believe we fixed that in latest master
which version are you using? are you getting this thru maven?
=Bill.Barnhill
@BillBarnhill
You did, the buffer cast fixes it. I discovered my problem was that the Maven Central flatbuffers jar was built with Java 9 or later. I downloaded the Flatbuffers source for 2.0.2, built the jar with maven package, and used that jar rather than the one from Central Repo. That got me past my error. Thank you for your help.
Wouter van Oortmerssen
@aardappel
PRs where this was addressed: google/flatbuffers#6658 google/flatbuffers#6671
=Bill.Barnhill
@BillBarnhill
Thanks
Wouter van Oortmerssen
@aardappel
so 2.0.2 from maven still doesn't work with Java 8?
can you talk to @krojew on those PRs about it?
=Bill.Barnhill
@BillBarnhill
I'm not being clear. It works with Java 8, if Flatbuffers jar is built with Java 8. If Flatbuffers jar is built with 9 or later, it does not seem to work with Java 8
Wouter van Oortmerssen
@aardappel
so Java 8 users can't use 2.0.2 from maven?
=Bill.Barnhill
@BillBarnhill
That's the case for me, yes
OS is Mac, Catalina, Java is OpenJDK v 52.0 (Java 8).
Correction...my Ivy deps were set to 2.0.0 from Maven central
Wouter van Oortmerssen
@aardappel
ok, then that was the actual problem
batwicket
@batwicket
hi all..
I would like to efficiently store a bag of c++ objects but am uncertain how to efficiently store it - question posted on stackoverflow: https://stackoverflow.com/questions/68399757/what-is-a-good-way-to-efficiently-store-a-list-of-c-objects-from-different-cla
will update question with whatever i can discover..
batwicket
@batwicket
received an answer on Discord.. will update stackoverflow..
jdees
@jdees

Hi - I have a usage questions:
I have a flatbuffers structure representing a tree (SQL Plan).

I would like to add some methods to the generated classes, e.g.:
table Join { size:int; }
would like to add a method:
bool Join::isLargerThan100(); // returns true if size>100

I am not sure if my request makes sense or how to implement that.
Is there some support from flatbuffers itself or some recommended way of doing this?
Is it allowed to derive from class Join (or rather use composition) with my custom class and add functions and add a special desieralizetion pass of the flatbuffers buffer?

jdees
@jdees
also a motivation: I want to write a visitor, which can iterative over all serialized objects (bottom-up/top-down/...) - here it would also be great to be able to inherit from the actual autogenerated classes - what do you suggest here?
(in order to have a virtual "accept()/visit()" method)
MikkelFJ
@mikkelfj
It will be difficult to add new methods to generated classes, but you can traverse the binary flatbuffers schema and generate new classes and inline functions to add new functionality.
=Bill.Barnhill
@BillBarnhill
Hi all. I am using Flatbuffers JAVA 2.0.2 and I've run into a problem where I need to generate C++ from fbs schema files that are compatible with the 2.0.2 JAVA version, but the latest C++ source code release (or tag in git) is 2.0.0. Is there a C++ 2.0.2 version?
=Bill.Barnhill
@BillBarnhill
Ok, I may have answered my own question. I built master branch and ./flatc --version reports 2.0.0. I take it Java version and flatc are different then, and 2.0.0 should be fine?
=Bill.Barnhill
@BillBarnhill
I also tried to find documentation on the various build commands to use, specifically how to enabled generating a dynamic lib. Wound up delving into CMakeFile and finding option(BUILD_SHARED_LIB), build with the following commands that seemed to do the trick. Please let me know if there's a better approach. 1) mkdir build && cd build 2) cmake3 .. -G "Unix Makefiles" -DFLATBUFFERS_BUILD_SHAREDLIB=ON -DCMAKE_BUILD_TYPE=Release 3) make
Wouter van Oortmerssen
@aardappel
@jdees we have no mechanism to add custom methods to the generated code.. typically you don't want to mix handwritten and generated code, and your functionality would likely be better off in an external function
the only way to add generic new generated code features is to add them to our code generator
@BillBarnhill C++ 2.0.0 is compatible with Java 2.0.2.. Java needed emergency fixes, C++ did not
generally, ALL version of ALL languages are compatible with eachother when it comes to exchanging binary FlatBuffer files, they are always forwards/backwards compatible
Max Burke
@maxburke
one of the things I was wondering is that with the deprecated attribute it prevents the generation of both accessors and builders; is there anything like a write_deprecated attribute to prevent only the generation of builders? in other words, so that data structures created previous to the deprecation can still have the field read but not created for new instances?
Wouter van Oortmerssen
@aardappel
@maxburke hah, I'd think you'd almost want it the other way around: if a field is deprecated you want to stop relying on it as soon as possible (read), but any writes are harmless (just take up space)
but no we don't have such options
if you want to gradually deprecate something best would be to first change the code before adding the attribute, I guess?
Max Burke
@maxburke

I guess it's a bit of a unique scenario in our case... the issue is we have an old type that has two possible fields (ie: explicitly named field1 and field2) , and we're replacing it with a new type that has n-possible fields (ie: a list). the old data is still valid as the new ones as on load we deserialize field1 and field2 into a 2-element list, but we want to not write the old type anymore.

the current solution is to name the old field "deprecated_old_field_do_not_use" which I think kind of gets the point across ( :) ), and I think I can live with, but I was thinking it would be good to make sure that no builders were available for it.