Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 22:45
    eolivelli commented #4914
  • Jan 31 2019 22:07
    samsartor starred google/flatbuffers
  • Jan 31 2019 21:28
    marang starred google/flatbuffers
  • Jan 31 2019 20:51
    thyrlian starred google/flatbuffers
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 18:56
    aardappel commented #4914
  • Jan 31 2019 18:54
    aardappel commented #5144
  • Jan 31 2019 18:51
    aardappel commented #5141
  • Jan 31 2019 18:51
    aardappel commented #5145
  • Jan 31 2019 18:51
    krojew commented #5142
  • Jan 31 2019 18:49
    krojew commented #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:47
    gabyx commented #5142
  • Jan 31 2019 18:47
    aardappel commented #5002
  • Jan 31 2019 18:43
    gabyx commented #5142
  • Jan 31 2019 18:43
    krojew commented #5142
  • Jan 31 2019 18:43
    aardappel commented #5143
  • Jan 31 2019 18:42
    gabyx commented #5142
tsindhuja
@tsindhuja
Yes, node is a table type and 'end' is a function that has to iterate to the end
The main issue was to iterate through the collection of tables that is a pointer in another table. And also every assignment operation had to be done via the move() function
jay logan
@jnexthop:matrix.org
[m]
I am exploring flatbuffers use in an embedded system between two unix processes. The messages are relatively small (200 bytes or so), flat non hierarchical, and all of the fields will be used to store some state and process. The messages themselves are about few hundred/minute. It is highly likely the received messages will be again copied into a local struct. Is this an appropriate use of flatbuffers?
Wouter van Oortmerssen
@aardappel
@jnexthop:matrix.org well certainly a "few hundred per minute" is so little any serializer would work
but yes, that sounds fine, you could use the object API instead of local structs
MikkelFJ
@mikkelfj
You can do that, but if you only have simple structs without strings or arrays, and it is process local, you are probably better off just using structs. If you do have strings and arrays, you may consider keeping data in flatbuffers to avoid the extra copying and allocation of data, even if it is not a performance concern. For what it is worth, the C interface allows you to create flatbuffers that have structs as root which can be accessed directly on little endian systems. Other languages require tables as root which are more expensive to create, but they can contain structs. Since you only have a small number of messages, this is hardly a critical point.
Edit: not process local, but system local, meaning same endian, same type of compiler, word size etc.
jay logan
@jnexthop:matrix.org
[m]
@aardappel: thanks, i am constrained to use C. I am assuming object API means the flatbuffers are unpacked into a local struct to give access like foo->bar->bas ? or do i use the flatbuffer offset api (generated) to get to the fields?
MikkelFJ
@mikkelfj
The object API is for C++ only working much like a DOM I suppose. Structs are directly accessible in the C Flatbuffer interface, but they cannot contain pointers like strings or arrays. You normally use functions to access struct fields because they provide endian translation when needed, and next to no overhead otherwise. You can access the structs natively in C on little endian systems.
If the data is simple enough to be contained in a C struct without pointers, you don’t really need FlatBuffers for your purposes, and otherwise you also need tables which requires accessor functions to navigate. You would not want to copy data out of FlatBuffers, that is not normally worthwhile. It is simpler to access the buffers directly with accessor funcitons. But depends.
To be clear: FlatBuffers has a concept of structs that is very similar to C structs, but C structs can contain pointers while FlatBuffer structs cannot. FlatBuffer tables can have offsets that resemble pointers.
jay logan
@jnexthop:matrix.org
[m]
@mikkelfj: thanks. The data that is exchanged is essentially 1) simple control messages (bring up something, bring down something etc). There are some minor substructs here. 2) network PDUs that will bypass flatbuffers since they are already in wire format and version controlled (ietf). I do need the flexibility of a schema since we anticipate the daemons involved to be upgraded independently. i essentially have a union of messages (tables) each with some minor structs, vectors (of ip addresses). I was earlier wondering if flatbuffers is an overkill for the application.
my current plan is to use flatbuffer schema to define and exchange these messages. use flatbuffer offset api to get at the fields of the received messages, create local collections of various objects. The application cannot work directly off of flatbuffer offset traversed fields anyway.
It also keeps the application code loosely coupled with the encoding/decoding details
MikkelFJ
@mikkelfj
Sounds like you have reasonable use case for FlatBuffers to gain flexibility, but you will also have a lot of work duplicating data at app level. I personally use FlatBuffers a lot for MQTT messaging between systems.
I also use FlatBuffers for the sake of having a schema where JSON would otherwise suffice.
jay logan
@jnexthop:matrix.org
[m]
@aardappel: thanks. i think the C api and the difficulty in figuring out the functions to use is the only drawback i am seeing so far. (yes i did use gcc -E equivalent to check out the actual functions the preprocessor generated). It is more of a language issue, i guess.
Wouter van Oortmerssen
@aardappel
API design definitely favors speed over convenience when the two are at odds, which is part of the value of FlatBuffers
which is also why C++ has the object API.. it is the optional "convenience over speed" layer on top of the base API :)
that wouldn't be so easy in C as it doesn't have the concept of standard resizable containers like string/vector
jay logan
@jnexthop:matrix.org
[m]
Another option i have is to use C++ for the serdes functions and use C from outside. So i am good at this point. thx
=Bill.Barnhill
@BillBarnhill

Good morning everyone. I have an immediate pressing problem I hope you can help with. I am attempting to use Flatbuffers over JeroMQ in a Java 8 only environment. I have code that works in a Java 9+ environment, but in Java 8 I get the exception below. Does anyone know how to fix this (other than obvious use Java 9 or 10)? For example, is there a flatc setting to generate Java 8 compatible classes?

Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at com.google.flatbuffers.FlatBufferBuilder.createString(FlatBufferBuilder.java:605)
at jeromq_and_flatbuffers.io.MonsterIO.writeTo(MonsterIO.java:54)
at jeromq_and_flatbuffers.io.MonsterIO.writeTo(MonsterIO.java:1)
at jeromq_and_flatbuffers.core.MonsterPublisher.sendMessage(MonsterPublisher.java:80)
at jeromq_and_flatbuffers.core.MonsterPublisher.run(MonsterPublisher.java:104)
at jeromq_and_flatbuffers.core.Main.lambda$0(Main.java:46)
at java.util.Optional.ifPresent(Optional.java:159)
at jeromq_and_flatbuffers.core.Main.main(Main.java:46)

I realize there were a number of NIO improvements in Java 9, but my network IT has constrained us to Java 8,despite all reasoning.
=Bill.Barnhill
@BillBarnhill
Discovered that it's not generated classes, but a FlatBuffers java class, this is the offending line: ((Buffer) bb).position(space -= length);
Wouter van Oortmerssen
@aardappel
@BillBarnhill I believe we fixed that in latest master
which version are you using? are you getting this thru maven?
=Bill.Barnhill
@BillBarnhill
You did, the buffer cast fixes it. I discovered my problem was that the Maven Central flatbuffers jar was built with Java 9 or later. I downloaded the Flatbuffers source for 2.0.2, built the jar with maven package, and used that jar rather than the one from Central Repo. That got me past my error. Thank you for your help.
Wouter van Oortmerssen
@aardappel
PRs where this was addressed: google/flatbuffers#6658 google/flatbuffers#6671
=Bill.Barnhill
@BillBarnhill
Thanks
Wouter van Oortmerssen
@aardappel
so 2.0.2 from maven still doesn't work with Java 8?
can you talk to @krojew on those PRs about it?
=Bill.Barnhill
@BillBarnhill
I'm not being clear. It works with Java 8, if Flatbuffers jar is built with Java 8. If Flatbuffers jar is built with 9 or later, it does not seem to work with Java 8
Wouter van Oortmerssen
@aardappel
so Java 8 users can't use 2.0.2 from maven?
=Bill.Barnhill
@BillBarnhill
That's the case for me, yes
OS is Mac, Catalina, Java is OpenJDK v 52.0 (Java 8).
Correction...my Ivy deps were set to 2.0.0 from Maven central
Wouter van Oortmerssen
@aardappel
ok, then that was the actual problem
batwicket
@batwicket
hi all..
I would like to efficiently store a bag of c++ objects but am uncertain how to efficiently store it - question posted on stackoverflow: https://stackoverflow.com/questions/68399757/what-is-a-good-way-to-efficiently-store-a-list-of-c-objects-from-different-cla
will update question with whatever i can discover..
batwicket
@batwicket
received an answer on Discord.. will update stackoverflow..
jdees
@jdees

Hi - I have a usage questions:
I have a flatbuffers structure representing a tree (SQL Plan).

I would like to add some methods to the generated classes, e.g.:
table Join { size:int; }
would like to add a method:
bool Join::isLargerThan100(); // returns true if size>100

I am not sure if my request makes sense or how to implement that.
Is there some support from flatbuffers itself or some recommended way of doing this?
Is it allowed to derive from class Join (or rather use composition) with my custom class and add functions and add a special desieralizetion pass of the flatbuffers buffer?

jdees
@jdees
also a motivation: I want to write a visitor, which can iterative over all serialized objects (bottom-up/top-down/...) - here it would also be great to be able to inherit from the actual autogenerated classes - what do you suggest here?
(in order to have a virtual "accept()/visit()" method)
MikkelFJ
@mikkelfj
It will be difficult to add new methods to generated classes, but you can traverse the binary flatbuffers schema and generate new classes and inline functions to add new functionality.
=Bill.Barnhill
@BillBarnhill
Hi all. I am using Flatbuffers JAVA 2.0.2 and I've run into a problem where I need to generate C++ from fbs schema files that are compatible with the 2.0.2 JAVA version, but the latest C++ source code release (or tag in git) is 2.0.0. Is there a C++ 2.0.2 version?
=Bill.Barnhill
@BillBarnhill
Ok, I may have answered my own question. I built master branch and ./flatc --version reports 2.0.0. I take it Java version and flatc are different then, and 2.0.0 should be fine?
=Bill.Barnhill
@BillBarnhill
I also tried to find documentation on the various build commands to use, specifically how to enabled generating a dynamic lib. Wound up delving into CMakeFile and finding option(BUILD_SHARED_LIB), build with the following commands that seemed to do the trick. Please let me know if there's a better approach. 1) mkdir build && cd build 2) cmake3 .. -G "Unix Makefiles" -DFLATBUFFERS_BUILD_SHAREDLIB=ON -DCMAKE_BUILD_TYPE=Release 3) make