Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 22:45
    eolivelli commented #4914
  • Jan 31 2019 22:07
    samsartor starred google/flatbuffers
  • Jan 31 2019 21:28
    marang starred google/flatbuffers
  • Jan 31 2019 20:51
    thyrlian starred google/flatbuffers
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 18:56
    aardappel commented #4914
  • Jan 31 2019 18:54
    aardappel commented #5144
  • Jan 31 2019 18:51
    aardappel commented #5141
  • Jan 31 2019 18:51
    aardappel commented #5145
  • Jan 31 2019 18:51
    krojew commented #5142
  • Jan 31 2019 18:49
    krojew commented #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:47
    gabyx commented #5142
  • Jan 31 2019 18:47
    aardappel commented #5002
  • Jan 31 2019 18:43
    gabyx commented #5142
  • Jan 31 2019 18:43
    krojew commented #5142
  • Jan 31 2019 18:43
    aardappel commented #5143
  • Jan 31 2019 18:42
    gabyx commented #5142
kofu145
@kofu145
image.png
Heya, was looking up how I could possibly implement an ecs model into flatbuffers for easy serialization, and I came across this:
if I have tons of components though, apparently vectors of unions are going to be inefficient? If so, would anyone know of a better way to do such a thing?
kofu
@kofu:matrix.org
[m]
(sorry I'll use this account from now on, just found out that you could access gitter rooms from element)
MikkelFJ
@mikkelfj
It depends. If you have a ton of union elements and few vectors to contain them, then it isn't bad: A union vector is really two vectors one of which is just a byte array of types. The other vector is similar to a vector of tables. If you store structs in unions these will not be stored inline in the union and thus come with overhead.
If you need more advice, you need to be more specific. I have a rough idea of what ECS is, but no idea how you intend to do it in flatbuffers.
MikkelFJ
@mikkelfj
well, I found your screenshot which was very tiny, but still it doesn't tell if you will have one vector per each object, or only one vector to control lots of objects, or a vector per property of which there can be many. Either way, as long as you don't have a lot of a very small vectors, you should be OK. Even good, since scanning a byte array for a specific object type will be efficient. Keep in mind that flatbuffers are read only.
tsindhuja
@tsindhuja
I'm trying to use flatbuffers in C++ with Object API. I am still unable to change the values in the tables that are given as a reference to another table. Is it possible to manipulate values of a table indirectly through another table in Object API?
Wouter van Oortmerssen
@aardappel
@tsindhuja yes that is possible.. what error are you getting?
Martin Hans
@martinhansdk
I'd like to use flatbuffers on an embedded system written in C++. We don't allow dynamic memory allocation here. Is it possible to have a pre-allocated buffer and hand it to flatbuffers to serialize the data into? What about when deserializing?
MikkelFJ
@mikkelfj
Reading FlatBuffers requires no allocation in C/C++ but might in Java etc. due to string allocation. As building buffers I can't answer for C++, but I can give some guidance on FlatBuffers for C. C requires some dynamic allocation but it can be limited in size if your buffers are simple. The allocator can be overriden so you can place allocations in a reserved area. There might be reallocation requests to grow areas, but that will not happen if your buffers are reasonable. It requires experimentation to decide the exact requirements for the problem at hand.
C generated buffer interfaces can be called from C++
Wouter van Oortmerssen
@aardappel
@martinhansdk yes you can specify an allocator for the C++ builder which can then supply existing memory. It can do all its building in that one buffer and will not allocate any further memory
tsindhuja
@tsindhuja
@aardappel I was unable to modify the table as the classes in it were generated as unique_ptr template. I am now able to access it after changing the schema file to modify it to a generic template.
Wouter van Oortmerssen
@aardappel
@tsindhuja again, what error are you getting? you should be able to modify unique_ptrs
tsindhuja
@tsindhuja
@aardappel the error was in terms of "Trying to access deleted function" when I was trying to use mutable_node()->end()
Wouter van Oortmerssen
@aardappel
and node is a table type? and what is end ?
tsindhuja
@tsindhuja
Yes, node is a table type and 'end' is a function that has to iterate to the end
The main issue was to iterate through the collection of tables that is a pointer in another table. And also every assignment operation had to be done via the move() function
jay logan
@jnexthop:matrix.org
[m]
I am exploring flatbuffers use in an embedded system between two unix processes. The messages are relatively small (200 bytes or so), flat non hierarchical, and all of the fields will be used to store some state and process. The messages themselves are about few hundred/minute. It is highly likely the received messages will be again copied into a local struct. Is this an appropriate use of flatbuffers?
Wouter van Oortmerssen
@aardappel
@jnexthop:matrix.org well certainly a "few hundred per minute" is so little any serializer would work
but yes, that sounds fine, you could use the object API instead of local structs
MikkelFJ
@mikkelfj
You can do that, but if you only have simple structs without strings or arrays, and it is process local, you are probably better off just using structs. If you do have strings and arrays, you may consider keeping data in flatbuffers to avoid the extra copying and allocation of data, even if it is not a performance concern. For what it is worth, the C interface allows you to create flatbuffers that have structs as root which can be accessed directly on little endian systems. Other languages require tables as root which are more expensive to create, but they can contain structs. Since you only have a small number of messages, this is hardly a critical point.
Edit: not process local, but system local, meaning same endian, same type of compiler, word size etc.
jay logan
@jnexthop:matrix.org
[m]
@aardappel: thanks, i am constrained to use C. I am assuming object API means the flatbuffers are unpacked into a local struct to give access like foo->bar->bas ? or do i use the flatbuffer offset api (generated) to get to the fields?
MikkelFJ
@mikkelfj
The object API is for C++ only working much like a DOM I suppose. Structs are directly accessible in the C Flatbuffer interface, but they cannot contain pointers like strings or arrays. You normally use functions to access struct fields because they provide endian translation when needed, and next to no overhead otherwise. You can access the structs natively in C on little endian systems.
If the data is simple enough to be contained in a C struct without pointers, you don’t really need FlatBuffers for your purposes, and otherwise you also need tables which requires accessor functions to navigate. You would not want to copy data out of FlatBuffers, that is not normally worthwhile. It is simpler to access the buffers directly with accessor funcitons. But depends.
To be clear: FlatBuffers has a concept of structs that is very similar to C structs, but C structs can contain pointers while FlatBuffer structs cannot. FlatBuffer tables can have offsets that resemble pointers.
jay logan
@jnexthop:matrix.org
[m]
@mikkelfj: thanks. The data that is exchanged is essentially 1) simple control messages (bring up something, bring down something etc). There are some minor substructs here. 2) network PDUs that will bypass flatbuffers since they are already in wire format and version controlled (ietf). I do need the flexibility of a schema since we anticipate the daemons involved to be upgraded independently. i essentially have a union of messages (tables) each with some minor structs, vectors (of ip addresses). I was earlier wondering if flatbuffers is an overkill for the application.
my current plan is to use flatbuffer schema to define and exchange these messages. use flatbuffer offset api to get at the fields of the received messages, create local collections of various objects. The application cannot work directly off of flatbuffer offset traversed fields anyway.
It also keeps the application code loosely coupled with the encoding/decoding details
MikkelFJ
@mikkelfj
Sounds like you have reasonable use case for FlatBuffers to gain flexibility, but you will also have a lot of work duplicating data at app level. I personally use FlatBuffers a lot for MQTT messaging between systems.
I also use FlatBuffers for the sake of having a schema where JSON would otherwise suffice.
jay logan
@jnexthop:matrix.org
[m]
@aardappel: thanks. i think the C api and the difficulty in figuring out the functions to use is the only drawback i am seeing so far. (yes i did use gcc -E equivalent to check out the actual functions the preprocessor generated). It is more of a language issue, i guess.
Wouter van Oortmerssen
@aardappel
API design definitely favors speed over convenience when the two are at odds, which is part of the value of FlatBuffers
which is also why C++ has the object API.. it is the optional "convenience over speed" layer on top of the base API :)
that wouldn't be so easy in C as it doesn't have the concept of standard resizable containers like string/vector
jay logan
@jnexthop:matrix.org
[m]
Another option i have is to use C++ for the serdes functions and use C from outside. So i am good at this point. thx
=Bill.Barnhill
@BillBarnhill

Good morning everyone. I have an immediate pressing problem I hope you can help with. I am attempting to use Flatbuffers over JeroMQ in a Java 8 only environment. I have code that works in a Java 9+ environment, but in Java 8 I get the exception below. Does anyone know how to fix this (other than obvious use Java 9 or 10)? For example, is there a flatc setting to generate Java 8 compatible classes?

Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at com.google.flatbuffers.FlatBufferBuilder.createString(FlatBufferBuilder.java:605)
at jeromq_and_flatbuffers.io.MonsterIO.writeTo(MonsterIO.java:54)
at jeromq_and_flatbuffers.io.MonsterIO.writeTo(MonsterIO.java:1)
at jeromq_and_flatbuffers.core.MonsterPublisher.sendMessage(MonsterPublisher.java:80)
at jeromq_and_flatbuffers.core.MonsterPublisher.run(MonsterPublisher.java:104)
at jeromq_and_flatbuffers.core.Main.lambda$0(Main.java:46)
at java.util.Optional.ifPresent(Optional.java:159)
at jeromq_and_flatbuffers.core.Main.main(Main.java:46)

I realize there were a number of NIO improvements in Java 9, but my network IT has constrained us to Java 8,despite all reasoning.
=Bill.Barnhill
@BillBarnhill
Discovered that it's not generated classes, but a FlatBuffers java class, this is the offending line: ((Buffer) bb).position(space -= length);
Wouter van Oortmerssen
@aardappel
@BillBarnhill I believe we fixed that in latest master
which version are you using? are you getting this thru maven?
=Bill.Barnhill
@BillBarnhill
You did, the buffer cast fixes it. I discovered my problem was that the Maven Central flatbuffers jar was built with Java 9 or later. I downloaded the Flatbuffers source for 2.0.2, built the jar with maven package, and used that jar rather than the one from Central Repo. That got me past my error. Thank you for your help.
Wouter van Oortmerssen
@aardappel
PRs where this was addressed: google/flatbuffers#6658 google/flatbuffers#6671
=Bill.Barnhill
@BillBarnhill
Thanks
Wouter van Oortmerssen
@aardappel
so 2.0.2 from maven still doesn't work with Java 8?
can you talk to @krojew on those PRs about it?