Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 22:45
    eolivelli commented #4914
  • Jan 31 2019 22:07
    samsartor starred google/flatbuffers
  • Jan 31 2019 21:28
    marang starred google/flatbuffers
  • Jan 31 2019 20:51
    thyrlian starred google/flatbuffers
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 18:56
    aardappel commented #4914
  • Jan 31 2019 18:54
    aardappel commented #5144
  • Jan 31 2019 18:51
    aardappel commented #5141
  • Jan 31 2019 18:51
    aardappel commented #5145
  • Jan 31 2019 18:51
    krojew commented #5142
  • Jan 31 2019 18:49
    krojew commented #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:47
    gabyx commented #5142
  • Jan 31 2019 18:47
    aardappel commented #5002
  • Jan 31 2019 18:43
    gabyx commented #5142
  • Jan 31 2019 18:43
    krojew commented #5142
  • Jan 31 2019 18:43
    aardappel commented #5143
  • Jan 31 2019 18:42
    gabyx commented #5142
MikkelFJ
@mikkelfj

So what if each table can specify its own serialization format as an annotation?

That might get a bit clumsy, and you might want to use multiple serialization formats, for example in a gateway.

Lijie.Jiang
@lijie-jiang
Would it be possible to include multiple input path with '-I' option? google/flatbuffers#6346
Sargun Dhillon
@sargun
I noticed flatbuffers doesn't have canonicalization:
RE: On purpose, the format leaves a lot of details about where exactly things live in memory undefined, e.g. fields in a table can have any order, and objects to some extent can be stored in many orders. This is because the format doesn't need this information to be efficient, and it leaves room for optimization and extension (for example, fields can be packed in a way that is most compact). Instead, the format is defined in terms of offsets and adjacency only. This may mean two different implementations may produce different binaries given the same input values, and this is perfectly valid.
Is there appetite for it?
MikkelFJ
@mikkelfj

Is there appetite for it?

What do you mean?
If implementations use this to pack better?
I'm not sure how much, but for example flatcc packs vtables at the end of the buffer by default.
And in general, data are placed as they arrive as much as possible on all implementations that I am aware of, which in itself helps performance by avoiding buffering.

Sargun Dhillon
@sargun
Is there a plan to add canonicalization to the specification, or similar?
Maxim Zaks
@mzaks
What would be the benefit of it, at this point in time?
I think if there would be a FlatBuffers 2.0 canonicalization might make sense. But pushing it on an established format without big benefits, is questionable.
It think it is ok to introduce guidelines, explaining that folowing form is more efficient and that people should follow it, if they can. But the format in current state is flexible and there is data already created in different way, so it needs to be supported anyways.
Sargun Dhillon
@sargun
Are structs themselves always guaranteed to be reproducible?
adsharma
@adsharma

Is there a plan to add canonicalization to the specification, or similar?

https://adsharma.github.io/flattools/ - pick a canonical serialization that works for you and implement it as a decorator in your favorite language, while enjoying the benefits of flatbuffer syntax as IDL.

Maxim Zaks
@mzaks

Are structs themselves always guaranteed to be reproducible?

Yes. They are rigid. You can not evolve them. You can specify some special layout properties through attributes though. But you can't change it after you have used it as it will be a breaking change.

@sargun ^
MikkelFJ
@mikkelfj
@sargun you can also print to JSON without spaces, that is probably as close as you can get. And yes structs are always the same - except potential flaws in exports where padding space is not zeroed - I just found a bug in a flatcc that failed to ensure that - because in some cases user code is allowed to influence that via a raw copy.
adsharma
@adsharma

One more blog post on flattools and where it fits in the stack:

https://adsharma.github.io/flattools-programs/

Happy New Year!

cyberquarks
@cyberquarks
Hi can this work with Flatbuffers?
message Entity {
  string dir = 1; 
  string entity_type = 2;
  string entity_id = 3;
  repeated EntityBlob blobs = 4;
  map<string, EntityProperty> properties = 5;
  map<string, EntityIdList> related_entities = 6;
}
message EntityProperty {
  oneof property_value {
    string string_value = 1;
    EntityArrayProperty array_value = 2;
    EntityObjectProperty object_value = 3;
    bool bool_value = 4;
    double double_value = 5;
  }
}
message EntityArrayProperty {
  repeated EntityProperty values = 1;
}
message EntityObjectProperty {
  map<string, EntityProperty> property_map = 1;
}
message EntityIdList {
  repeated EntityId ids = 1;
}
message EntityBlob {
  string blob_name = 1;
  bytes blob_bytes = 2;
}
message EntityId {
  string type = 1;
  string id = 2;
}
cyberquarks
@cyberquarks

I tried to translate this with flatc and I got this with the "Anonymous0" table:

// Generated from schema.proto

namespace ;

table Entity {
  dir:string;
  entity_type:string;
  entity_id:string;
  blobs:[EntityBlob];
  properties:[MapFieldEntry];
  links:[MultimapFieldEntry];
}

table EntityProperty {
  property_value:EntityProperty_.Anonymous0;
}

namespace EntityProperty_;

table Anonymous0 {
  string_value:string;
  array_value:EntityArrayProperty;
  object_value:EntityObjectProperty;
  bool_value:bool;
  double_value:double;
}

namespace ;

table EntityArrayProperty {
  values:[EntityProperty];
}

table EntityObjectProperty {
  properties:[MapFieldEntry];
}

table EntityIdList {
  ids:[EntityId];
}

table EntityBlob {
  blob_name:string;
  blob_bytes:[ubyte];
}

table EntityId {
  type:string;
  id:string;
}

table MapFieldEntry {
  key:string;
  value:EntityProperty;
}

table MultimapFieldEntry {
  key:string;
  value:EntityId;
}

What does this Anonymous0 mean?

Wouter van Oortmerssen
@aardappel
nice @adsharma
@cyberquarks Protobuf to FlatBuffers is not a 1:1 mapping, and FlatBuffers doesn't have the oneof construct.. you can just rename it to something else. And since its the only field in EntityProperty you can just replace it with EntityProperty directly. Or use a FlatBuffers union.
also MapFieldEntry should probably have a (key) attribute on the key field, so you can actually use it with dictionary lookup
vjani
@vjani
@here Had a question about tags for the flatbuffers repo, do they indicate the official releases? If so, there has been quite some time since the last one(March 2020) and I need to consume some of the later fixes, what is a good way to do this?
except compiling from master. I have tried that and it works however there is a danger of using official flatbuffer package from repository and using flatc from the master may generate code which may be incompatible with the classes in the official package. I ran today into similar issue where builder.EndVector signature changed and caused an incomptatibility
Wouter van Oortmerssen
@aardappel
I think I already answered you on discord..
kofu145
@kofu145
image.png
Heya, was looking up how I could possibly implement an ecs model into flatbuffers for easy serialization, and I came across this:
if I have tons of components though, apparently vectors of unions are going to be inefficient? If so, would anyone know of a better way to do such a thing?
kofu
@kofu:matrix.org
[m]
(sorry I'll use this account from now on, just found out that you could access gitter rooms from element)
MikkelFJ
@mikkelfj
It depends. If you have a ton of union elements and few vectors to contain them, then it isn't bad: A union vector is really two vectors one of which is just a byte array of types. The other vector is similar to a vector of tables. If you store structs in unions these will not be stored inline in the union and thus come with overhead.
If you need more advice, you need to be more specific. I have a rough idea of what ECS is, but no idea how you intend to do it in flatbuffers.
MikkelFJ
@mikkelfj
well, I found your screenshot which was very tiny, but still it doesn't tell if you will have one vector per each object, or only one vector to control lots of objects, or a vector per property of which there can be many. Either way, as long as you don't have a lot of a very small vectors, you should be OK. Even good, since scanning a byte array for a specific object type will be efficient. Keep in mind that flatbuffers are read only.
tsindhuja
@tsindhuja
I'm trying to use flatbuffers in C++ with Object API. I am still unable to change the values in the tables that are given as a reference to another table. Is it possible to manipulate values of a table indirectly through another table in Object API?
Wouter van Oortmerssen
@aardappel
@tsindhuja yes that is possible.. what error are you getting?
Martin Hans
@martinhansdk
I'd like to use flatbuffers on an embedded system written in C++. We don't allow dynamic memory allocation here. Is it possible to have a pre-allocated buffer and hand it to flatbuffers to serialize the data into? What about when deserializing?
MikkelFJ
@mikkelfj
Reading FlatBuffers requires no allocation in C/C++ but might in Java etc. due to string allocation. As building buffers I can't answer for C++, but I can give some guidance on FlatBuffers for C. C requires some dynamic allocation but it can be limited in size if your buffers are simple. The allocator can be overriden so you can place allocations in a reserved area. There might be reallocation requests to grow areas, but that will not happen if your buffers are reasonable. It requires experimentation to decide the exact requirements for the problem at hand.
C generated buffer interfaces can be called from C++
Wouter van Oortmerssen
@aardappel
@martinhansdk yes you can specify an allocator for the C++ builder which can then supply existing memory. It can do all its building in that one buffer and will not allocate any further memory
tsindhuja
@tsindhuja
@aardappel I was unable to modify the table as the classes in it were generated as unique_ptr template. I am now able to access it after changing the schema file to modify it to a generic template.
Wouter van Oortmerssen
@aardappel
@tsindhuja again, what error are you getting? you should be able to modify unique_ptrs
tsindhuja
@tsindhuja
@aardappel the error was in terms of "Trying to access deleted function" when I was trying to use mutable_node()->end()
Wouter van Oortmerssen
@aardappel
and node is a table type? and what is end ?
tsindhuja
@tsindhuja
Yes, node is a table type and 'end' is a function that has to iterate to the end
The main issue was to iterate through the collection of tables that is a pointer in another table. And also every assignment operation had to be done via the move() function
jay logan
@jnexthop:matrix.org
[m]
I am exploring flatbuffers use in an embedded system between two unix processes. The messages are relatively small (200 bytes or so), flat non hierarchical, and all of the fields will be used to store some state and process. The messages themselves are about few hundred/minute. It is highly likely the received messages will be again copied into a local struct. Is this an appropriate use of flatbuffers?
Wouter van Oortmerssen
@aardappel
@jnexthop:matrix.org well certainly a "few hundred per minute" is so little any serializer would work
but yes, that sounds fine, you could use the object API instead of local structs
MikkelFJ
@mikkelfj
You can do that, but if you only have simple structs without strings or arrays, and it is process local, you are probably better off just using structs. If you do have strings and arrays, you may consider keeping data in flatbuffers to avoid the extra copying and allocation of data, even if it is not a performance concern. For what it is worth, the C interface allows you to create flatbuffers that have structs as root which can be accessed directly on little endian systems. Other languages require tables as root which are more expensive to create, but they can contain structs. Since you only have a small number of messages, this is hardly a critical point.
Edit: not process local, but system local, meaning same endian, same type of compiler, word size etc.
jay logan
@jnexthop:matrix.org
[m]
@aardappel: thanks, i am constrained to use C. I am assuming object API means the flatbuffers are unpacked into a local struct to give access like foo->bar->bas ? or do i use the flatbuffer offset api (generated) to get to the fields?
MikkelFJ
@mikkelfj
The object API is for C++ only working much like a DOM I suppose. Structs are directly accessible in the C Flatbuffer interface, but they cannot contain pointers like strings or arrays. You normally use functions to access struct fields because they provide endian translation when needed, and next to no overhead otherwise. You can access the structs natively in C on little endian systems.