by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 22:45
    eolivelli commented #4914
  • Jan 31 2019 22:07
    samsartor starred google/flatbuffers
  • Jan 31 2019 21:28
    marang starred google/flatbuffers
  • Jan 31 2019 20:51
    thyrlian starred google/flatbuffers
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 18:56
    aardappel commented #4914
  • Jan 31 2019 18:54
    aardappel commented #5144
  • Jan 31 2019 18:51
    aardappel commented #5141
  • Jan 31 2019 18:51
    aardappel commented #5145
  • Jan 31 2019 18:51
    krojew commented #5142
  • Jan 31 2019 18:49
    krojew commented #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:47
    gabyx commented #5142
  • Jan 31 2019 18:47
    aardappel commented #5002
  • Jan 31 2019 18:43
    gabyx commented #5142
  • Jan 31 2019 18:43
    krojew commented #5142
  • Jan 31 2019 18:43
    aardappel commented #5143
  • Jan 31 2019 18:42
    gabyx commented #5142
Wouter van Oortmerssen
@aardappel
crazy :)
MikkelFJ
@mikkelfj
yes
MikkelFJ
@mikkelfj
I once had a convention of many thousands of dining philosphers seated a different tables to test concurrent concurrency and garbage collection. It was quite amazing to see it spit out a million chop stick passes without breaking a sweat.
Wouter van Oortmerssen
@aardappel
haha nice
classical test :)
Tsingson
@tsingson
@aardappel
this one ?
```
Namespace *Parser::UniqueNamespace(Namespace *ns) {
  for (auto it = namespaces_.begin(); it != namespaces_.end(); ++it) {
    if (ns->components == (*it)->components) {
      delete ns;
      return *it;
    }
  }
  namespaces_.push_back(ns);
  return ns;
}
this func clean up ?
Tsingson
@tsingson
memory leaks bug is fixed.
pavanbadugu
@pavanbadugu
can anyone help me on this
a,b are tables
c is a table in which a,b are fields
so i got the a flatbuffer data and b flatbuffer data which are loader seperately from file so now i need to construct a flatbuffer for c using .. is there any way just for copy raw bytes to flatbuffer and construct c table any api's for c++
MikkelFJ
@mikkelfj
No you cannot trivially do that in C++ as far as I know. You may be able to use the object API which is slower, but I am not familiar with that. For C, there is a clone method which can copy tables from other sources into a new buffer - it will do what you want, but it is still not simple block copy - each table field is copied recursively into the the new buffer.
pavanbadugu
@pavanbadugu
oh ok thanks
MikkelFJ
@mikkelfj
You can, however, store a and b as nested buffers if that is acceptable to you. That will allow storing each sub-buffer directly as a blob. It requires a schema change.
Normally I’d not recommend that, but it depends on the circumstances.
pavanbadugu
@pavanbadugu
yeah this was my alternative thanks for confirming i'll do this thanks @mikkelfj
Tsingson
@tsingson

In terms of the fbs specification, a table consists of two parts, a vtable , indicating all the fields in a table, and a binary array, containing all the actual field data. If you need to get a table as a field inside another table from a generated flatbuffer, just get the corresponding vtable and the actual data, then add a header to be a new flatbuffer.

In other words, it is theoretically possible to intercept the table fields from the generated flatbuffer and recreate a new flatbuffers.

In practice, there is no such API available. (If you are familiar with fbs, you can develop it yourself.)

pavanbadugu
@pavanbadugu
@tsingson will try that where can i find doc on how flatbuffer serelisation specification
Wouter van Oortmerssen
@aardappel
@tsingson and the table fields may point to more tables, strings and vectors, so this is not trivial
Maxim Zaks
@mzaks
Em .. not that simple @tsingson the values are stored together only if they are scalars (bool, int, uint, float). If properties in table are of type vector, string, union or other table, than there is a relative offset, which points to some region in the buffer
:)
Wouter van Oortmerssen
@aardappel
see, I typed a shorter response, so I got to go first :P
Maxim Zaks
@mzaks
Story of my life :)
pavanbadugu
@pavanbadugu
quick question when is latest grpc compatible code generation support for c++ happening
currently flatc generated code is supported till grpc 16.x
@aardappel @mzaks
Maxim Zaks
@mzaks
I have no idea, sorry.
MikkelFJ
@mikkelfj
@tsingson you can do this in flatcc for C - but it is a lot of work - you can bring your own vtable and you can get a pointer to a source table, but fields must still be processed recursively as Wouter says.

see, I typed a shorter response, so I got to go first :P

Missing a thumbs up here.

pavanbadugu
@pavanbadugu
quick question when is latest grpc compatible code generation support for c++ happening and javascript support
Wouter van Oortmerssen
@aardappel
@pavanbadugu some gRPC related fixes were merged just like.. a week ago. Are you working from latest?
pavanbadugu
@pavanbadugu
yes still facing couple of issues will push some example code to github and put it as a issue in repository please check it once @aardappel thank you
Wouter van Oortmerssen
@aardappel
Reminder, there is also a Discord FlatBuffers chat: discord.gg/6qgKs3R
Tsingson
@tsingson

@tsingson and the table fields may point to more tables, strings and vectors, so this is not trivial

yes. but, it's still can found the table field in a table. a table as a field , it's a vector.

Tsingson
@tsingson

@tsingson you can do this in flatcc for C - but it is a lot of work - you can bring your own vtable and you can get a pointer to a source table, but fields must still be processed recursively as Wouter says.

i can do it for go, i m not good for c / c++.

i m trying some job in fbs in go, make it's API clear and easy to use. at lease , in my business project.
Michael Ramos
@backnotprop

Hey all. Are there any examples of building nested flatbuffer in Javascript/Typescript.... I have an inter-service deployment working well with Golang...struggling on the client front specifically building the nested FB... using same approach as go, the JS lacks builder.createbytevector so have tried working a similar approach with asUint8Array ...

// 1. create nested FB with its own builder
... store finished buffer into asUint8Array

// 2. builder for root_table
... offset from generated table field createvector with nested FB asUint8Array
// 3. start root table
... add root table nested FB with above offset

the nested FB is not accessible on the server side when sent from the client....
maybe its my fetch request but I am sending raw bytes via fetch

Michael Ramos
@backnotprop
seems related: google/flatbuffers#4500
oh
that was it
asUInt8Array().slice(0) needed for nested and send data
William
@imWillX
Hello,
I am trying to get tables within a flatbuffer schema that is not a root type. Is this a feature?
MikkelFJ
@mikkelfj
It depends on the target language, but I believe most languages support having any table as the root table in the buffer, if that is what you mean. The root declaration in the schema is mostly for JSON parsing where the parser must know what table to parse as root. In FlatCC for C, you can also parse other tables as root, but there is more direct support for the declared root.
Konstantin Bläsi
@konstantinblaesi
I am trying to compare buffer/payload sizes of flatbuffers vs JSON, basically trying to see how much I save regarding wire transfer size. I came up with this code after reading some stackoverflow post on how to parse/serialize from/to JSON https://dpaste.org/BtXL According to that the ratio JSON / flatbuffer is 1.66237 . Does that make sense what I am doing ? I was expecting a bigger difference after seeing https://google.github.io/flatbuffers/flatbuffers_benchmarks.html, but what do I know :)
Wouter van Oortmerssen
@aardappel
@konstantinblaesi that really depends on your data.. the bulk of that data is two strings and a double. The strings are by themselves slightly larger in FlatBuffers, but you save not having to write the key or any of the formatting around it. The double is always 8 bytes in FlatBuffers, in your example it is sometimes shorter (0) and sometimes longer (58096.25652706856), and again you save the keys etc.
You could turn the double into a float, but thats not a big savings. The bigger one is to turn unit from a string into an enum, making it as small as 1 byte
Konstantin Bläsi
@konstantinblaesi
@aardappel thanks for the quick feedback and insights :)
MikkelFJ
@mikkelfj
If you have gzip compressed JSON, you can generally assume that JSON will be larger than GZIP compressed FlatBuffers by a factor 2 for smaller messages - say below 1K, but for messages above 100K GZIP compressed JSON will smaller than GZIP compressed flatbuffers by a factor 2. Alls this will vary on payload and schema. For uncompressed data, JSON tend to be large - I’m guessing a factor 4-10 for small messages. The reason is that JSON keywords take up space and flatbuffer pointers take up space. Keywords compress better when there are many of them.
Michael Ramos
@backnotprop
@mikkelfj and thus upon exiting inter-service processing, it might be best to send gzip json to a client that intends to immediately parse into json, correct? Or could there still be better efficiencies in deserializing a flatbuffer into native objects and skipping over json entirely (even in JS) ... guess here what would matter is flattbuffer vector iteration efficiency vs array reference/copy assignments
MikkelFJ
@mikkelfj
I’m not sure what you mean by inter-service processing - if you mean internal processing vs. public API, then I think it often makes most sense to use HTTPS gzip’ed JSON on the API but FlatBuffers internally. I work with MQTT interfaces that are uncompressed FlatBuffers both internally and externally via a browser MQTT connection. If you use JSON, you can parse it very quickly into FlatBuffers using FlatCC generated parsers (and printers for the opposite direction). However, using FlatBuffers over MQTT is more concerned with processing speed and overhead than with size. If you are bandwidth sensitive (such as paying for traffic), or you need to adhere to public conventions (REST API), then gzip’ed JSON might make sense, but you need to test for size.
Also FlatBuffers over MQTT from browser is not necessarily for performance reasons, but to avoid having an additional gateway, and to use the FlatBuffers schema vs more loose JSON. You do need some access control and validation though. JSON parsing is more robust against abuse.
MikkelFJ
@mikkelfj
As to (de)serializing to native objects, that very much depends on the use case - but generally that would be a lot of difficult to maintain code that is not very portable, and likely not faster. But it depends on the language and use case. You can also find techniques that are even faster or smaller than flatbuffers, especially when creating buffers, but FlatBuffers strikes the balance of schema evolution, size and speed prettye well. You would loose some of that with a custom to native framework.