by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 22:45
    eolivelli commented #4914
  • Jan 31 2019 22:07
    samsartor starred google/flatbuffers
  • Jan 31 2019 21:28
    marang starred google/flatbuffers
  • Jan 31 2019 20:51
    thyrlian starred google/flatbuffers
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 18:56
    aardappel commented #4914
  • Jan 31 2019 18:54
    aardappel commented #5144
  • Jan 31 2019 18:51
    aardappel commented #5141
  • Jan 31 2019 18:51
    aardappel commented #5145
  • Jan 31 2019 18:51
    krojew commented #5142
  • Jan 31 2019 18:49
    krojew commented #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:47
    gabyx commented #5142
  • Jan 31 2019 18:47
    aardappel commented #5002
  • Jan 31 2019 18:43
    gabyx commented #5142
  • Jan 31 2019 18:43
    krojew commented #5142
  • Jan 31 2019 18:43
    aardappel commented #5143
  • Jan 31 2019 18:42
    gabyx commented #5142
MikkelFJ
@mikkelfj
If you have gzip compressed JSON, you can generally assume that JSON will be larger than GZIP compressed FlatBuffers by a factor 2 for smaller messages - say below 1K, but for messages above 100K GZIP compressed JSON will smaller than GZIP compressed flatbuffers by a factor 2. Alls this will vary on payload and schema. For uncompressed data, JSON tend to be large - I’m guessing a factor 4-10 for small messages. The reason is that JSON keywords take up space and flatbuffer pointers take up space. Keywords compress better when there are many of them.
Michael Ramos
@backnotprop
@mikkelfj and thus upon exiting inter-service processing, it might be best to send gzip json to a client that intends to immediately parse into json, correct? Or could there still be better efficiencies in deserializing a flatbuffer into native objects and skipping over json entirely (even in JS) ... guess here what would matter is flattbuffer vector iteration efficiency vs array reference/copy assignments
MikkelFJ
@mikkelfj
I’m not sure what you mean by inter-service processing - if you mean internal processing vs. public API, then I think it often makes most sense to use HTTPS gzip’ed JSON on the API but FlatBuffers internally. I work with MQTT interfaces that are uncompressed FlatBuffers both internally and externally via a browser MQTT connection. If you use JSON, you can parse it very quickly into FlatBuffers using FlatCC generated parsers (and printers for the opposite direction). However, using FlatBuffers over MQTT is more concerned with processing speed and overhead than with size. If you are bandwidth sensitive (such as paying for traffic), or you need to adhere to public conventions (REST API), then gzip’ed JSON might make sense, but you need to test for size.
Also FlatBuffers over MQTT from browser is not necessarily for performance reasons, but to avoid having an additional gateway, and to use the FlatBuffers schema vs more loose JSON. You do need some access control and validation though. JSON parsing is more robust against abuse.
MikkelFJ
@mikkelfj
As to (de)serializing to native objects, that very much depends on the use case - but generally that would be a lot of difficult to maintain code that is not very portable, and likely not faster. But it depends on the language and use case. You can also find techniques that are even faster or smaller than flatbuffers, especially when creating buffers, but FlatBuffers strikes the balance of schema evolution, size and speed prettye well. You would loose some of that with a custom to native framework.
If you want smaller, use protocol buffers, if you want faster, use FastBuffers or SBE, but these make other trade offs. If you want more dynamic schemaless, use MessagePack or JSON. In my view JSON, compressed JSON, and FlatBuffers makes the most sense in most cases, especially JSON that is compatible with FlatBuffers so it can be converted.
Michael Ramos
@backnotprop
"internal processing vs. public API" ah yea, better/simpler way of explaining. I create and send flatbuffers into internal processing that is first balanced, parallelly, across workers (nodes/cores) (all receiving original FB but calculated split assignment for processing), and then an aggregated bulk/large json package is created and sent back to the client. Flatbuffers already gave us serious performance gains in the processing and especially because we could balance between available workes without reconstructing. Thanks for the feedback.
we require optimized performance and FB is definitely giving us that. Just looking into further improvements now on the client-side.
MikkelFJ
@mikkelfj
I’d consider MQTT and FlatBuffers client side if you control everything. But you need to consider the traffic cost vs performance tradeoff. Mind you - parsing and printing JSON via FlatCC is only 4 times slower than creating a buffer and 40 slower than reading a buffer. (Parsing would require reading the buffer subsequently). FlatBuffer reading is 2x slower than the fastest possible native data structure access. These are rough figures.
Note that MQTT gives an extra hop, and thus added latency. You can reduce that by hosting the broker on same machine as the server endpoint. But it is very flexibible, fast, and robust.
Kubernetes can also give extra hops BTW - use ExternalTrafficPolicy to avoid.
Michael Ramos
@backnotprop
good read/call on K8s. Will look into. & yea we made those considerations in the processing layers except flatcc/json ...interesting... because one hard bottleneck on processing side is list & numpy array construcion & concat (tf inputs)
MikkelFJ
@mikkelfj
There is the Arrow format which uses FlatBuffers for metadata - more suited for large volumes of similar data and with Python support - but not sure how mature.
Wouter van Oortmerssen
@aardappel
Wouter van Oortmerssen
@aardappel
Ok, from now on we'll have flatc binaries on every commit thanks to github actions: https://github.com/google/flatbuffers/actions/runs/96547077
MikkelFJ
@mikkelfj
are there limits on how long they are stored?
Wouter van Oortmerssen
@aardappel
I don't actually know.. I know it is limited, not sure if its just a FIFO thing
but yeah, that means you probably don't want to hard-link to any of them, instead just go find the latest
Ori Popowski
@oripwk

Hi, I'm struggling with building a FlatBuffer which has a field of string array. Can someone please help? I've posted my question here:
https://groups.google.com/forum/#!topic/flatbuffers/DUp-QYvF9hA

Thanks!

Ori Popowski
@oripwk
Hi! I've solved the problem by translating this Javascript code to Java line-by-line:
https://stackoverflow.com/q/46043360/1038182
Ori Popowski
@oripwk

Okay, after I solved this issue, I'm having another problem, and this time I cannot build a FlatBuffer from a byte-array. It's described here: https://groups.google.com/forum/#!topic/flatbuffers/gFOBEtvnO48.

Any help will be much appreciated! :)

Wouter van Oortmerssen
@aardappel
Matthias Vallentin
@mavam
Awesome!
I provided you some details on my question about Flexbuffers vs MsgPack: https://news.ycombinator.com/item?id=23598512
Wouter van Oortmerssen
@aardappel
@mavam cool, replied there :)
Jay Narale
@carbylamine
I think the flat buffers schema does not support maps in golang. Is there any workaround to support serialization of maps without traversing an array each time?
MikkelFJ
@mikkelfj

Maps in flatbuffers are just sorted arrays. I don’t know about Go, but if the buffer is created by another language, you can use binary search on the array in Go, you just have to write the search logic yourself. If you need to sort, it is more complicated. The FlatCC (for C) tool can generate code that can sort a buffer after it has been created, if you trust the buffer. So you could write a tool to sort your buffers.

Additionally I am using hash tables in Go Flatbuffers. It is not supported by the language, but linear open addressing hashes are fairly easy to implement. It requires a data table and a hash index table.

Jay Narale
@carbylamine
@mikkelfj makes sense, thanks!
Lijie.Jiang
@lijie-jiang
Hi all, the generated inline func UnPackTo() caused bus error from my program, does anyone experienced it?
The scenario was below:
Two threads retrieved the identical flatbuffer data, I print them for double check. Then the first thread call UnPackTo() was ok, But the second one failed always. I can't figure out the root reason. .
MikkelFJ
@mikkelfj
Do you unpack to separate object instances?
Lijie.Jiang
@lijie-jiang
Do you unpack to separate object instances?
yes, the two threads unpack their receieved flatbuffer data to local object instance. I've print out the holder buffer and they are definitely identical! This quite confused me.
MikkelFJ
@mikkelfj
Could it be that the original read-only buffers lifetime is controlled by a thread that releases memory too early?
Lijie.Jiang
@lijie-jiang
I used zeromq to publish and subscribe the released flat buffer, the two subscriber should received and hold their data buffer separately I believe.
MikkelFJ
@mikkelfj
so they get a separate copy of the same data?
Lijie.Jiang
@lijie-jiang
yes, I believe so.
And I print them to console, it shows that the contents are the same.
MikkelFJ
@mikkelfj
I really have no clue - but what happens if you comment out the unpack that works?
Lijie.Jiang
@lijie-jiang
yes, if I comment out the unpack statements, every thing is ok.
MikkelFJ
@mikkelfj
no, I mean - comment out the unpack in the one thread that works, and keep the other.
And if you can’t do that, how about testing with a critical section around unpack - not that it should be necessary, but for testing.
Lijie.Jiang
@lijie-jiang
I also tried to close the other receiving thread and just keep one subscriber, it didn't work. I aslo rebuild all the library and program, it reached the same result.
MikkelFJ
@mikkelfj
sorry I can’t help here. But one clue is if the error is reproducible. If it were a concurrency problem you would see things working randomly and not.
Have you tried build with -g option and see if you get any malloc assertions?
If the problem is in unpack, it might be an allocation size error that triggers a problem on second allocation because of memory that has been overwritten.
Lijie.Jiang
@lijie-jiang
If I build with debug, it pointed out the execption place to flatbuffers/base.h and codes are below:

template<typename T> T EndianScalar(T t) {

if FLATBUFFERS_LITTLEENDIAN

return t;

else

return EndianSwap(t);

endif

}

MikkelFJ
@mikkelfj
what hardware, what endinanness are you running?
Lijie.Jiang
@lijie-jiang
The real h/w is an ARM target and little endian.