Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 22:45
    eolivelli commented #4914
  • Jan 31 2019 22:07
    samsartor starred google/flatbuffers
  • Jan 31 2019 21:28
    marang starred google/flatbuffers
  • Jan 31 2019 20:51
    thyrlian starred google/flatbuffers
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 18:56
    aardappel commented #4914
  • Jan 31 2019 18:54
    aardappel commented #5144
  • Jan 31 2019 18:51
    aardappel commented #5141
  • Jan 31 2019 18:51
    aardappel commented #5145
  • Jan 31 2019 18:51
    krojew commented #5142
  • Jan 31 2019 18:49
    krojew commented #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:47
    gabyx commented #5142
  • Jan 31 2019 18:47
    aardappel commented #5002
  • Jan 31 2019 18:43
    gabyx commented #5142
  • Jan 31 2019 18:43
    krojew commented #5142
  • Jan 31 2019 18:43
    aardappel commented #5143
  • Jan 31 2019 18:42
    gabyx commented #5142
Jay Narale
@carbylamine
Is there any other way of performing mutations using the Go API except creating the buffer again
and using mutation buffers?
MikkelFJ
@mikkelfj
You cannot mutate a buffer without mutations … but you can avoid rewriting the buffer if you have a scalar, or an array of scalars where the new length is not longer, and the same for strings. How to do that depends on the language interface.
However, you MUST trust the buffer before doing this because a malicious buffer could overlap seemingly harmless data with other buffer structures that then goes out of bounds even if the buffer has been verified.
Jay Narale
@carbylamine
I mean for mutating complex tables etc
MikkelFJ
@mikkelfj
You can only update existing table fields or truncated the length of existing arrays. Of course you can do more if there is somehow space in the buffer, but that amounts to writing a new buffer essentially.
I know see you mention Go, I’m not sure what Go permits, but if you can get the buffer offset to a field value, you can probably update it on the raw array, if you are on a little endian host.
Jay Narale
@carbylamine
But the C++ API supports mutations right? So I thought maybe there must be some tweak in go
MikkelFJ
@mikkelfj
See above, posted same time as your comment.
It is likely that you can get that offset if you look at Gos implementation, but it is probably not guaranteed to work for future releases.
I mean look at the generated code in detail.
More generally, I think it is better avoided to mutate buffers unless you have a specific highperformance case where you use mutations as a template for sending data at gigabit rates.
I do wish Go would make it easier to clone a part of a buffer, though.
Jay Narale
@carbylamine
Yes I think we can get The offset in golang, will try it. So you would suggest creating a new encoding again?
MikkelFJ
@mikkelfj
Yes. If your buffer is very large, and the changes small, better keep the changes in a separate buffer, and if the buffer is smallish, just rewrite it.
Jay Narale
@carbylamine
Okay will try, thanks a lot!
MikkelFJ
@mikkelfj
You are welcome
SSorbello
@SSorbello

Good Morning everybody, i just started to utilize flatbuffers on C# , inside a table i defined:
Value:[ubyte] (flexbuffer);

there is C# support for the flexbuffer in the original repo or i should rely on https://github.com/mzaks/FlexBuffers-CSharp ?

MikkelFJ
@mikkelfj
I don’t know, but it is rare that someone says goodmorning when it is actually morning where I live, so good morning to you too. My guess is you should use Max’ implementation you reference.
SSorbello
@SSorbello
Thanks, i will continue on this way.
Maxim Zaks
@mzaks
@SSorbello you are welcome to use my implementation. Let me know if you have any issues with it. AFAIK there is no C# FlexBuffers support on Googles repo and there is probably no extra support in flatc for C# code gen.
cyberquarks
@cyberquarks
Hi, can you use FlatBuffers to upload an image or short video?
MikkelFJ
@mikkelfj
@cyberquarks yes, you can have arbitrary binary data in flatbuffers, as long as it is below 2GB per buffer. FlatBuffers is perhaps best suited for containing metadata such as video title, copyright, image encoding etc., but raw frames can also be encoded. It is not suited for actually playing back video, at least not without a new dedicated video format.
Jon Lederman
@jon2718

Hi,

I am trying to use FlatBuffers in Swift.
After compiling the Monstes example with flatc --swift monsters.fbs, I then add the generated file into my Xcode project as a swift file.

When trying to build I get many errors of the type:

Cannot use mutating member on immutable value: 'fbb' is a 'let' constant, where fbb is a FlatBuffersBuilderType

I've tried the cocoapods as well as building from source. I am on Flatc 1.12.0
Not sure if the code is still being updated and whether Swift is even supported.

Any help would be appreciated.

Thank you.

MikkelFJ
@mikkelfj
@mzaks is the one you should ask, I can't help you here. But generally, building flatbuffers and reading flatbuffers are two separate things, so perhaps you are mixing types from the read api with the types from builder api.
Jon Lederman
@jon2718
perhaps I am making a mistake in my use of it. the docs weren't 100% clear
can you explain the diff between read api and builder api types
I just used flatc --swift monsters.fbs to generate the code and imported that into my Xcode project for ios
MikkelFJ
@mikkelfj
I’m not working with swift so I cannot be specific. The difference between build and read is that the buffers is first constructed once using various types that usually holds temporary data references, whereas the read types are designed to efficiently reference data in the constructed buffer.
Wouter van Oortmerssen
@aardappel
@jon2718 open an issue on github, and tag @mustiikhalil who is the swift maintainer
Jon Lederman
@jon2718
Ok. Can u provide instructions
Jon Lederman
@jon2718
Flatc seems to generate wrong code b/c it is trying to generate a struct for FlatBuffersBuilder and pass that to static method, which cannot be marked mutating. Something appears quite wrong.
cyberquarks
@cyberquarks
@mikkelfj thanks
adsharma
@adsharma

I put together a bunch of schemas in flatbuffer format, primarily to see if the ideas expressed in my earlier blog post (https://adsharma.github.io/flattools/) actually work.

https://github.com/adsharma/fbs-schemas/

This is all experimental work - there may be bugs or syntax errors in the schemas in the submodules. Please file bug reports or send me fixes if it looks interesting. The telegram schema is the most extensive one I've found to date.

MikkelFJ
@mikkelfj
I also use the flatcc compilers library to parse flatbuffers in another program and generate protocol code for fast simple memory layout, but without nested type support. Someone else once mentioned using flatbuffers schema to drive a huge project, not sure what it was, UI elements or OpenGL or something.
I prefer the int32 and uint64 over int and ulong types since you don't have to very familiar with flatbuffers to understand the size of the types. Which is especially important for cross protocol IDL.
FYI you can use int32 as an alias for int in Flatbuffers schema.
MikkelFJ
@mikkelfj

From the blog:

Using the deprecated attribute is recommended.

I can’t wait till we deprecate that feature ;)

adsharma
@adsharma

I prefer the int32 and uint64 over int and ulong types
Good tip thanks. I'd use int/ulong only when the author wants to use the corresponding types in the languages we generate from the schema.

Yeah - deprecation is complicated. But I like it so much better than field numbering when generating schema at a conceptual level (telegram_api.tl stuff).

Maxim Zaks
@mzaks
I few suggestions from me regarding the core fbs:
  • IMHO language should be an enum instead of a string, the set is final so there is no need for wasting space, even when deduplication for strings is on a reference to a string is 4 bytes long by itself.
  • MimeType type is a bit complicated as the set can be non final, but in this case I would also define most of the known types as enum, and then add a custom case.
Maxim Zaks
@mzaks
enum StandardMimeType: byte {application_octstream, application_json, ... , custom}
table MimeType {
  mimeType: StandardMimeType;
  customMimeType: string;
  params:string;
}
And for URL, I think it makes sense to break it up. In a payload it is often that atleast base url repeats itself very often. In this case IMHO it is better to turn the String deduplication on and save some space.
something like:
table URL {
  baseUrl: string;
  pathComponents: [string];
  parameters: string;
}
Maxim Zaks
@mzaks
pathComponents are an array of string because I guess it is often that parts of the path repeat themselves.
I made parameters as just one concatenated string, becuase my guess is often the params are repeated as whole as well. But those are just my speculations, if you have examples it is better to check those and see what separation would be better
Maxim Zaks
@mzaks
and why would box String and Bool? if it is for null comparibility, there is a new feature which just implemented default nulls for scalars. In fact, types like Date, Duration, int128, int256 should be defined as struct. Structs are non evolvable, but we are talking about base types which should not be evolved and structs, don't have the overhead of virtual table and are stored inline.
Well regarding structs I guess if you use just the fbs and not the FlatBuffers as is, then it probably does not matter
Maxim Zaks
@mzaks
Ah OK, I looked a bit further through your examples I guess you have your own own generator from fbs to other languages, or do you use flatc? It seems to me that the generated classes are not typical for flatc output. So you don't use FlatBuffers for Flatbuffers sake but rather just the schema syntax. So all the deduplication and tables vs. structs remarks I did are not really applicable in this case.