Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 22:45
    eolivelli commented #4914
  • Jan 31 2019 22:07
    samsartor starred google/flatbuffers
  • Jan 31 2019 21:28
    marang starred google/flatbuffers
  • Jan 31 2019 20:51
    thyrlian starred google/flatbuffers
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 19:19
    harshshah903 commented #5144
  • Jan 31 2019 18:56
    aardappel commented #4914
  • Jan 31 2019 18:54
    aardappel commented #5144
  • Jan 31 2019 18:51
    aardappel commented #5141
  • Jan 31 2019 18:51
    aardappel commented #5145
  • Jan 31 2019 18:51
    krojew commented #5142
  • Jan 31 2019 18:49
    krojew commented #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:48
    gabyx edited #5142
  • Jan 31 2019 18:47
    gabyx commented #5142
  • Jan 31 2019 18:47
    aardappel commented #5002
  • Jan 31 2019 18:43
    gabyx commented #5142
  • Jan 31 2019 18:43
    krojew commented #5142
  • Jan 31 2019 18:43
    aardappel commented #5143
  • Jan 31 2019 18:42
    gabyx commented #5142
MikkelFJ
@mikkelfj
I mean look at the generated code in detail.
More generally, I think it is better avoided to mutate buffers unless you have a specific highperformance case where you use mutations as a template for sending data at gigabit rates.
I do wish Go would make it easier to clone a part of a buffer, though.
Jay Narale
@carbylamine
Yes I think we can get The offset in golang, will try it. So you would suggest creating a new encoding again?
MikkelFJ
@mikkelfj
Yes. If your buffer is very large, and the changes small, better keep the changes in a separate buffer, and if the buffer is smallish, just rewrite it.
Jay Narale
@carbylamine
Okay will try, thanks a lot!
MikkelFJ
@mikkelfj
You are welcome
SSorbello
@SSorbello

Good Morning everybody, i just started to utilize flatbuffers on C# , inside a table i defined:
Value:[ubyte] (flexbuffer);

there is C# support for the flexbuffer in the original repo or i should rely on https://github.com/mzaks/FlexBuffers-CSharp ?

MikkelFJ
@mikkelfj
I don’t know, but it is rare that someone says goodmorning when it is actually morning where I live, so good morning to you too. My guess is you should use Max’ implementation you reference.
SSorbello
@SSorbello
Thanks, i will continue on this way.
Maxim Zaks
@mzaks
@SSorbello you are welcome to use my implementation. Let me know if you have any issues with it. AFAIK there is no C# FlexBuffers support on Googles repo and there is probably no extra support in flatc for C# code gen.
cyberquarks
@cyberquarks
Hi, can you use FlatBuffers to upload an image or short video?
MikkelFJ
@mikkelfj
@cyberquarks yes, you can have arbitrary binary data in flatbuffers, as long as it is below 2GB per buffer. FlatBuffers is perhaps best suited for containing metadata such as video title, copyright, image encoding etc., but raw frames can also be encoded. It is not suited for actually playing back video, at least not without a new dedicated video format.
Jon Lederman
@jon2718

Hi,

I am trying to use FlatBuffers in Swift.
After compiling the Monstes example with flatc --swift monsters.fbs, I then add the generated file into my Xcode project as a swift file.

When trying to build I get many errors of the type:

Cannot use mutating member on immutable value: 'fbb' is a 'let' constant, where fbb is a FlatBuffersBuilderType

I've tried the cocoapods as well as building from source. I am on Flatc 1.12.0
Not sure if the code is still being updated and whether Swift is even supported.

Any help would be appreciated.

Thank you.

MikkelFJ
@mikkelfj
@mzaks is the one you should ask, I can't help you here. But generally, building flatbuffers and reading flatbuffers are two separate things, so perhaps you are mixing types from the read api with the types from builder api.
Jon Lederman
@jon2718
perhaps I am making a mistake in my use of it. the docs weren't 100% clear
can you explain the diff between read api and builder api types
I just used flatc --swift monsters.fbs to generate the code and imported that into my Xcode project for ios
MikkelFJ
@mikkelfj
I’m not working with swift so I cannot be specific. The difference between build and read is that the buffers is first constructed once using various types that usually holds temporary data references, whereas the read types are designed to efficiently reference data in the constructed buffer.
Wouter van Oortmerssen
@aardappel
@jon2718 open an issue on github, and tag @mustiikhalil who is the swift maintainer
Jon Lederman
@jon2718
Ok. Can u provide instructions
Jon Lederman
@jon2718
Flatc seems to generate wrong code b/c it is trying to generate a struct for FlatBuffersBuilder and pass that to static method, which cannot be marked mutating. Something appears quite wrong.
cyberquarks
@cyberquarks
@mikkelfj thanks
adsharma
@adsharma

I put together a bunch of schemas in flatbuffer format, primarily to see if the ideas expressed in my earlier blog post (https://adsharma.github.io/flattools/) actually work.

https://github.com/adsharma/fbs-schemas/

This is all experimental work - there may be bugs or syntax errors in the schemas in the submodules. Please file bug reports or send me fixes if it looks interesting. The telegram schema is the most extensive one I've found to date.

MikkelFJ
@mikkelfj
I also use the flatcc compilers library to parse flatbuffers in another program and generate protocol code for fast simple memory layout, but without nested type support. Someone else once mentioned using flatbuffers schema to drive a huge project, not sure what it was, UI elements or OpenGL or something.
I prefer the int32 and uint64 over int and ulong types since you don't have to very familiar with flatbuffers to understand the size of the types. Which is especially important for cross protocol IDL.
FYI you can use int32 as an alias for int in Flatbuffers schema.
MikkelFJ
@mikkelfj

From the blog:

Using the deprecated attribute is recommended.

I can’t wait till we deprecate that feature ;)

adsharma
@adsharma

I prefer the int32 and uint64 over int and ulong types
Good tip thanks. I'd use int/ulong only when the author wants to use the corresponding types in the languages we generate from the schema.

Yeah - deprecation is complicated. But I like it so much better than field numbering when generating schema at a conceptual level (telegram_api.tl stuff).

Maxim Zaks
@mzaks
I few suggestions from me regarding the core fbs:
  • IMHO language should be an enum instead of a string, the set is final so there is no need for wasting space, even when deduplication for strings is on a reference to a string is 4 bytes long by itself.
  • MimeType type is a bit complicated as the set can be non final, but in this case I would also define most of the known types as enum, and then add a custom case.
Maxim Zaks
@mzaks
enum StandardMimeType: byte {application_octstream, application_json, ... , custom}
table MimeType {
  mimeType: StandardMimeType;
  customMimeType: string;
  params:string;
}
And for URL, I think it makes sense to break it up. In a payload it is often that atleast base url repeats itself very often. In this case IMHO it is better to turn the String deduplication on and save some space.
something like:
table URL {
  baseUrl: string;
  pathComponents: [string];
  parameters: string;
}
Maxim Zaks
@mzaks
pathComponents are an array of string because I guess it is often that parts of the path repeat themselves.
I made parameters as just one concatenated string, becuase my guess is often the params are repeated as whole as well. But those are just my speculations, if you have examples it is better to check those and see what separation would be better
Maxim Zaks
@mzaks
and why would box String and Bool? if it is for null comparibility, there is a new feature which just implemented default nulls for scalars. In fact, types like Date, Duration, int128, int256 should be defined as struct. Structs are non evolvable, but we are talking about base types which should not be evolved and structs, don't have the overhead of virtual table and are stored inline.
Well regarding structs I guess if you use just the fbs and not the FlatBuffers as is, then it probably does not matter
Maxim Zaks
@mzaks
Ah OK, I looked a bit further through your examples I guess you have your own own generator from fbs to other languages, or do you use flatc? It seems to me that the generated classes are not typical for flatc output. So you don't use FlatBuffers for Flatbuffers sake but rather just the schema syntax. So all the deduplication and tables vs. structs remarks I did are not really applicable in this case.
MikkelFJ
@mikkelfj

Yeah - deprecation is complicated.

It is, but I was being sarcastic:
deprecated has been deprecated ...

As to boxed types, I can see why you would want them all to be tables, but boxing actually makes more sense with structs as Max points out, except for variable length strings. int128 can be represented as a struct int128 (align: 16) { val: [byte:16]; }. I have actually thought about adding typedef support for such wrappers. These structs have no overhead compared to native integers, but contrary to native integers (for supported sizes), they can be used in Unions.
MikkelFJ
@mikkelfj
You might want to use Int128 in upper case, and also define Int64 as struct { val: int64; } etc. because then all types can work in Unions, and all wrapped types are consistently uppercase. Lowercase int128 could theoretically conflict with a future FlatBuffer native type.
MikkelFJ
@mikkelfj
@mzaks I would keep URL simple. If needed you could added a ParsedURL or something in a separate library. Optimizing for reuse of string components likely cost more than they offer unless you have a lot of urls with long reused components. E.g. a 6 byte component would use 16 bytes: 4 for reference in the components in array, 4 for length, 6 for content, 1 for 0 terminator, and 1 for padding. Using it embedded in a another string would only use 7 bytes including a ‘/'. But of course this is for metadata and not actual contents.
adsharma
@adsharma

Ah OK, I looked a bit further through your examples I guess you have your own own generator from fbs to other languages, or do you use flatc? It seems to me that the generated classes are not typical for flatc output. So you don't use FlatBuffers for Flatbuffers sake but rather just the schema syntax. So all the deduplication and tables vs. structs remarks I did are not really applicable in this case.

That's right. This is the compiler I use:

https://github.com/adsharma/flattools/blob/master/bin/flatc.py

adsharma
@adsharma

The idea is that flatbuffers are useful when you have large buffers and you want zero overhead deserialization. Perhaps at other times you want to choose a different serialization format (thrift supported multiple, but I haven't looked lately).

So what if each table can specify its own serialization format as an annotation? It should be possible to translate that into a swift attribute?

table message (serde: flatbuffers) {
...
}

could end up with:

@flatbuffer
class Message {
...
}

But perhaps you want to use @grpc or some other attribute for other tables. This way the choice of serialization format doesn't drive the choice of IDL.

As to boxed types, I can see why you would want them all to be tables, but boxing actually makes more sense with structs as Max points out, except for variable length strings. int128 can be represented as a struct int128 (align: 16) { val: [byte:16]; }. I have actually thought about adding typedef support for such wrappers. These structs have no overhead compared to native integers, but contrary to native integers (for supported sizes), they can be used in Unions.

I didn't write those types. The boxed types exist in Telegram's TL (type language) described here:

https://core.telegram.org/mtproto/TL

The included script (compiler.py - which I derived from another project) spits out all the fbs files in the directory.

Yes - unions are a major use case for boxed types. Having multiple constructors is another I think.

adsharma
@adsharma

You might want to use Int128 in upper case, and also define Int64 as struct { val: int64; } etc. because then all types can work in Unions, and all wrapped types are consistently uppercase. Lowercase int128 could theoretically conflict with a future FlatBuffer native type.

Yup. Int128 makes more sense. Will do.

https://github.com/adsharma/fbs-schemas/blob/main/core.fbs

Most of these types came from ActivityStreams 2.0 (which is a JSON-LD schema spec). I feel the flatbuffer schema is 100x more readable for the average programmer, even if JSON-LD is more general and might have other capabilities.

MikkelFJ
@mikkelfj

So what if each table can specify its own serialization format as an annotation?

That might get a bit clumsy, and you might want to use multiple serialization formats, for example in a gateway.

Lijie.Jiang
@lijie-jiang
Would it be possible to include multiple input path with '-I' option? google/flatbuffers#6346