Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Shauheen
    @shauheen
    The workgroup's first meeting is about to start.
    Simon Long
    @hobei
    @shauheen Hello, I am interested in joining data pipline working group, could you please invite me to the next working group meeting. Many thanks Simon
    Svetlana Levitan
    @sveta-levitan
    Hello! Is there any activity going on with the Pipelines working group? Thank you.
    Michał Zientkiewicz
    @mzient
    Hello! I'm now investigating what's the overlap between current ONNX support for data pipelines and what we've found useful in our experience in the subject. I'll work with @ptrendx and will post initial findings soon, hopefully in a form that can evolve into a spec proposal.
    Our teams efforts so far are focused on image and video processing; contribution in other areas is more than welcome.
    Svetlana Levitan
    @sveta-levitan
    Are there regular meetings of this group?
    Janusz Lisiecki
    @JanuszL
    Not yet, but after posting some initial findings we should consider this
    Svetlana Levitan
    @sveta-levitan
    And you will announce it here, right? Thank you!
    Janusz Lisiecki
    @JanuszL
    Definitely :)
    Svetlana Levitan
    @sveta-levitan
    Hi Michal and Janusz, do you have any progress to report? Are you also looking into ONNX ML or only original ONNX?
    Janusz Lisiecki
    @JanuszL
    @sveta-levitan - we have the first draw. We want to share it after an internal review
    Simon Long
    @hobei
    @JanuszL Looking forward too it
    Michał Zientkiewicz
    @mzient
    ONNX data pipelines 1.0.0.0.pdf
    I've just sent the first version of the requirements. That title may be a bit of an overstatement, since there are many unknowns and this version is rather a starting point for the discussion than a guideline for implementing anything.
    Simon Long
    @hobei

    Thanks @mzient for the first draft, a couple of comments

    1. Can you please state your abbreviations for clarity. DL ?
    2. Do you have any thoughts on how the pipe line definition will be integrated into the model definition. And then how to link the pipline to a model input.
    3. For section 1.2 Option2. Are you considering a single data tensor and multiple shape tensors? Also did you consider an upper bound for the data tensors size to have evenly-size input data with padding?
    4. For section 2. do we need to consider other external sources, ftp, html etc. Should the onnx pipeline only start at the stream of 'in-memory-data' and not define how data is read from a source, such as a file system
    5. For the operations. Do you think we need randomize version of some of these like RandomizedCrop?
    6. Does your resample operation provide normalization of the input?
    7. I assume that the onnx converters would be extended to exporting for the 'data preparation' stage of existing frameworks. I think this should appear in the requirements. It would also be good to show a mapping table from common framework pre-processing option to the onnx definitions.
    8. Should new operations for pipelines be in the ai.Onnx / ai.Onnx-ML domain or should a new domain be defined?

    Looking forward to the first meeting.

    Michał Zientkiewicz
    @mzient
    Ad 1. DL = Deep Learning. The whole Background section was introduced quite recently, I'll need to fix it - and credit the author, which I forgot to do.
    Ad 2. That's the part I wish to discuss
    Ad 3. One flat data array (tensor) and one 2D tensor where one dimension is number of samples and the other is axis.
    e.g. two RGB images of size: 1920x1080, 720x576 would be represented by a flat data array of size 7464960 and a tensor:
    [ [ 1080, 1920, 3],
    [ 576, 720, 3 ] ]
    If it's unclear from the description, I'll add this example to the document.
    Ad 4. Readers (file system or data files) are part of the input pipeline. I think starting from "data in memory" may miss some important problems, e.g. difficult seeking in some record formats which imposes some restrictions on shuffling schemes. I don't know if we should consider data in non-native formats (e.g. embedded in HTML) or ordinary network resources (FTP, HTTP, etc). Network file systems are handled as ordinary files and some more specialized network resources may require custom readers, but when we have the very concept of a reader in place, they would just be yet another (custom) operator.
    Ad 5. I advocate separating generation of random parameters and applying them to the data. I'm aware this is not always possible and/or beneficial, as is the case e.g. in Single Shot Detector where crop window generation is driven by the object bounding boxes associated with given image.
    Ad 6. No, normalization is a separate operation - and it's more or less glorified multiply-add. I think ONNX already has such operators - unless you're talking about per-image normalization (mean and variance calculated from each image) - in which case additional operator is necessary, but since it involves a reduction step, I wouldn't fuse it with anything else.
    Ad 7. That may be quite hard, given the discrepancies between frameworks - supposedly same operation can give very different results in PyTorch vs TF vs MXNET, all of which use different image processing backend (respectively: PIL, custom TF, gluon)
    Ad 8. No opinion here yet.
    Simon Long
    @hobei
    @mzient thank you for the responses. I think we need to discuss comment 7 further in respect to the requirements for the pipelines. Should the converters export the pipeline definition? And conversely how a framework will import the pipeline definition? It may also be worth consulting the converter SIG.
    Michał Zientkiewicz
    @mzient
    It'd be ideal if the data pipeline was a part of the pipeline definition - I'm not sure, however, if it's even possible in all frameworks. Regarding the discrepancies - it might be quite hard to convince frameworks that their preprocessing operator is somehow "wrong" and adopt different semantics. Parameterizing operators so they can be coerced into behaving like a particular framework doesn't sound like a good idea, either. It serves the purpose of portability, but perpetuates models' dependence on framework idiosyncrasies.
    G. Ramalingam
    @gramalingam
    Hi: I am slightly confused by definition 1.1. What is a "tensor list" meant to capture or represent? It is unclear why it has the constraint that all these tensors have same element-type and number-of-dimension, so I am probably not understanding its context. My interpretation of a data-batch is that it represents a batch (with a batch-size) of inputs … and when the input consists of more than one tensor, we want the batch-size to be the same for all these input-tensors, but why should all the input-tensors have same element-type or number-of-dimension? I am also confused how this proposal interacts with the widely used convention of making batch another dimension of a tensor.
    Michał Zientkiewicz
    @mzient
    Hello @gramalingam ! I'll try to explain/elaborate.
    You said that you store a batch in a batch-sized tensor, but that's wasteful in many cases - consider unprocessed ImageNet - we can't pad all images there to maximum size, since images there vary from as small as 200x200 to 4k or even bigger.
    The tensor list is a represntation of a "batch tensor" where each sample can have different size but otherwise represents same kind of data (all samples represent images / volumes / audio recordings / spectrograms). It's reasonable to assume that each sample has same type/format and number of dimensions.
    G. Ramalingam
    @gramalingam
    @mzient : ok, I understand the usage, thanks for the explanation. I would suggest a type like "list<tensor<float>>" or "list<tensor<int>>" … which would mean a variable-length list of tensors of same element-type (but not necessarily same rank) … this is a more general type, and common usage scenarios like the one you mention might use same-rank tensors. We could restrict all elements to have same-rank … but would that be too restrictive in some cases? Alternatively, would it simplify the implementation to assume they are of the same rank or enable some optimization? I guess that would be a good reason to impose the same-rank restriction.
    Michał Zientkiewicz
    @mzient
    The elements of this list having same rank is essential for efficient implementation of batch processing. While even the different size case may be challenging in some respects (i.e. achieving reasonable work distribution on a GPU), it would be really daunting a task if we allow for different ranks. Ultimately, though, I think this case is of little utility - what could be gained by allowing different ranks? I can't imagine a use case for that, except perhaps the case of mixing single-channel and multi-channel data, which can be successfully handled by using degenerate tensors (size in channel dim = 1).
    Changming Sun
    @snnn
    Padding is not good. We need an async buffer operator. I just tried to implement such logic in my newly created onnxruntime sample: https://github.com/microsoft/onnxruntime/blob/master/samples/c_cxx/imagenet/async_ring_buffer.h
    Simon Long
    @hobei
    @snnn Can you elaborate on why padding is not good.
    G. Ramalingam
    @gramalingam
    @mzient : If we consider "Sequence" in ONNXML, it lets us capture both scenarios using appropriate types. For example, consider an input/intermediate-value/output X. We can say all of the following: (a) X is a list of 2-dimensional tensors of the same dimensions MxN if X has the type "Sequence<Tensor<float, [M,N]>". (b) X is a list of 2-dimensional tensors, each of potentially different sizes, if X has the type "Sequence<Tensor<float, [_, _]>" where I use underscore to denote an unknown shape-dimension (which ONNX lets us specify), (c) X is a list of potentially different ranks if X has the type "Sequence<Tensor<float, _>>" where I use "_" to indicate a tensor-type with no shape information.
    Michał Zientkiewicz
    @mzient
    @gramalingam The "Sequence" sounds a bit like a strongly typed array or vector - a sequence of tensors would then be much like DALI's TensorList. However, I could find only one usage of it - ZipMap seems to be using sequences of maps - but that's all I could find. Are there more uses of it anywhere?
    Simon Long
    @hobei

    @gramalingam I think that the '_' is not supported, instead you have to not set dim_value or dim_param

    Historical Notes: The following extensions were considered early on, but were never implemented or supported.
    The use of an empty string (as a dimension variable) to denote an unknown dimension not related to any other dimension. This was discarded in favor of using a Dimension with neither dim_value nor dim_param set.

    G. Ramalingam
    @gramalingam
    @hobei Thanks for clarifying this … I don't want to confuse people further by suggesting that we can use a special string to indicate unknown shape. I was using "_" only for this chat-message, since there is no simple way of showing "neither dim_value nor dim_param set" in our chat here. As an aside, I added the clarification historical note you quote above to IR.md :-) since there has been quite some confusion over this.
    G. Ramalingam
    @gramalingam
    @mzient : you are right that there are only a few uses of Sequence at this point. (I don't know the exact number.) It has not been used in ONNX. But there are ongoing discussions about using it more, for example to simplify exporting PyTorch into ONNX.
    Changming Sun
    @snnn
    I suggest avoiding using sequence. So far, only onnxruntime supports ONNX's non-tensor types. Even ONNX itself has two versions: with non-tensor type(like seq), or without. Whether the design is good or not, the truth is: it is not accepted widely.
    Spandan Tiwari
    @spandantiwari
    There are increasing number of use cases coming up when exporting PyTorch models to ONNX where Sequences would be needed. torch.split, torch.unbind are couple of examples use cases. Another area would be the models that require custom loops with a list of tensors is being manipulated within the loop. There are production PyTorch models using this.
    Changming Sun
    @snnn
    Are you saying, PyTorch can export this onnx model, but can't run the model exported from it?
    Spandan Tiwari
    @spandantiwari
    @snnn - I am not sure I understand your question exactly. Here' some clarification anyway. PyTorch does not export this model today because ONNX does not have Sequence (goal is to export to ONNX and not ONNX-ML). If you are asking about running the JITed model back in PyTorch, then that is possible today because PyTorch's IR supports Lists.
    Bowen Bao
    @BowenBao
    Onnx_Tensor_List_Discussion.pdf
    Adding to what Spandan has said, we have created a doc listing some of the use cases of Sequence/List from PyTorch. Tensor lists are widely used in PyTorch models, such as custom RNN models, and torch vision detection models.
    Spandan Tiwari
    @spandantiwari
    To circle back: ONNX now has two new datatypes, Sequence and Map, that were moved from ONNX-ML to ONNX (onnx/onnx#2244) . Also new ops related to Sequence have been added to ONNX (https://github.com/onnx/onnx/pull/2249).
    Michał Zientkiewicz
    @mzient
    Moving these types to ONNX is indeed great news!
    Changming Sun
    @snnn
    These types can't be serialized to model, and onnxruntime has very poor support to the non-tensor types. They hurt the performance, they break the shape inference. The seq/map type in ONNX is so complex that I can't find any good way to expose them through ONNXRuntime's C API.
    And we are trying to avoid using ZipMap operator in scikit-learn converter.
    Ryan Loney
    @ryan-nervana

    This is an announcement from the ONNX Foundation Working Group. We have created a poll to collect preferences from the ONNX community on what non-profit foundation we should join.

    Please make your voice heard by voting here: https://forms.gle/z4zEs2MGcQiV6tar9

    The options are Linux Foundation, Apache or Eclipse. Our selection and interview process is shared in the link above.

    Thank you,

    Jim Spohrer (IBM) and Ryan Loney (Intel)
    Co-chairs, ONNX Foundation WG

    Prasanth Pulavarthi
    @prasanthpul
    The next ONNX Community Workshop will be held on November 18 in Shanghai! If you are using ONNX in your services and applications, building software or hardware that supports ONNX, or contributing to ONNX, you should attend! This is a great opportunity to meet with and hear from people working with ONNX from many companies. You’ll also have opportunities to participate in technical breakout sessions. Due to limited space, please submit a proposal for a short talk if you would like to attend.
    Ryan Loney
    @ryan-nervana
    The ONNX Foundation WG put together a whitepaper for members of the ONNX community who would like to better understand the reasons for establishing a non-profit, vendor-neutral legal entity. You can read it here: https://github.com/onnx/working-groups/blob/master/foundation/whitepaper.md - any feedback is welcome. We are still collecting input through our poll and gitter.im/foundation channel.
    Zhenhao Li
    @Zhen-hao
    hi, can someone explain the concept of symbolic dimensions used in onnx runtime?