Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    vfdev
    @vfdev-5
    Hi there, I'm wondering about the issue ElementAI/baal#47 ("Support for Pytorch Lightning")
    Is there any other reasons to use it except "increasingly popular" and comparing, for example, to Ignite ? Thanks
    Another question on your roadmap, do you plan to add any research code for semantic segmentation task ?
    Frédéric Branchaud-Charron
    @Dref360

    1- The main reason is that we are not a "high-level framework API" and trying to add all the features that both ignite and lightning provides is futile. So we had to choose one and one of our collaborators uses PL. I will investigate what is required to run on ignite if it is not too much I will make a tutorial or extract some methods to make it easy to use. (I guess that BaaL can be easily used for both cases)

    2- yes! we will make the code available as soon as we publish our paper on BaaL and then we are working on a weakly-supervised segmentation paper which would also be released as soon as it is published.

    If you have any questions on semantic segmentation, feel free to ask here! We would be happy to help you :)

    vfdev
    @vfdev-5
    Thanks for the answer! Definitely, I would be interested in your weakly-supervised segmentation paper with Baysian AL and if it could be ported to ignite it would be more simple for me :)
    What kind of application you will cover in semantic segmentation BaaL : computer vision like PascalVOC or CityScapes; maybe arial/satellite imagery ISPRS etc ?
    If you could share some insights on how do you choose the most uncertain samples for semantic segmentation case...
    Frédéric Branchaud-Charron
    @Dref360

    We can probably provide some scripts with PascalVOC/COCO. When doing SatImagery, we have many optimization available to make this easier! Like storing predictions in float16, predicting on a portion of the pool, etc. We will make a guide for these usecases that are memory hungry.

    Right now, the state-of-the-art for Seg is to simply compute BALD or BatchBALD per pixel and take the mean per image. I know there are some techniques that will create "regions" or superpixels and compute the maximum of each regions before taking the mean. We have not tried these methods yet.

    vfdev
    @vfdev-5

    Sounds good !

    Right now, the state-of-the-art for Seg is to simply compute BALD or BatchBALD per pixel and take the mean per image.

    Yes, I had a chat with one of the authors of BatchBALD and it seems he wasn't too optimistic about how much there is to gain from active learning on image segmentation task... Anyway, I'm looking forward to test your implementation :)

    And if I can provide some help with a tutorial of BaaL with Ignite, do not hesitate to ask me.

    Frédéric Branchaud-Charron
    @Dref360

    Hey @vfdev-5
    I think I'm done, but I would need your help for review.

    Basically, I'm making a new engine that will perform Monte-Carlo sampling and then when we are done with training, we perform the active step (prediction, heuristic, labelling).

    What do you think?
    https://gist.github.com/Dref360/c9c81b263ac9647b5f7c3ae5ff6706c2

    vfdev
    @vfdev-5
    Hey Frédéric @Dref360 , the example looks good. However I can not run the code, because of mc_inference is absent in baal.modelwrapper. Another point I would like understand, in the code you use loss as torch.nn.NLLL but the model seems to output logits ?
    Frédéric Branchaud-Charron
    @Dref360

    Oh my bad I took the example from Ignite without thinking.
    You're right with mc_inference it is in the branch pytorch_lightning, but should be merged quickly.
    https://github.com/ElementAI/baal/blob/pytorch_lightning/src/baal/modelwrapper.py#L419

    If we can extract all the utils necessary for both framework and make proper tutorial for both I think this can be very valuable!
    I updated the gist with what is lacking. But they are true for PL as well.

    vfdev
    @vfdev-5

    Thanks for the update. Yes, I found mc_inference in PL branch.

    If we can extract all the utils necessary for both framework and make proper tutorial for both I think this can be very valuable!

    sounds good !

    vfdev
    @vfdev-5
    @Dref360 I'm trying to understand your code and baal library and I have the following question: why do you have 2 heuristics modules: numpy and GPU with torch ?
    Can't it be a single one with torch ?
    Frédéric Branchaud-Charron
    @Dref360
    In the beginning, we wanted to be framework agnostic, but this seems less and less possible with all the features that we add.
    We do have a tutorial with Sklearn that should work without Pytorch.
    vfdev
    @vfdev-5
    Sounds reasonable. Hope that both modules are compatible and have the same API.
    vfdev
    @vfdev-5
    @Dref360 why there is a difference between 2 examples: pl and ignite. In pl there is ActiveLearningLoop which is absent in ignite ?
    Frédéric Branchaud-Charron
    @Dref360
    They are equivalent, the ActiveLearningLoop worked with a Dataset, not a DataLoader. We will rework on this API. Maybe switch everything to DataLoader as ignite and Lightning do.
    vfdev
    @vfdev-5
    @Dref360 what would be qualifying metrics for AL for this example ? I run the code but as other users I have no idea if active learning worked or not...
    Frédéric Branchaud-Charron
    @Dref360

    Good point! This should be more explicit.

    In the litterature, people compare the loss/precision/recall between random selection and their method for each "dataset size"

    vfdev
    @vfdev-5
    Thanks. Maybe I can add computing precision/recall on test dataset while AL on training part only. Anyway, I'm experimenting here : https://colab.research.google.com/drive/1tSBDDzgPlhRKb2sU3WT36L06_NO0PP_v
    Frédéric Branchaud-Charron
    @Dref360
    Awesome! We do have a metric called ClassificationReport which computes COnfusionMatrix, precision, recall for each class :) We tend to use that in our research
    vfdev
    @vfdev-5
    We have also ConfusionMatrix in Ignite (https://github.com/pytorch/ignite/blob/master/ignite/metrics/confusion_matrix.py#L13), but if you could provide the link on the code of your ClassificationReport, we could create the same in Ignite (and everyone could benefit)
    Frédéric Branchaud-Charron
    @Dref360

    Sorry for the late answer, I'm really busy with our NeurIPS submission.

    One emtric I think would be useful to Pytorch Ignite is ECE, the Expected Calibration Error.
    We do have an implementation here:
    https://github.com/ElementAI/baal/blob/master/src/baal/utils/metrics.py#L88

    vfdev
    @vfdev-5
    Thanks for the link !
    Frédéric Branchaud-Charron
    @Dref360

    I'm trying to build a FAQ. Here is somethign that can be useful to both of you.

    How to predict uncertainty per sample in a dataset @salman1993 @luistelmocosta

    model = YourModel()
    dataset = YourDataset()
    wrapper = ModelWrapper(model, criterion=None)
    
    heuristic = BALD()
    
    uncertainty = heuristic.get_uncertainties(wrapper.predict_on_dataset(dataset, batch_size=32, iterations=20, use_cuda=True)

    If memory is an issue, you can use both get_uncertainties_generator and predict_on_dataset_generator.

    Frédéric Branchaud-Charron
    @Dref360
    You can also compute the Expected Calibration Error (ECE) per class: https://github.com/ElementAI/baal/blob/master/src/baal/utils/metrics.py#L167
    Salman Mohammed
    @salman1993
    Thanks @Dref360
    Frédéric Branchaud-Charron
    @Dref360
    FYI, the first version of our FAQ is live!
    https://baal.readthedocs.io/en/latest/faq.html
    Frédéric Branchaud-Charron
    @Dref360

    Hello, we have an issue with the recently merged Pytorch Lightning API. In particular with the partial uncertainty sampling (with the max_sample argument).
    I would like to get the input of the community.

    In PL, it is the responsability of the user to make the DataLoaders. To stay in line with this design, we let the user define pool_loaderwhich defines the pool dataloader. If the users wants to do partial uncertainty sampling, he can make a Subset of the pool there. The issue is to recover this set of index.

    Possibilities:

    • As @ThierryJudge proposed, drop pool_loader and use the original ALLoop.step method.
    • override the Dataloader sampler in BaalTrainer.
    • Have an object keep the state of the indices that the user would use if needed. ( We can add checks to be sure that the user does not misuse this by mistake).

    I'm open to any other suggestions! :)

    9 replies
    Frédéric Branchaud-Charron
    @Dref360
    We need to add a small example on how to use baal for semantic segmentation. We also need to open-source the region-based BALD heuristic :) (bald per pixel aggregated by regions)
    3 replies
    Frédéric Branchaud-Charron
    @Dref360

    If someone wants to know more about our Pytorch Lightning integration, this is a nice bug to look at and know more :P

    ElementAI/baal#86