Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 00:44
    michaelamoako synchronize #787
  • Jun 20 15:59
    pritishaw1 commented #814
  • Jun 20 15:14
    adrinjalali commented #814
  • Jun 20 15:05
    pritishaw1 commented #814
  • Jun 20 13:40
    kurianbenoy opened #871
  • Jun 20 13:36
    adrinjalali commented #814
  • Jun 20 13:30
    pritishaw1 commented #814
  • Jun 20 12:29
    kurianbenoy commented #764
  • Jun 20 12:11
    galuhsahid commented #784
  • Jun 20 11:58
    adrinjalali commented #764
  • Jun 20 11:54
    adrinjalali commented #814
  • Jun 19 18:22
    kurianbenoy commented #764
  • Jun 19 17:39
    kurianbenoy edited #870
  • Jun 19 17:35
    kurianbenoy opened #870
  • Jun 19 16:26
    pritishaw1 commented #814
  • Jun 19 16:17
    kurianbenoy commented #764
  • Jun 19 16:16
    kurianbenoy commented #764
  • Jun 19 16:16
    kurianbenoy commented #764
  • Jun 19 16:07
    kurianbenoy commented #764
  • Jun 19 15:56
    kurianbenoy commented #764
Roman Lutz
@romanlutz
YES!!!! In fact I was going to open issues for this. If you have any suggestions right away please open issues. I'm sure you'll get to it faster than me.
Sorry, finding it hard to contain my excitement. That's one of my all-time favorite papers :-)
1 reply
Hilde Weerts
@hildeweerts
I think whether the usage of a sensitive attribute is problematic would really depend on the use case, no? E.g. let’s say you were using income predictions to decide who should get extra help in social services or whatever ( and for some reason 50k is the perfect threshold??? 😂 ). We know that for income gender is sadly sometimes even causally related... In that case I wouldn’t be as opposed to using it as in, say, deciding whether somebody should get a loan or not. and removing the sensitive feature does not necessarily make things better (eg a proxy here could be occupation).
15 replies
But I suppose pointing out some of these problems would make sense
Roman Lutz
@romanlutz
This is the promised community call platform discussion with all the information I've gathered so far: https://github.com/fairlearn/fairlearn/discussions/777 Please do chime in since we want a solution that works for everyone!
Nils
@shimst3r:matrix.org
[m]

The biggest issue I see in this generic approach is, well, that it’s generic. 😅 If we go down this path, it emphasizes that fairness can be seen as an intrinsic problem of the model, rather than being a sociotechnical one.

On the other hand, it’s a lot better than using an inappropriate data set. So it might be a good alternative until we find a better one.

What do you think, does this make any sense?

1 reply
Nils Müller
@shimst3r:matrix.org
[m]

I’d be okay with gender queer as a term, as it makes clear the intention is to represent people outside a binary gender scheme. I can’t speak for all non-binary though. 😬

@Roman that’s actually a good idea. Without having read the original papers, I wouldn’t have understood it either. Having a approachable theoretical section is a unique selling point for the fairlearn documentation I guess.

2 replies
MiroDudik
@MiroDudik

Re. collecting gender demographics, I have recently chatted with Queer in AI folks, and they suggest the following:

Gender: (please select the options which are applicable to you)
[] Woman
[] Man
[] Non-binary / Genderqueer / Third gender
[] Genderfluid / Gender non-conforming
[] Questioning
[] Prefer not to say
[] Specify your own (open text box)

The key aspect is that this is a multi-select. When this kind of data is collected, there should be an explanation that the data is collected to audit for fairness and/or mitigate any fairness issues, and that if you pick "prefer not to say" you will not be included in any fairness audits.

I also like these two sources:
Gender (apa.org)
Respectful Collection of Demographic Data | by Sarai Rosenberg | Managing on the Margins | Medium

10 replies
Roman Lutz
@romanlutz
FYI folks we have a Twitter account now that's actually used! https://twitter.com/fairlearn/status/1390719811592753161?s=20
Release v0.6.2 just went out. The main highlight is the bugfix on mitigation with multiple sensitive features. If you spot any issues please file an them on GitHub :-) Have a great weekend!
3 replies
Nils
@shimst3r
I have thought about two things regarding podcasts I wanted to share with you: 1. a good way to reach a bigger audience in the Python ecosystem is the Talk Python podcast (https://talkpython.fm/home). Have you thought about this already? 2. I wanted to do a podcast myself for some time now. I think fairlearn and its scientific framework could work as an interesting topic: maybe discussing related papers and case studies?
3 replies
Roman Lutz
@romanlutz
If you want to join the "general" sprint at PyCon on the 17th and 18th of this month and work on Fairlearn: https://discord.gg/X9D5V5Z9kY
I'll be there both days. I believe they're planning to set up channels for each project (incl. one for Fairlearn).
Hilde Weerts
@hildeweerts
Oh no it happened again xD sorry for missing the meeting, I was preparing our stairs for a new layer of paint lol. There don't seem to be new notes for the meeting in the hackmd - or did we start in a new one?
3 replies
vincent d warmerdam
@koaning
Last time I toyed around in fairlearn there wasn't a MetricFrame yet ... gotta say ... I like it!
8 replies
Roman Lutz
@romanlutz
If anyone participating in the PyCon sprints ends up here, you're in the right place! This is the Fairlearn community's Gitter channel. Feel free to post questions here, this is very informal. If you're looking for the Discord channels for the PyCon sprints you can find them in my sprint instructions: https://github.com/romanlutz/fairlearn-pycon-sprint/blob/main/README.md
I'm also available right now for the next 3.5h (until 1pm Pacific Time) if you want to discuss anything or pair program etc.
24 replies
Thng Kai Ting Brenda
@breadplop_twitter
image.png
15 replies
Thng Kai Ting Brenda
@breadplop_twitter
8 replies
vincent d warmerdam
@koaning
Hi folks, I'm likely to make a set of calmcode videos on fairlearn this weekend. Just to check, as long as I am using keyword arguments can I assume that the API for MetricFrame, make_derived_metric and ExponentiatedGradient won't change?
4 replies
Naveen Kumar
@kumarnav03
Hi Everyone , Is there any example/information how we can mitigate bias for a multicategory feature or how we can extend ThresholdOptimizer & ExponentiatedGradient?
Roman Lutz
@romanlutz
We don't have any examples on this yet. The only related issue is fairlearn/fairlearn#752 but I think you already commented there.
vicohart
@vicohart
Hi! I got two different times for the meeting tomorrow and would like to confirm. Is it 11 a.m. EDT?
5 replies
Kate Stohr
@kstohr
@vicohart ok... just getting rolling, here:
This is the ticket I am working on: fairlearn/fairlearn#758
Kiana Alikhademi
@kalikhademi
Hi everyone, I just joined this community and I would love to help fairlearn in any way I can.
1 reply
Kate Stohr
@kstohr
@romanlutz Quick q... what's the naming convention here around
'y_test' vs. 'y_true'? Similarly what does 'y_pred' refer to? clf.predict_proba() or clf.predict() I ask because I am seeing them used in different ways and/or interchangeably. Just want to be clear how this group standardizes around common ml conventions ...
Roman Lutz
@romanlutz
We use y_true. pred is for predictions
Interchangeably? Where's that?
We follow sklearn conventions
Kate Stohr
@kstohr
Ok what do you use for clf.predict_proba()?
Kiana Alikhademi
@kalikhademi
@kstohr the clf.predict_prob is to get the probabilities for each class from the classifier
Kurian Benoy
@kurianbenoy
@romanlutz for issue #764, can we go ahead with the simple approach which is right now followed in scikit-documentation and then later have it like the pytorch documentation. I am slightly confused on how to go ahead with issue
Kurian Benoy
@kurianbenoy

In scikit repo conf.py which you pointed in description of the issue:
https://github.com/scikit-learn/scikit-learn/blob/309f135c3284d7db6e23ca81a87948c7066a3949/doc/conf.py#L338.

I am confused what should we put for in the case of branch and dependencies - requirements.txt. If you look at scikit-repo requirements - https://github.com/scikit-learn/scikit-learn/blob/309f135c3284d7db6e23ca81a87948c7066a3949/doc/binder/requirements.txt, it's confusing.

What should we put in the requirements.txt in case of fairlearn repository?

2 replies
Roman Lutz
@romanlutz
@kstohr @kalikhademi That's right, whenever you have a classifier that is able to produce probabilities you use predict_proba. Note that our meta estimators like ExponentiatedGradient do NOT produce probabilities the same way, so we've very consciously chosen not to name it predict_proba.
@kurianbenoy let's keep this discussion in the issue. I'll respond there.
Roman Lutz
@romanlutz
For those of you interested in trying out Discord with us, this is the invite link: https://discord.gg/sjUVwDvy
This will allow us to do things like
  • hanging out!
  • pair programming and ride-alongs
    quite easily. It's worth noting that this doesn't mean this Gitter channel isn't active anymore. We're just trying out Discord for now. Feedback is more than welcome!
Roman Lutz
@romanlutz
Seems like Discord invite links can expire... (by default at least) so I created one that won't: https://discord.gg/hgPnSagHUv
If it's used by trolls I can still deactivate it, though. But for now this one should work and the above one may not work for you
Arpan Mahapatra
@ArpanSM
Has anyone implemented regression task using fairlearn? If so, could you please share any sample notebook or snippet?
Roman Lutz
@romanlutz
@ArpanSM are you looking for assessment or mitigation?
Arpan Mahapatra
@ArpanSM
@romanlutz I'm looking for both of them.
Roman Lutz
@romanlutz

Assessment: our user guide covers both classification and regression metrics https://fairlearn.org/main/user_guide/assessment.html#metrics
Mitigation: The table shows which techniques should work for regression: https://fairlearn.org/main/user_guide/mitigation.html (everything but ThresholdOptimizer)
Admittedly, the user guide isn't super comprehensive yet and has some gaps, but there is a regression section for fairness constraints that can be plugged into our fairlearn.reductions techniques: https://fairlearn.org/main/user_guide/mitigation.html#fairness-constraints-for-regression
We don't have examples with regression mitigation yet, but this test case code might be useful to you: https://github.com/fairlearn/fairlearn/blob/62fc80c77bcd3bef6a3d7bc44e54827ec9fb8d09/test/unit/reductions/grid_search/test_grid_search_regression.py#L64

If you're willing to explain your use case we could try to advise on how to use Fairlearn. Let us know!

Arpan Mahapatra
@ArpanSM
@romanlutz Thank you for the resources. My task is to determine "fitment %" which is a regression task based on many factors such as education, experience and also sensitive factors such as age, gender and language .
I have been asked to predict based on taking all these factors into account and then remove bias for anyone sensitive elements such as age or gender. I was hoping to use this fairlearn to analyze and correct such bias.
Arpan Mahapatra
@ArpanSM
I have to make sure that the mentioned bias does not produce any inequality while predicting and make it fair for everyone for the bias-free opportunity.
Hilde Weerts
@hildeweerts
Hi @ArpanSM! What does "fitment %" mean? Generally speaking, the way unfairness can be assessed/mitigated highly depends on the specific context of your application. A good way to get started might be to think further about what exactly you mean with "bias" in your scenario. At Fairlearn, we usually like to think in terms of specific fairness-related harms, rather than (more abstract) "bias". This section of the user guide might be helpful for you to think this through: https://fairlearn.org/main/user_guide/fairness_in_machine_learning.html
Adrin Jalali
@adrinjalali
In a single day, we ended up having to use a descriptor (https://docs.python.org/3/howto/descriptor.html), simpler than metaclass customisation of class creation (https://www.python.org/dev/peps/pep-0487/#requiring-an-explicit-decorator-on-init-subclass), and dynamically modifying a function/method's signature (https://www.python.org/dev/peps/pep-0362/). It's like doing wizard stuff and it was all new to me, I need a break lol
1 reply
MiroDudik
@MiroDudik
anybody knows how to build our website locally without the gallery? (it takes too long and i don't need it to check the docs build.)
5 replies
Hilde Weerts
@hildeweerts
I have to say the lack of a threads feature on discord makes the conversations become quite chaotic fairly quickly... @shimst3r noted that discord is actively working on this (https://discord.com/developers/docs/topics/threads) but I do feel like it's a real downside of the platform
3 replies
Roman Lutz
@romanlutz
Your periodic reminder that lots of people are now using our Discord for conversations about papers, the website, etc., so if you like please join us there: https://discord.gg/hgPnSagHUv