Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Dec 05 13:41
    riedgar-ms commented #1162
  • Dec 05 13:37
    riedgar-ms commented #1125
  • Dec 05 13:31
    riedgar-ms synchronize #1125
  • Dec 03 02:46
    romanlutz synchronize #614
  • Dec 03 02:42
    romanlutz synchronize #614
  • Dec 02 21:13
    romanlutz closed #935
  • Dec 02 21:13
    romanlutz commented #935
  • Dec 02 21:10
    romanlutz demilestoned #232
  • Dec 02 21:10
    romanlutz demilestoned #96
  • Dec 02 21:03
    romanlutz unlabeled #991
  • Dec 02 21:02
    romanlutz closed #1068
  • Dec 02 21:01
    romanlutz commented #1101
  • Dec 02 12:17
    romanlutz commented #827
  • Dec 02 12:14
    romanlutz commented #908
  • Dec 01 20:01
    MiroDudik commented #999
  • Dec 01 19:54
    MiroDudik commented #999
  • Dec 01 16:12
    romanlutz commented #614
  • Dec 01 08:39
    hildeweerts closed #1161
  • Dec 01 08:39
    hildeweerts commented #1161
  • Nov 30 23:32
    MiroDudik commented #999
Nils
@shimst3r
I have thought about two things regarding podcasts I wanted to share with you: 1. a good way to reach a bigger audience in the Python ecosystem is the Talk Python podcast (https://talkpython.fm/home). Have you thought about this already? 2. I wanted to do a podcast myself for some time now. I think fairlearn and its scientific framework could work as an interesting topic: maybe discussing related papers and case studies?
3 replies
Roman Lutz
@romanlutz
If you want to join the "general" sprint at PyCon on the 17th and 18th of this month and work on Fairlearn: https://discord.gg/X9D5V5Z9kY
I'll be there both days. I believe they're planning to set up channels for each project (incl. one for Fairlearn).
Hilde Weerts
@hildeweerts
Oh no it happened again xD sorry for missing the meeting, I was preparing our stairs for a new layer of paint lol. There don't seem to be new notes for the meeting in the hackmd - or did we start in a new one?
3 replies
vincent d warmerdam
@koaning
Last time I toyed around in fairlearn there wasn't a MetricFrame yet ... gotta say ... I like it!
8 replies
Roman Lutz
@romanlutz
If anyone participating in the PyCon sprints ends up here, you're in the right place! This is the Fairlearn community's Gitter channel. Feel free to post questions here, this is very informal. If you're looking for the Discord channels for the PyCon sprints you can find them in my sprint instructions: https://github.com/romanlutz/fairlearn-pycon-sprint/blob/main/README.md
I'm also available right now for the next 3.5h (until 1pm Pacific Time) if you want to discuss anything or pair program etc.
24 replies
Thng Kai Ting Brenda
@breadplop_twitter
image.png
15 replies
Thng Kai Ting Brenda
@breadplop_twitter
8 replies
vincent d warmerdam
@koaning
Hi folks, I'm likely to make a set of calmcode videos on fairlearn this weekend. Just to check, as long as I am using keyword arguments can I assume that the API for MetricFrame, make_derived_metric and ExponentiatedGradient won't change?
4 replies
Naveen Kumar
@kumarnav03
Hi Everyone , Is there any example/information how we can mitigate bias for a multicategory feature or how we can extend ThresholdOptimizer & ExponentiatedGradient?
Roman Lutz
@romanlutz
We don't have any examples on this yet. The only related issue is fairlearn/fairlearn#752 but I think you already commented there.
vicohart
@vicohart
Hi! I got two different times for the meeting tomorrow and would like to confirm. Is it 11 a.m. EDT?
5 replies
Kate Stohr
@kstohr
@vicohart ok... just getting rolling, here:
This is the ticket I am working on: fairlearn/fairlearn#758
Kiana Alikhademi
@kalikhademi
Hi everyone, I just joined this community and I would love to help fairlearn in any way I can.
1 reply
Kate Stohr
@kstohr
@romanlutz Quick q... what's the naming convention here around
'y_test' vs. 'y_true'? Similarly what does 'y_pred' refer to? clf.predict_proba() or clf.predict() I ask because I am seeing them used in different ways and/or interchangeably. Just want to be clear how this group standardizes around common ml conventions ...
Roman Lutz
@romanlutz
We use y_true. pred is for predictions
Interchangeably? Where's that?
We follow sklearn conventions
Kate Stohr
@kstohr
Ok what do you use for clf.predict_proba()?
Kiana Alikhademi
@kalikhademi
@kstohr the clf.predict_prob is to get the probabilities for each class from the classifier
Kurian Benoy
@kurianbenoy
@romanlutz for issue #764, can we go ahead with the simple approach which is right now followed in scikit-documentation and then later have it like the pytorch documentation. I am slightly confused on how to go ahead with issue
Kurian Benoy
@kurianbenoy

In scikit repo conf.py which you pointed in description of the issue:
https://github.com/scikit-learn/scikit-learn/blob/309f135c3284d7db6e23ca81a87948c7066a3949/doc/conf.py#L338.

I am confused what should we put for in the case of branch and dependencies - requirements.txt. If you look at scikit-repo requirements - https://github.com/scikit-learn/scikit-learn/blob/309f135c3284d7db6e23ca81a87948c7066a3949/doc/binder/requirements.txt, it's confusing.

What should we put in the requirements.txt in case of fairlearn repository?

2 replies
Roman Lutz
@romanlutz
@kstohr @kalikhademi That's right, whenever you have a classifier that is able to produce probabilities you use predict_proba. Note that our meta estimators like ExponentiatedGradient do NOT produce probabilities the same way, so we've very consciously chosen not to name it predict_proba.
@kurianbenoy let's keep this discussion in the issue. I'll respond there.
Roman Lutz
@romanlutz
For those of you interested in trying out Discord with us, this is the invite link: https://discord.gg/sjUVwDvy
This will allow us to do things like
  • hanging out!
  • pair programming and ride-alongs
    quite easily. It's worth noting that this doesn't mean this Gitter channel isn't active anymore. We're just trying out Discord for now. Feedback is more than welcome!
Roman Lutz
@romanlutz
Seems like Discord invite links can expire... (by default at least) so I created one that won't: https://discord.gg/hgPnSagHUv
If it's used by trolls I can still deactivate it, though. But for now this one should work and the above one may not work for you
Arpan Mahapatra
@ArpanSM
Has anyone implemented regression task using fairlearn? If so, could you please share any sample notebook or snippet?
Roman Lutz
@romanlutz
@ArpanSM are you looking for assessment or mitigation?
Arpan Mahapatra
@ArpanSM
@romanlutz I'm looking for both of them.
Roman Lutz
@romanlutz

Assessment: our user guide covers both classification and regression metrics https://fairlearn.org/main/user_guide/assessment.html#metrics
Mitigation: The table shows which techniques should work for regression: https://fairlearn.org/main/user_guide/mitigation.html (everything but ThresholdOptimizer)
Admittedly, the user guide isn't super comprehensive yet and has some gaps, but there is a regression section for fairness constraints that can be plugged into our fairlearn.reductions techniques: https://fairlearn.org/main/user_guide/mitigation.html#fairness-constraints-for-regression
We don't have examples with regression mitigation yet, but this test case code might be useful to you: https://github.com/fairlearn/fairlearn/blob/62fc80c77bcd3bef6a3d7bc44e54827ec9fb8d09/test/unit/reductions/grid_search/test_grid_search_regression.py#L64

If you're willing to explain your use case we could try to advise on how to use Fairlearn. Let us know!

Arpan Mahapatra
@ArpanSM
@romanlutz Thank you for the resources. My task is to determine "fitment %" which is a regression task based on many factors such as education, experience and also sensitive factors such as age, gender and language .
I have been asked to predict based on taking all these factors into account and then remove bias for anyone sensitive elements such as age or gender. I was hoping to use this fairlearn to analyze and correct such bias.
Arpan Mahapatra
@ArpanSM
I have to make sure that the mentioned bias does not produce any inequality while predicting and make it fair for everyone for the bias-free opportunity.
Hilde Weerts
@hildeweerts
Hi @ArpanSM! What does "fitment %" mean? Generally speaking, the way unfairness can be assessed/mitigated highly depends on the specific context of your application. A good way to get started might be to think further about what exactly you mean with "bias" in your scenario. At Fairlearn, we usually like to think in terms of specific fairness-related harms, rather than (more abstract) "bias". This section of the user guide might be helpful for you to think this through: https://fairlearn.org/main/user_guide/fairness_in_machine_learning.html
Adrin Jalali
@adrinjalali
In a single day, we ended up having to use a descriptor (https://docs.python.org/3/howto/descriptor.html), simpler than metaclass customisation of class creation (https://www.python.org/dev/peps/pep-0487/#requiring-an-explicit-decorator-on-init-subclass), and dynamically modifying a function/method's signature (https://www.python.org/dev/peps/pep-0362/). It's like doing wizard stuff and it was all new to me, I need a break lol
1 reply
MiroDudik
@MiroDudik
anybody knows how to build our website locally without the gallery? (it takes too long and i don't need it to check the docs build.)
5 replies
Hilde Weerts
@hildeweerts
I have to say the lack of a threads feature on discord makes the conversations become quite chaotic fairly quickly... @shimst3r noted that discord is actively working on this (https://discord.com/developers/docs/topics/threads) but I do feel like it's a real downside of the platform
3 replies
Roman Lutz
@romanlutz
Your periodic reminder that lots of people are now using our Discord for conversations about papers, the website, etc., so if you like please join us there: https://discord.gg/hgPnSagHUv
vincent d warmerdam
@koaning
Just checking in. I saw that there's 1000 stars on Github which prompted me to check in again. I still want to make calmcode content on fairlearn, but I recall that there may be an API change to change metric= to metrics= in MetricFrame. Is this discussion still on-going? I see the thread is still active here fairlearn/fairlearn#756
Roman Lutz
@romanlutz
@koaning that particular part with changing the arg name already happened (check v0.6.2 vs. v0.7.0 in https://fairlearn.org/v0.7.0/api_reference/fairlearn.metrics.html) but the conversation in #756 is a bit broader and still ongoing. Depending on how that conversation goes it may or may not change significantly. @adrinjalali made a valid point about not overcomplicating existing pieces (https://github.com/fairlearn/fairlearn/issues/756#issuecomment-877069394). If we're going with that MetricFrame would stay as is.
@mmadaio @MiroDudik @hildeweerts @riedgar-ms @adrinjalali should we discuss this topic on the Thursday call?
Hilde Weerts
@hildeweerts
Good idea, @romanlutz. Especially now that @riedgar-ms made a nice summary i think people should be able to chime in more easily
vincent d warmerdam
@koaning
Any conclusions :) ?
2 replies
vincent d warmerdam
@koaning
I'll be making a calmcode course on fairlearn soon, so I figured I'd ask ... are there common points of confusion that would be nice to address?
Roman Lutz
@romanlutz

I'll be making a calmcode course on fairlearn soon, so I figured I'd ask ... are there common points of confusion that would be nice to address?

Some that come to mind:

  • using mitigation techniques does not mean you don't need standard things anymore (such as cross validation), you still do! It's just a wrapper...
  • lots of confusion around which fairness criteria matter in which cases
  • most mitigation techniques produce probabilistic models in the end without deterministic outputs, which is VERY surprising to some users
  • getting a fairness criterion to the best value (e.g. 0) does not mean that a model is "fair" (deployment time population may differ, and we'd need to look much more broadly than just technical aspects like the model outputs)
vincent d warmerdam
@koaning
Thanks!
vincent d warmerdam
@koaning
Some folks might be interested in this PhD position https://www.cwi.nl/jobs/vacancies/900774
amahenthiran
@amahenthiran
Hi there, a colleague of mine and I were wondering if the monthly developer calls were still happening?
Adrin Jalali
@adrinjalali
@amahenthiran they're done weekly and on discord. You can join the server using the invitation link on our main page: https://fairlearn.org/