fairlearn/community

fairlearn.org - everything related to the Fairlearn package. Please feel free to ask any kinds of questions!

Richard Edgar

@riedgar-ms

Happy New Year to all.... a PR to get our builds working again:
fairlearn/fairlearn#669

1 reply

Richard Edgar

@riedgar-ms

One more PR to fix a problem with the TensorFlow tests:
fairlearn/fairlearn#671

Richard Edgar

@riedgar-ms

And another build break:
fairlearn/fairlearn#675
This was caused by a flake8 rule updating. Now I look at the code, I think that one of these is actually a legitimate complaint - why replace the actual exception by a ValueError?

4 replies

trivenigandhi

@trivenigandhi

Hi everyone! I am so excited to come across the fairlearn community and I am eager to help out however I can. I was thinking about an example notebook using publicly available health data, but I would also be happy to work through any simple code base issues...please let me know where I can best direct my energy!

8 replies

vincent d warmerdam

@koaning

So ... not to start a rant. But! Anybody seen this announcement

https://www.bcg.com/press/12january2021-facet-open-source-library-for-human-explainable-artificial-intelligence

Quote

BCG GAMMA FACET Helps Human Operators Understand Advanced Machine Learning Models So They Can Make Better and More Ethical Decisions

Their guide meanwhile uses the load_boston dataset to explain the tool without acknowledging the B column.

Feel free to pitch in BCG-Gamma/facet#202

Also, can anybody confirm if the scikit-learn team is actually contributing to this project? It feels strange that this didn't get flagged before.

vincent d warmerdam

@koaning

The original announcement suggests a collaboration, but it feels strange since I thought it was common knowledge what the issues with load_boston were.

Hilde Weerts

@hildeweerts

Good to see they're immediately taking action :D

trivenigandhi

@trivenigandhi

That's disappointing to see, but good they responded quickly. I think it only highlights the need for people to have the social context before diving into solutioning.

Roman Lutz

@romanlutz

Indeed... I think we need to write that user guide section about the Boston dataset so that we can point them to something more structured :-)

vincent d warmerdam

@koaning

That's disappointing to see, but good they responded quickly. I think it only highlights the need for people to have the social context before diving into solutioning.

This is especially true when strategy consultancies are marketing these tools. Especially if it is at global scale.

vincent d warmerdam

@koaning

Also, just sayin' ... if folks are interested in speaking about fairlearn this year. Scipy talks are open for submission https://www.scipy2021.scipy.org/talk-poster-presentations

trivenigandhi

@trivenigandhi

Hi all! I am working in a community edition of Pycharm. Any suggestions on how to render the FairLearnDashboard without a Jupyter notebook?

Roman Lutz

@romanlutz

@trivenigandhi that's not supported. The dashboard is moving to another package (called raiwidgets, to be released at the end of this month) and there's no further work planned before we remove it from Fairlearn. We are adding matplotlib-based plots, though, but only in the next version.

Manojit Nandi

@LeJit

Here's the PyCon talk proposal I presented in today's community call for those of you who want to take a closer look at it: https://github.com/fairlearn/fairlearn/discussions/674#discussioncomment-300005

MiroDudik

@MiroDudik

@hildeweerts , @trivenigandhi , @romanlutz : potato!

3 replies

trivenigandhi

@trivenigandhi

Following up on the same threat as @LeJit here is the link to the SciPy proposal! Looking forward to your feedback! https://github.com/fairlearn/fairlearn/discussions/674#discussioncomment-306760

André Cruz

@AndreFCruz

hey all! I'm having an error when fitting an Exponentiated Gradient with FPR constraint

ValueError: Phase 1 of the simplex method failed to find a feasible solution. (...) Consider increasing the tolerance to be greater than 3.9e-01. (...)

;
Any idea which parameter I should change if I want to use the classifier with the solution that was found as is ? Is it EG's nu ? Or the constraint's ratio_bound_slack ?

5 replies

vincent d warmerdam

@koaning

To anybody who is bored. I've found another global consultancy company that didn't look at the contents of the load_boston dataset.

Upvotes appreciated. quantumblacklabs/causalnex#91

17 replies

vincent d warmerdam

@koaning

@romanlutz crazy idea. But would it perhaps be an idea to build a github bot that automatically warns repos of the load_boston dataset?

Adrin Jalali

@adrinjalali

I mean, they should already be getting a warning that it's deprecated I think

vincent d warmerdam

@koaning

True, but not everyone tests their docs.

I just checked, there's like 30K+ repos on github that uses load_boston in the docs/tests/script.

vincent d warmerdam

@koaning

Do we know any Belgian journalists in this group? I may have found something that deserves investigation.
https://traicie.com/

9 replies

Richard Edgar

@riedgar-ms

Can I ping for a second review for:
fairlearn/fairlearn#689

trivenigandhi

@trivenigandhi

@romanlutz @MiroDudik @hildeweerts @LisaIbanez let me know if you would like access to the scipy proposal! I posted the link on the github discussion and manojit already has access

Roman Lutz

@romanlutz

FYI this week Prof. Steven Wu from CMU is speaking about “Involving Stakeholders in Building Fair ML Systems" at the TrustML community:
https://groups.google.com/g/trustworthyml/c/GP2YHDOEG2g/m/erQq7FlZEAAJ

Richard Edgar

@riedgar-ms

I'm looking to get the v0.6.0 release rolling (starting tomorrow, I think) - reply to this if there's some critical bugfix you think we're missing. The release process will likely result in some PRs that get put out and merged without the usual 24 hour opportunity for comment. Tagging @adrinjalali @hildeweerts @romanlutz @MiroDudik in particular.

15 replies

Frédéric Branchaud-Charron

@Dref360

Hello, I had a question on the different types of biases in datasets. I couldn't find much of it online.
So far, my team and I explored two types of biases:

There is a sensitive feature that correlates with y but it would be unfair to use it for making the prediction. Even in the infinite amount of data and a perfect predictor, it would still use that feature for making the prediction and it would be unfair.
The minority class does not correlate with y, but there is much less samples for training the model in that subspace (since the distribution is uneven). In the limit of infinite data, this is no longer a problem, you converge to the optimal classifier and this variable was not a sensitive variable.

Are you aware of other types of biases in datasets?
Thanks for your help!

4 replies

Richard Edgar

@riedgar-ms

Another small update.... Python 3.9 is starting to look promising:
fairlearn/fairlearn#646
Unfortunately, this doesn't cover all potential 3.9 builds, but a couple of weeks ago, even this limited set wasn't working

LisaIbanez

@LisaIbanez

@trivenigandhi yes, I would love access to review! Unfortunately, I can't make tomorrow's meeting, but will def check back here for updates.

Richard Edgar

@riedgar-ms

@adrinjalali to your query on the Python 3.9 PR just merged.... does SciKit-Learn have Python 3.9 wheels out for v0.22.1?

7 replies

Hilde Weerts

@hildeweerts

@trivenigandhi @LisaIbanez @brkifle @WesDeng @kevinrobinson during yesterday's community call we discussed some ideas for fairlearn's roadmap. As frequent attendees we'd love to get your opinion as well, so if you have any ideas on where you would like to see the project going, please feel free to reach out! :D

7 replies

(i apologize in advance if i forgot somebody)

Roman Lutz

@romanlutz

Since v0.6.0 is out I moved us from master to main as the default branch. To make things work with main please approve my PR fairlearn/fairlearn#694 @hildeweerts @adrinjalali @riedgar-ms @MiroDudik @mmadaio

2 replies

vincent d warmerdam

@koaning

Hi all.

With my PR on the correlation removed out there, I feel like I might start working on some educational content for fairlearn. My plan is to host it on https://calmcode.io but I imagine that content could be re-used on the docs as well.

There's one thing though, I don't have a clear intuition on how the reductions actually work. The docs host an example that's a bit too theoretical for me to immediately get my head around ... so is there a call that I can join with a whiteboard soon-ish? I'm not sure if I'm the only one who feels this way, or if the usual meetings are the best place for this, but I'd certainly appreciate a better "paper-napkin"-level of intuition.

4 replies

Hilde Weerts

@hildeweerts

Hi everyone! In this Thursday's community call, we will discuss @hannawallach 's recent work on fairness through the lens of measurement modeling and think about ways to translate these ideas into a toolkit/educational materials. I think this will be a super interesting discussion, here is a link to the paper: https://arxiv.org/abs/1912.05511?

Richard Edgar

@riedgar-ms

PR to cover a testing gap I noticed today:
fairlearn/fairlearn#701
We didn't have any purely public tests of the MetricFrame with sample parameters

Adrin Jalali

@adrinjalali

@romanlutz could you please add the link to the notes (https://hackmd.io/QcJ8WBAbQTq6iOgoi0JT4g?both) to the calendar invite?

2 replies

Roman Lutz

@romanlutz

As you may know we have fairlearn.org as the domain name. So far https was not set up for that, so I'm currently trying to do that. If you have issues accessing the page in the meanwhile please respond by creating a thread here. https still doesn't work since GitHub first needs to issue a certificate, but http://fairlearn.org and fairlearn.github.io should both work and redirect to fairlearn.org (watch what your browser resolves to). Works at least on my computer and my phone. Https should work later tonight/early tomorrow depending on which timezone you're in.

6 replies

Hilde Weerts

@hildeweerts

Hi everybody! Because some people couldn't make it last week, the discussion of @hannawallach's measurement and fairness paper is postponed to this Thursday (25/2/2021). @MiroDudik's talk on reductions will be next week (4/3/2021).

8 replies

Adrin Jalali

@adrinjalali

I'm trying to figure out what the vibe around the model-card-toolkit is. Anybody here knows anything? Seems like there isn't really a community around it. I'm wondering if we have any contacts in the team that we could talk to, to see what their plans are. Also wondering if it makes sense or if there's enough support to actually fork it and create a community around it.

9 replies

It also has a hard dependency on a bunch of tensorflow related packages, which I don't think is necessary for a model card toolkit kinda library.

Adrin Jalali

@adrinjalali

Is this something we'd wanna talk about in our weekly?

Adrin Jalali

@adrinjalali

I was trying to find out where we say how often and how we meet in our docs (to check when it was NY time to make sure I don't miss it due to day light saving changes), but I can't seem to find it

1 reply

Hilde Weerts

@hildeweerts

Thanks everyone for the interesting discussion! I felt like I needed a brain dump, so I opened a discussion (https://github.com/fairlearn/fairlearn/discussions/707)
Please feel free to add your thoughts, perhaps its helpful to collect some of the things we're thinking about to structure next week's followup :)

Where communities thrive

People

Repo info

Activity