2nd
Oct 2016
Hèlen Grives
@mesmoiron
Oct 02 2016 06:14
@evaristoc you should read Taleb. He even makes the statement that most people in industry like quants don't know their statistics and hence wreck havock in finance and politics. Yes I don't have a degree either, but I am blessed with a statistical mind that is I tend to see where the errors of reasoning lie without the ability to calculate it. Taleb call that Fat Tony-people. To understand is not always necessary to know statistics as it obscures good thinking by making things more complicated than it should be. Hence it will be used by clever person without good intention like greed etc. Anyway I am studying statistics as much as possible, but mainly to spot bad statistics and to be able to explain why it is bad. I must say I don't see many people do this or even aspire one tiny bit. I too had to overcome my hate for statistics, because it alienates the human too much. Everything becomes a number. My curiosity started when I noticed that people homeless etc often lack good reasoning skills and they are more and more depending on people having the skills to advocate there cause. So I focused on making clear with government officials that people needed good tools that can calculate and 'reason' for them. To make it visible what the good choices are. I realised that I was very lonely in holding that position; because economics tend to drive people where the money most easily is available. Doing long hours of study statistics is not compelling for many as a means to improve their lives.
Hèlen Grives
@mesmoiron
Oct 02 2016 06:40
Can anyone help me with this question from Dataquest site: Figure out what proportion of the values in predictions are the exact same as the values in titanic["Survived"]. I am trying to get familiar with Kaggle; however I do not know what it is precisely that they want me to calculate. Do I have to compare each value with the prediction and if it is predicted correctly? Or are they referring to a standard calculation I should have known like calculating accuracy with means squared method. I am still in the phase of vague notions working on mastery. why it is that you do what you do.
evaristoc
@evaristoc
Oct 02 2016 11:36

### People

just in case you are interested, I am reviewing again previous theory:

## Some MCMC methods for sampling: Gibbs

http://csg.sph.umich.edu//abecasis/class/815.23.pdf

I feel like needing to study many things again... trying not to forget...
@mesmoiron I think they are asking which values are exactly predicted and which ones not when compared to the actual data?
About kaggle: do all what you want, make mistakes and keep it simple. Suggest to avoid non-technical discussions there: not the platform for that. Come a discuss/ask questions here if you want.
CamperBot
@camperbot
Oct 02 2016 11:41
Hèlen Grives
@mesmoiron
Oct 02 2016 13:14
@evaristoc thank you; I am also following along https://www.youtube.com/channel/UCfzlCWGWYyIQ0aLC5w48gBQ
CamperBot
@camperbot
Oct 02 2016 13:14
mesmoiron sends brownie points to @evaristoc :sparkles: :thumbsup: :sparkles:
evaristoc
@evaristoc
Oct 02 2016 14:46
@mesmoiron : :+1:! A bit advanced, if you want to know my opinion... if no previous experience with programming or stats, I strongly advise a traditional statistics course instead, even with python (pandas + scipy + numpy + statmodels + (general graphics)). Up to you.
Hèlen Grives
@mesmoiron
Oct 02 2016 14:52
@evaristoc haha; I like challenges. I have done various introductory stats moocs (not to brag or pretend anything); I also know something about Weka, learning by doing. I have traded forex for a while, but I also want to be able to program and test strategies. So then you just dive in. I'm not afraid of autodicatic learning. I will piece together my knowledge. Fear not ;-) For now I'm just doing everything under the radar...
evaristoc
@evaristoc
Oct 02 2016 15:18
@mesmoiron :) Good luck! There is a lot around there so it might take some time but it is not unrealistic goal.
@mesmoiron the link you are using in youtube... hmmm... not bad but be careful... better to complement with more serious material for some of the videos... he is not presenting correctly some fundamentals in some cases...
Jacqueline S. Homan
@jacqueline-homan
Oct 02 2016 17:28

@mesmoiron : Much of the root cause is not only due to a real lack of opportunities for people struggling in poverty but also the lack of any real safety net for the jobless poor here in the US courtesy of Welfare Reform.

Even with new jobs being created, many Americans in deep poverty because of joblessness remain 100% shut out of the economy - especially the long-term unemployed over age 40.

And age discrimination impacts women and minorities much harder because of the sex and/or race discrimination that also impacted us resulting in opportunities denied (which translates to real deprivation of income and access to medical and dental care) throughout our lives before many of us got to be "too old" for anyone to deem us a "good fit" for being hired - regardless of how hard we worked at building skills or what we tried.

The life expectancy rate for America's poor (across all racial lines) has dropped to just age 60 since Welfare Reform. The life expectancy rate for the middle class and the affluent is age 80.

Pulitzer prize winning journalist Christopher Hedges recently debated former US Labor Secretary Robert Reich on Democracy Now! and he talked extensively about the appalling conditions of poverty (not just for people of color but also whites, too - the majority of the people who are among the ranks of the jobless poor and/or homeless in the US are white).

And if anyone were interested in a data-gathering project, try counting the number of people in tent cities who are over age 40 that ended up losing everything and becoming homeless because of not getting hired/re-hired (and often also getting hit with the double whammy of a major health crisis while unemployed, too) despite having "re-tooled" and learning new skills. Age discrimination (as well as sex and race discrimination and deeply entrenched classism) is a huge problem and the burden is unfairly placed 100% on the victims instead of on those who are the gatekeepers who are doing the discriminating in hiring. (And the "winners" who are able to leverage their privileges (white, male young, and middle class/rich) to get all the good jobs and other economic opportunities certainly aren't sharing their paychecks with those who lost out to them who are being shut out and sent away poor and empty-handed with no job opportunities. Society's "winners" have no sense of noblesse oblige here in the US.)

Being sent away poor and empty-handed with no job due to age or any other kind of discrimination -> no way to provide for your basic living needs.

Before Welfare Reform, we had a way of counting the number of people in poverty that were shut out of opportunity. But now we only have morgue and prison stats. To the best of my knowledge, no one is attempting to count the number of destitute people struggling for their lives in "tent cities" - they would rather the poor and homeless just not be seen or heard.

I am personally against the idea of maintaining a status quo of no real safety net for the jobless poor and having only a patch-work of NGOs/charities as the only safety net and as the only poverty alleviation solution. The charity/NGO model is not working for helping the poor up onto their feet and lifting them out of poverty because the problem (and its symptoms) is systemic and structural.

Most desperately poor people in dire need are turned away and NOT helped with what they need: incomes and medical care - hence a burgeoning homelessness and poverty problem that is not getting better for those who've been economically pushed out and who remain shut out of the job market due to age discrimination. (But charity execs and paid staff sure always seem to get paid and have health benefits.)

Hèlen Grives
@mesmoiron
Oct 02 2016 17:37
@evaristoc I know, but I think it is a good source before I'll sink my teeth in any MIT, or Stanford opencourseware. It will make me more confident if I know the general workflow, before my brain resource are spent on the heavy math.
Jacqueline S. Homan
@jacqueline-homan
Oct 02 2016 17:41

I think it goes back to what the author of weapons of math destruction said about models b "Models are opinions embedded in mathematics."

I agree 100% @Lightwaves.

Hèlen Grives
@mesmoiron
Oct 02 2016 17:52
@jacqueline-homan thank you, I'm too aware of that. I do think at least some organisations are trying to do this Invisible people and Prison Reform are amongst those that I follow on Twitter. But I do think that they lack the computational resources and basic survey skills (the smaller organisations in Holland at least do). It is my goal to enter the vacuum, but it would be a very small contribution. It will be harder and harder for organizations because of the knowledge gap. Financialization and mathification of services leave the strong ones with the brains and capacity/resources to manipulate every attempt to increase well being for the bottom poor or under priviledged. I have had the discussion with the Black Lives Matter, that I pointed out that maybe only a few persons tend to educate themselves in being more effective counterparts. As the role models are quite busy selling expensive gear to their followers instead of donating to learning opportunities or grants on a larger scale. I am also part of the Basic Income movement; however no one seems to care about visualisations, computations and other forms of knowledge gathering. Once again my motivation in an attempt at least to understand how to be a good debating partner. I held the opinion that a job is only a narrow minded concept. There is so much work out there for those who can see it and are able to do something with it. Amir Sen (not sure about the name) has great work on capability approach etc. Martha Nussbaum. For now I like this channel. It gave me already a lot of worth. I don't mind being a beginner. Have a nice evening/day to all.
CamperBot
@camperbot
Oct 02 2016 17:52
mesmoiron sends brownie points to @jacqueline-homan :sparkles: :thumbsup: :sparkles:
Jacqueline S. Homan
@jacqueline-homan
Oct 02 2016 17:54

My curiosity started when I noticed that people homeless etc often lack good reasoning skills and they are more and more depending on people having the skills to advocate there cause...

We also need to ask another question: How much of this poor decision-making ability is due to the horrific toll that abject poverty takes on people psychologically as well as physically. Poverty destroys your health. Suffering is not ennobling. It destroys.

Jacqueline S. Homan
@jacqueline-homan
Oct 02 2016 18:07

@mesmoiron : A bit of self-disclosure here. I have a Bachelors degree in pure math w/ a minor in physics. I am also a self-taught programmer and my strongest tech skillset is F#.

I am also someone who has first-hand experience w/ suffering many years of abject poverty and no access to medical and dental care as a result of a lifetime of marginalization due to classism on top of sexism. Not to thread-jack, but you can read more about my background here written by my mentor if you're interested: https://hackernoon.com/this-is-what-marginalized-looks-like-3115fb4edf13#.gkj7tbgho

Statistics are only as good as the accuracy in the raw data being collected, which of course begins with knowing how to ask the right questions (a very simple concept that so many people overlook). If you want, hit me up about learning F# and harnessing the power of type-driven development for building data-rich programs. :smile:

Hèlen Grives
@mesmoiron
Oct 02 2016 18:13
@jacqueline-homan yes I am interested. Will read tomorrow with a fresh mind and get back to you.
Jacqueline S. Homan
@jacqueline-homan
Oct 02 2016 18:16
Awesome! And let me reiterate: F# is my favorite programming language for tackling problems such as the ones you brought up. (As well as many other problems, like solving cold-case files and whatnot). :+1:
Lightwaves
@Lightwaves
Oct 02 2016 19:15
@jacqueline-homan It's been awhile, it's cool to see you back.
Jacqueline S. Homan
@jacqueline-homan
Oct 02 2016 20:51
@Lightwaves , yeah been battling a major health crisis (without access to any medical care because poor). I am just now starting to get back into coding again after getting sick in June. I am just now starting to pick up where I left off and I am building a Patreon page in hopes of being able to earn a modest, livable sustainable income through offering screencast tutorials on Data Science, F#, Linux, and eventually Haskell and R, and other technologies as well as other interesting educational videos - some related to coding and some not.