These are chat archives for thunder-project/thunder

4th
Jun 2016
Nikita Vladimirov
@nvladimus
Jun 04 2016 22:24
why in linear regression computing model.betas is very quick, and computing model.score(regressors, imSeries) is now separate function which takes forever?
Jason Wittenbach
@jwittenbach
Jun 04 2016 22:28
the technical answer is that fit only takes a map, but score needs to join the betas with the appropriate records in the Series
the practical answer is that you comput them both at the same time with
model, scores = LineareRegression().fit_and_score(X, y)
Nikita Vladimirov
@nvladimus
Jun 04 2016 22:30
this helps, thanks
I like the practical answer more, for some reason :)
Jason Wittenbach
@jwittenbach
Jun 04 2016 22:34
:D :+1:
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:22

so, I have betas:

Series
mode: spark
dtype: float64
shape: (41, 1024, 2048, 4)

and I call betas.tolocal() to get a numpy array. For some reason, this operation takes forever. Any idea why?

Jason Wittenbach
@jwittenbach
Jun 04 2016 23:23
two reasons
  1. that’s what actually kicks off the model fit
  2. that object is > 2 GB and you’re collecting it back to the driver; for whatever reason, collecing things that big tends to be slow
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:26
at this point the job starts losing executors, the old problem. I am using 20 nodes, which was sufficient in previous Thunder version
Jason Wittenbach
@jwittenbach
Jun 04 2016 23:29
how many time points in the dataset?
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:30
will try to restart everything
1100
Jason Wittenbach
@jwittenbach
Jun 04 2016 23:30
yeah, we’ve been seeing some strange cluster activity of late
which makes it hard to say if it’s Thunder-related or cluster-related
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:30
ok, this gives some hope
Jason Wittenbach
@jwittenbach
Jun 04 2016 23:31
is your workflow effecitvely doing:
  1. load images
  2. convert to series
  3. fit model
  4. collect
?
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:32
yes
the usual business :)
Jason Wittenbach
@jwittenbach
Jun 04 2016 23:32
cool cool :)
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:32
also 1.1 registration
Jason Wittenbach
@jwittenbach
Jun 04 2016 23:33
ah, cool
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:33

BTW, when I call

algorithm = LinearRegression(fit_intercept=True)
model, stats = algorithm.fit_and_score(regressors, imSeries)

it complains with /usr/local/python-2.7.6/lib/python2.7/site-packages/scipy/linalg/basic.py:884: RuntimeWarning: internal gelsd driver lwork query error, required iwork dimension not returned. This is likely the result of LAPACK bug 0038, fixed in LAPACK 3.2.2 (released July 21, 2010). Falling back to 'gelss' driver. warnings.warn(mesg, RuntimeWarning)

should I just ignore this whining? Any idea what it wants?
Jason Wittenbach
@jwittenbach
Jun 04 2016 23:34
woah, never seen that one before
I think you can ignore that
seems like an issues inside scipy/scikit-learn, which we use for fitting the model
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:35
ok, that's my favorite method
Jason Wittenbach
@jwittenbach
Jun 04 2016 23:35
though it’s possible that that’s what’s causing the crashes
I wonder if the cluster needs to update it’s scipy/sklearn packages….
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:37
the cluster guys updated scipy to the latest version last week by my request
Davis Bennett
@d-v-b
Jun 04 2016 23:37
i totally recommend using your own python
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:38
?
Davis Bennett
@d-v-b
Jun 04 2016 23:38
so you don't have to ask the cluster guys to install stuff
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:38
I don't mind asking, it happened just once so far
Davis Bennett
@d-v-b
Jun 04 2016 23:39
sometimes they can't install stuff you might want (this happened to me)
Nikita Vladimirov
@nvladimus
Jun 04 2016 23:39
ok