These are chat archives for astropy/astropy

10th
Nov 2015
Thomas Robitaille
@astrofrog
Nov 10 2015 09:33
@michaelaye - I think this comes from the setup_package.py file from astropy/wcs
feel free to report that, along with your compiler version!
Sidhant Bhavnani
@cosmicAsymmetry
Nov 10 2015 09:34
Anyone taking part in PUPC?
Thomas Robitaille
@astrofrog
Nov 10 2015 09:37
@bhavnanisidhant what is PUPC?
Sidhant Bhavnani
@cosmicAsymmetry
Nov 10 2015 09:38
princeton university physics competition.
Thomas Robitaille
@astrofrog
Nov 10 2015 09:41
Ah right, not familiar with it.
Sidhant Bhavnani
@cosmicAsymmetry
Nov 10 2015 09:50
Although it's only highschool level.
Thomas Robitaille
@astrofrog
Nov 10 2015 12:39
@bsipocz - just to let you know, I'm online this afternoon if you have any questions about things relating to ci-helpers
Brigitta Sipocz
@bsipocz
Nov 10 2015 13:51
@astrofrog - Thanks. It seems that I managed to make it work with photutils (finally), so now can clean it up regarding the optional dependencies, jinja, etc.
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:02
OK, so I shouldn't have counted those chickens yet. Somehow for older numpys it pickes up ancient 0.4.x astropy versions...
Thomas Robitaille
@astrofrog
Nov 10 2015 14:17
Thanks for working on this!
Wonder if the older numpy version thing is related to the astropy-ci-extras channel
@bsipocz - I think to some extent it makes sense it picks up old astropy version, we don't have all combinations in astropy-ci-extras
but are these combinations that worked better before?
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:19
Yes they worked. Like numpy 1.6/1.7/1.8 and astropy stable
Thomas Robitaille
@astrofrog
Nov 10 2015 14:20
stable meaning 1.0?
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:20
it should never go back to 0.4.x
yes
stable meaning that it gets it from conda
without a version number
Thomas Robitaille
@astrofrog
Nov 10 2015 14:20
right so it should fine astropy 1.0.3 and numpy 1.6 1.7 1.8 from here
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:20
vs here
the latter is using ci-helpers
the previous with the old infrastructrue
Thomas Robitaille
@astrofrog
Nov 10 2015 14:21
ok checking
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:21
so I guess it doesn't pick up astropy-ci-extras
Thomas Robitaille
@astrofrog
Nov 10 2015 14:21
yes, seems so
wonder if it needs to have the channel in the install command
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:22
as in normal conda none of these combinations is available
ahh, thats is
it
Thomas Robitaille
@astrofrog
Nov 10 2015 14:23
Is it that before we used
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:23
I overwrote it
Thomas Robitaille
@astrofrog
Nov 10 2015 14:23
- CONDA_INSTALL='conda install -c astropy-ci-extras --yes'?
ok np
so I guess it ignores the channel specified here?
maybe that doesn't do anything?
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:24
later at numpy install it's redefined
L36
Thomas Robitaille
@astrofrog
Nov 10 2015 14:26
Right, but I thought it would still use the channel specified during the environment creation, but now I realize that's probably used only to install the actual initial dependencies in the environment (I had assumed before that it was some kind of global setting for the environment)
Rather than specify the channel every time, we could do this
conda config --add channels r
well not r but astropy-ci-extras
conda config --add channels astropy-ci-extras
what do you think?
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:28
OK. I don't use conda normally, so this is very much at the experimental level for me. Try to tweek it until it works :)
Thomas Robitaille
@astrofrog
Nov 10 2015 14:28
that's what I do too! ;)
I think it might be cleaner to specify the channel in the config as mentioned above as a one off, at the same time as setting the always_yes option
then users of ci-helpers don't need to remember it either when adding conda commands to .travis.yml
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:29
adding it to the config sounds good, don't like the extra long bash strings, it's good if they can be avoided
Thomas Robitaille
@astrofrog
Nov 10 2015 14:29
:+1:
I'm glad we're working on ci-helpers, I think the multitude of packages was starting to become unmanageable!
each time something breaks in conda....
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:33
yes, I thought all of them can be fixed in a few ours, then ended up spending a full afternoon and a night on it and still not all of them got fixed, etc.
it's also good for the other issues we had recently with fixing coveralls version, adding jinja, etc
Thomas Robitaille
@astrofrog
Nov 10 2015 14:34
Yes, indeed
I guess we'll have one last round of manual updating to implement all this ;)
once we have a PR open for photutils, package-template, and the core package, I think we should email the list
to propose this and see if anyone has feedback
before we open any other PRs
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:36
OK. And I still have the open PRs for many of the packages, can update those easily.
Thomas Robitaille
@astrofrog
Nov 10 2015 14:38
Ok, great
:)
So as discussed in the issue, I was wondering whether it would make sense to replace INSTALL_OPTIONAL and OPTIONAL _DEPENDENCIES by CONDA_ADDITIONAL and PIP_ADDITIONAL?
not sure
at the moment not sure if we need the two options we have in the sense that we could just use OPTIONAL_DEPENDENCIES as a blank string to mean don't install
so I think we definitely don't need INSTALL_OPTIONAL
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:43
I would keep INSTALL_OPTIONAL as it's a bool in the matrix, but don't have any strong preference for the rest
e.g. is there any way to test whether something is not in conda?
in that case we don't even need the two lists, just one
and if there is any other weird dependency, that can be dealt at the package level in .travis.yml
Thomas Robitaille
@astrofrog
Nov 10 2015 14:45
Regarding INSTALL_OPTIONAL, what I meant is that in the matrix, we could replace
        - INSTALL_OPTIONAL=true
        - OPTIONAL_DEPENDENCIES='scipy scikit-image matplotlib'
by
        - OPTIONAL_DEPENDENCIES=''
then in the jobs that do use them:
        - python: 2.7
          env: SETUP_CMD='test' OPTIONAL_DEPENDENCIES='scipy scikit-image matplotlib'
the benefit of doing that is that then it's not all or nothing
you can have jobs with some optional dependencies but not all
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:47
ohh, ok now I see the logic. For me it was, if we do it in one place updating it will be less sensitive to errors, and at the top of the file.
Thomas Robitaille
@astrofrog
Nov 10 2015 14:48
right, I mean my way is a bit annoying if you have 5 builds all with the same optional dependencies
but more flexible, and just to be clear packages could still choose to do something similar to before:
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:49
yes, you're right.
Thomas Robitaille
@astrofrog
Nov 10 2015 14:49
        - ALL_OPTIONAL_DEPS='scipy scikit-image matplotlib'
        - OPTIONAL_DEPENDENCIES=''
then could set OPTIONAL_DEPENDENCIES=ALL_OPTIONAL_DEPS
for example
by the way I don't feel strongly about all this, so feel free to shoot down any of these ideas :)
also I'm jet lagged :airplane: so my suggestions should be taken with a pinch of salt
;)
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:50
so is there a way to feed $OPTIONAL_DEPENDENCIES to conda first and what's not there to pip? can it provide such a failed list to a variable or tempfile or something?
Thomas Robitaille
@astrofrog
Nov 10 2015 14:51
not sure, checking
would be nice
doesn't look like it
Here it just says you have to build the conda packages: http://www.peterbronez.com/Using%20PyPi%20Packages%20with%20Conda
not a good solution
I think we can either have two env variables, or keep a whitelist of packages we know are available in conda
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:55
having a former is more clear I think, managing the latter might not worth it
Thomas Robitaille
@astrofrog
Nov 10 2015 14:56
yeah, and maybe we can open a feature request for conda package
to have a --pip-fallback option or something
Brigitta Sipocz
@bsipocz
Nov 10 2015 14:57
:+1:
===
it may be unrelated but the new build seem to be ~25-30% slower than it used to be 63min vs 46 min
Thomas Robitaille
@astrofrog
Nov 10 2015 14:59
huh
are all builds slower?
There seems to be a big variance though: https://travis-ci.org/astropy/photutils/builds
There's even a 2 hour build!
If we do CONDA_ADDITIONAL and PIP_ADDITIONAL we can set CONDA_ADDITIONAL='Cython jinja2' for the core package
is that what you were thinking too?
(for the default configs)
Brigitta Sipocz
@bsipocz
Nov 10 2015 15:05
we can have that, but then we still need two variables for the optional depts
whatabout calling it CONDA_DEPENDENCIES and PIP_DEPENDENCIES? rather than CONDA_ADDITIONAL?
Thomas Robitaille
@astrofrog
Nov 10 2015 15:15
Sounds good!
We can just make it clear in documentation that numpy and astropy don't need to be included in that list
Brigitta Sipocz
@bsipocz
Nov 10 2015 15:22
is that normal that pip install git+http://github.com/astropy/astropy.git#egg=astropy doesn't bring on jinja2?
Thomas Robitaille
@astrofrog
Nov 10 2015 15:54
yes
well normal no but expected at this point yes
astropy/astropy#3229
so need to specify it manually as a dependency for developer version
@bsipocz - note that the installation of CONDA_DEPENDENCIES should come after the Numpy installation and use $CONDA_INSTALL
also I would put all the pip stuff after all the conda install stuff
otherwise pip might end up installing e.g. numpy
Thomas Robitaille
@astrofrog
Nov 10 2015 16:05
Ah I think you did this already, was looking at an old commit
sorry! :)
back a bit later
Brigitta Sipocz
@bsipocz
Nov 10 2015 16:08
OK, I'm just back now from tea, and looking at it again
Thomas Robitaille
@astrofrog
Nov 10 2015 16:09
ok - I think you had already done what I was suggesting, but you can check
basically the order should be that Numpy should be the first thing we ever install, then we should always use CONDA_INSTALL after
and pip stuff should always come after conda to make sure we don't pip install anything complicated
Brigitta Sipocz
@bsipocz
Nov 10 2015 16:13
nope, I still have to move the additional dependencies after numpy (to avoid installing it before the shell script gets there), so another line of install will be needed (as we need pip for the pep8 before, and that shouldn't wait for installing numpy first)
Thomas Robitaille
@astrofrog
Nov 10 2015 16:17
Right yes, pep8 is a special case and should be before
Brigitta Sipocz
@bsipocz
Nov 10 2015 16:58
ok, so travis is done for photutils, PR in photutils/photutils#291
Also updated the appveyor script, but that may still be failing (e.g. you didn't seem to install astropy in the ci-helpers script?)
Thomas Robitaille
@astrofrog
Nov 10 2015 17:52
@bsipocz - you're right, I don't deal with dependencies for appveyor at this point
so for now leave the dependency installs in appveyor.yml
will take a look at the photutils PR
Brigitta Sipocz
@bsipocz
Nov 10 2015 17:53
OK. I give it another few minutes to figure out the powershell's syntax magic to put it there, but will give up afterwards.
Thomas Robitaille
@astrofrog
Nov 10 2015 17:55
haha yes, I spent a little while on that myself to get the path working correctly
ah btw if %ASTROPY_VERSION% doesn't work, try $env:ASTROPY_VERSION
Brigitta Sipocz
@bsipocz
Nov 10 2015 17:58
yes, I've just pushed it
hopefully it will work
I always loved shell scripts, but this one's syntax is just makes me sick :)
Thomas Robitaille
@astrofrog
Nov 10 2015 17:59
yeah, it's painful!
Thomas Robitaille
@astrofrog
Nov 10 2015 18:07
@bsipocz - is there any reason to have separate CONDA_OPTIONAL_DEPENDENCIES and CONDA_ADDITIONAL_DEPENDENCIES on the ci-helpers side? One env variable should be enough, right?
(I meant on the travis side)
Brigitta Sipocz
@bsipocz
Nov 10 2015 18:12
yes, that's indeed a mess (I didn't notice it)
Thomas Robitaille
@astrofrog
Nov 10 2015 18:12
no problem! and by the way I don't mean you need to fix all these things, just thinking out loud :)
Brigitta Sipocz
@bsipocz
Nov 10 2015 18:13
not, but it's a mess, as they are not picked up anyway as I intended
Brigitta Sipocz
@bsipocz
Nov 10 2015 21:05
appveyor is still not working, it seems that it thinks that the dependency list is a single package?
Activating environment "C:\conda\envs\test"...
Python 2.7.10
Error: Invalid package specification: Cython scipy jinja2 scikit-image matplotlib
Also I cound't really pass the variable values within the environments, thus still there are the two globals, CONDA_OPTIONAL_DEPENDENCIES and CONDA_ADDITIONAL_DEPENDENCIES
by default both are installed, but CONDA_DEPENDENCIES can be given in the matrix, and then only those gets installed.
Thomas Robitaille
@astrofrog
Nov 10 2015 21:16
hmm that's weird (about the appveyor issue with package list)
@bsipocz - for AppVeyor, maybe you could try and put .Split(" ") at the end?
I'm happy to try too
i.e. $env:CONDA_ADDITIONAL_DEPENDENCIES.Split(" ")
for Travis, just so I understand, why can't we do away with CONDA_ADDITIONAL_DEPENDENCIES and CONDA_OPTIONAL_DEPENDENCIES? Does it make it too difficult to write the photutils matrix?
Thomas Robitaille
@astrofrog
Nov 10 2015 21:22
(just to make sure I understand)
Thomas Robitaille
@astrofrog
Nov 10 2015 21:32
@bsipocz - I opened a PR to your branch, but not sure if it would pass on Travis.
bsipocz/photutils#2
just an idea though
(so don't feel obliged to merge)
(I'm running it on Travis on my branch)
Thomas Robitaille
@astrofrog
Nov 10 2015 21:50
I'm off for the evening, but we can chat about this again whenever you have time tomorrow or later :)
thanks for your work on this!
Brigitta Sipocz
@bsipocz
Nov 10 2015 23:27
It's still not working, with the same issue I had before. It's all back to the shell substitutions, it just simply doesn't goes through.
(this build should have installed the optional dependencies)
also had the same with PYTHON_VERSION``` andTRAVIS_PYTHON_VERSION. whenever I saidPYTHON_VERSION=$TRAVIS_PYTHON_VERSIONin .travis.yml it didn't work, thus it's ended up inci-helpers`` in an if conditional