These are chat archives for dropbox/pyston

6th
Sep 2016
Thomas Markovich
@tmarkovich
Sep 06 2016 16:36
Hi, I've been trying to get a docker image working with pyston + scipy + numpy, but haven't been able to get anything working yet. Any advice, or working dockerfiles that can be provided?
Sun
@Daetalus
Sep 06 2016 17:05
Would you mind to give more information about "haven't been able to get anything working", please?
Thomas Markovich
@tmarkovich
Sep 06 2016 17:06
sure!
using the following docker file:

FROM buildpack-deps:jessie

# http://bugs.python.org/issue19846
# > At the moment, setting "LANG=C" on a Linux system *fundamentally breaks Python 3*, and that's not OK.
ENV LANG C.UTF-8

ENV PYSTON_TAG v0.5.1
ENV PYSTON_VERSION 0.5.1

ENV PATH /pyston_env/bin/:$PATH

RUN set -x \
    && curl -SL "https://github.com/dropbox/pyston/releases/download/${PYSTON_TAG}/pyston-${PYSTON_VERSION}-linux64.tar.gz" | tar -xzC / \
    && /pyston-${PYSTON_VERSION}-linux64/pyston /pyston-${PYSTON_VERSION}-linux64/virtualenv/virtualenv.py /pyston_env; \
    pip install https://github.com/dropbox/pyston/releases/download/${PYSTON_TAG}/Cython-0.24-pyston.tar.gz; \
    apt-get update && apt-get install -y --no-install-recommends libpng-dev freetype* libblas-dev liblapack-dev libatlas-base-dev gfortran gcc g++ gfortran-multilib gfortran-4.9-multilib; \
    pip install nose mock flake8 ipython ipdb cloudpickle;\
    pip install -vvv git+git://github.com/numpy/numpy; \
    echo "source /pyston_env/bin/activate" >> /root/.bashrc

and then using pip install within the docker image I run into issues with finding atlas. It complains that atlas isn't installed to /usr/lib

when I close scipy and then run this guy /pyston_env/bin/pyston -c "import setuptools, tokenize;__file__='setup.py'; exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record install-record.txt --single-version-externally-managed --compile --install-headers /pyston_env/include/site/python2.7gfortran I run into an issue with _vc.o
cc1: some warnings being treated as errors error: Command "gcc -pthread -fno-strict-aliasing -O3 -DNDEBUG -fwrapv -Wall -Wstrict-prototypes -fPIC -DATLAS_INFO="\"3.10.2\"" -I/pyston_env/site-packages/numpy/core/include -I/usr/include/atlas -I/pyston_env/site-packages/numpy/core/include -I/pyston_env/from_cpython/Include -c -Werror=implicit-function-declaration scipy/cluster/_vq.c -o build/temp.linux-x86_64-2.7/scipy/cluster/_vq.o" failed with exit status 1
Thomas Markovich
@tmarkovich
Sep 06 2016 17:11
@Daetalus please let me know if you need anything else!
Marius Wachtler
@undingen
Sep 06 2016 19:02
does not directly fix the problem you are seeing but you can use our prebuilt docker images: docker run -it pyston/pyston-numpy
Thomas Markovich
@tmarkovich
Sep 06 2016 19:11
@undingen I tried to start that one and then build scipy within that image. Even with installing atlas/openblas/etc, scipy gave similar errors
Thomas Markovich
@tmarkovich
Sep 06 2016 19:17

@undingen ```
apt-get update && apt-get install -y --no-install-recommends libpng-dev freetype* libblas-dev liblapack-dev libatlas-base-dev gfortran gcc g++ gfortran-multilib gfortran-4.9-multilib

pip install scipy

yields:

scipy/cluster/_vq.c: In function ‘__Pyx_AddTraceback’:

scipy/cluster/_vq.c:8961:13: error: dereferencing pointer to incomplete type

     py_frame->f_lineno = py_line;

             ^

scipy/cluster/_vq.c: At top level:

scipy/cluster/_vq.c:9011:27: fatal error: longintrepr.h: No such file or directory

   #include "longintrepr.h"

                           ^

cc1: some warnings being treated as errors

compilation terminated.

```

that was run in the pyston/pyston-numpy container
Marius Wachtler
@undingen
Sep 06 2016 19:35
oh sorry, I missed that you had problems compiling scipy not numpy :-(
did you get scipy from https://github.com/cython/cython 0.24 ?
Thomas Markovich
@tmarkovich
Sep 06 2016 19:37
no, I just grabbed from the github
just like numpy does
Marius Wachtler
@undingen
Sep 06 2016 19:38
because AFAIK if you use it from pip it already comes with a generated cython output. and unfortunately that one that not work with pyston (the new master cython should work)
ok :-(
I will try and check if it works for me
Kevin Modzelewski
@kmod
Sep 06 2016 19:40
We have have a scipy integration test which could be a guide on how to install scipy
Also the scipy stuff has been actively worked on recently, so 0.5.1 might not have full scipy support
Marius Wachtler
@undingen
Sep 06 2016 19:42
did you get the scipy from the github repo?
(sorry before I posted the link to cython instead of scipy)
if not I think this is the problem
Thomas Markovich
@tmarkovich
Sep 06 2016 19:43
@undingen I did. I cloned and then built
Marius Wachtler
@undingen
Sep 06 2016 20:12
Ok unfortunately it does not work out of the box with the numpy docker image but I got it working with a small change:
apt-get update && apt-get install -y --no-install-recommends libpng-dev freetype* libblas-dev liblapack-dev libatlas-base-dev gfortran gcc g++ gfortran-multilib gfortran-4.9-multilib vim

cd pyston_env/
. bin/activate
git clone --depth 1 --branch  v0.17.1 https://github.com/scipy/scipy
cd scipy/
then I had to comment out this two lines in/pyston-0.5.1-linux64/from_cpython/Lib/distutils/ccompiler.py :
 if not any ('scipy' in s for s in pp_opts):
    cc_args = cc_args + ["-Werror=implicit-function-declaration"]
Sun
@Daetalus
Sep 06 2016 20:14
Sorry, just came back...
Marius Wachtler
@undingen
Sep 06 2016 20:14

then I run

python setup.py build
python setup.py install
cd ..
python

and it imported (but I was too lazy to run the tests :-()

Sun
@Daetalus
Sep 06 2016 20:15
scipy/cluster/_vq.c generated by Cython. Did you use the Pyston patched Cython?
Marius Wachtler
@undingen
Sep 06 2016 20:16
@kmod fixed this error in dropbox/pyston@c5aa72a so in the next pyston release it should work without changes
Sun
@Daetalus
Sep 06 2016 20:17
@kmod what's your opinion about comment out bunch of codes in from_cpython/setup.py in https://github.com/dropbox/pyston/pull/1354/commits/d09f37c59f4c232225d8ce10bf65a761606cda5a
Do not install scipy from pip. Because it will contains the C source code which generated by Cython for CPython.
Kevin Modzelewski
@kmod
Sep 06 2016 20:40
@Daetalus just responded on the PR :) it looks fine but I'm hoping it can be cleaned up a bit
Sun
@Daetalus
Sep 06 2016 20:40
Thank you!
Kevin Modzelewski
@kmod
Sep 06 2016 20:41
their setup.py looks pretty messy so if the changes can't be cleaned up then that's ok, but it would just make things easier down the road if we can easily tell what our differences are
Thomas Markovich
@tmarkovich
Sep 06 2016 21:53
@undingen thanks for the help! I've got a working dockerfile. I'll post to dockerhub in a bit