wiseio/datascience-docker

Docker images for data science from Wise.io

Karthik Ram

@karthik

Previous message at @jhamrick

Jessica B. Hamrick

@jhamrick

That could be fun!

Dan Starr

@dstarr1

Pyspark + Docker https://github.com/dockerparis/datascience-dockerized/tree/master/pyspark-docker

Raymond Yee

@rdhyee

How many folks are there physically present at Wise.io HQ for the hackday?

Henrik Brink

@brinkar

There is also this Docker Spark project: https://github.com/sequenceiq/docker-spark

Raymond Yee

@rdhyee

Here's my variant: https://github.com/rdhyee/ipython-spark/blob/master/Dockerfile

Dan Starr

@dstarr1

hey, we have about a dozen people here right now, although we expect more later

Raymond Yee

@rdhyee

the extra mesos stuff in my Dockerfile inspired by trying to get the container to work on mesos. see: https://issues.apache.org/jira/browse/SPARK-2691?focusedCommentId=14194679&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14194679

thanks @dstarr1 for answering my question....nice to get some sense of what's happening at the hackspace itself.

note, my Dockerfile for folding pyspark with Jupyter scipyserver image does not include any hadoop libraries....I'm pondering how to include those dependencies.

Raymond Yee

@rdhyee

BTW, what are people hacking on today?

Henrik Brink

@brinkar

To cleanup your Docker host (old containers and images):

docker rm $(docker ps -a -q)
docker images | grep ubuntu # Find ID of base image if you want to remove everything BUT
docker rmi $(docker images -q | grep -v 5506de2b643b)

Stefan van der Walt

@stefanv

@anabranch docker/docker-py#360

@rdhyee https://github.com/stefanv/docker-tree

Raymond Yee

@rdhyee

@stefanv: cool!

Raymond Yee

@rdhyee

Any of you who want to write about what you're doing -- go for it....as someone who's not in the room, I'd greatly appreciate your narration. :-)

back in a bit.

Ariel Rokem

@arokem

I made this: https://registry.hub.docker.com/u/arokem/itorch/

Raymond Yee

@rdhyee

@arokem --> cool: I'm looking at your Dockerfile https://github.com/arokem/itorch/blob/master/Dockerfile I take it that it's working ok now.

Ariel Rokem

@arokem

Works for me...

Jessica B. Hamrick

@jhamrick

@rdhyee I’ve been working on stuff for my class setup: https://github.com/compmodels/jupyterhub-deploy

Spent time earlier getting the scheduling in docker swarm to work

Now I am working on running nginx in a docker container and serving static files from IPython notebooks through that

Raymond Yee

@rdhyee

@jhamrick I've not looked at docker swarm myself -- so that sounds fun. When does your class start?

Jessica B. Hamrick

@jhamrick

Tuesday :)

Raymond Yee

@rdhyee

more to the point, when does your technology have to work? :-)

Jessica B. Hamrick

@jhamrick

Though, we are not releasing the problem set until Thursday

So less than a week

But we’re almost there!

Raymond Yee

@rdhyee

Good luck!!

Jessica B. Hamrick

@jhamrick

Thanks!

Stefan van der Walt

@stefanv

@anabranch

" Toggle autoindent when pasting
nnoremap <F2> :set invpaste paste?<CR>
imap <F2> <C-O><F2>
set pastetoggle=<F2>

Joshua Bloom

@profjsb

note: pushed nltk, gensim, and psycopg2 (postgresql) into the recent version

Joshua Bloom

@profjsb

cut down the size of the base container

Joshua Bloom

@profjsb

http://www.wise.io/tech/make-docker-images-smaller-with-this-trick

Stefan van der Walt

@stefanv

@profjsb Really? I would imagine if you snapshot the final image and compress the layers you should end up with the same thing?

Joshua Bloom

@profjsb

maybe, but for auto builds in docker hub you cannot do that compression

so for pulling over images from registries you need to have a natively compressed image

Stefan van der Walt

@stefanv

@profjsb Neat! Lots of tools out there for doing compression, but good to know a way to do this without any intervention.

Joshua Bloom

@profjsb

I'm sure one could make these smaller with effort, but this is so simple and I couldn't find anyone talking about it...thus, the blog

Stefan van der Walt

@stefanv

I like it. Pragmatic!

And, I have never seen this trick described before.

Here is a link on how to do squashing: http://jasonwilder.com/blog/2014/08/19/squashing-docker-images/

Joshua Bloom

@profjsb

maybe: try to squash this image and post here what you find?

Stefan van der Walt

@stefanv

Good idea!

How do you compute the size of a docker image locally?

Joshua Bloom

@profjsb

docker pull <image:tag>

docker images

Stefan van der Walt

@stefanv

wiseio/datascience-docker latest 500404ab220f 22 hours ago 1.489 GB

Where communities thrive

People

Repo info

Activity