These are chat archives for nextflow-io/nextflow

23rd
Sep 2016
Paolo Di Tommaso
@pditommaso
Sep 23 2016 07:54
Nice
Matthieu Foll
@mfoll
Sep 23 2016 08:32
Hi all, I have been playing with the new Windows 10 Subsystem for Linux and nextflow and it seems to work well. Here are the basic steps:
  • Install Windows 10 Subsystem for Linux (see https://msdn.microsoft.com/commandline/wsl/install_guide#)
  • Install Java JRE: $ sudo apt-get install default-jre
  • There is currently a bug with java (see Microsoft/BashOnWindows#541), but an easy workaround is to add: export _JAVA_OPTIONS="-Xint”
  • Install nextflow: wget -qO- get.nextflow.io | bash
  • Enjoy:
    $ ./nextflow run helloworld
    Picked up _JAVA_OPTIONS: -Xint
    Picked up _JAVA_OPTIONS: -Xint
    N E X T F L O W  ~  version 0.22.0
    Pulling nextflow-io/helloworld ...
    downloaded from https://github.com/nextflow-io/hello.git
    Launching `nextflow-io/helloworld` [tiny_feynman] - revision: 6b9515aba6 [master]
    [warm up] executor > local
    [5f/e40692] Submitted process > sayHello (2)
    [be/b563cd] Submitted process > sayHello (4)
    [54/51232c] Submitted process > sayHello (3)
    [c0/90c9d8] Submitted process > sayHello (1)
    Hola world!
    Bonjour world!
    Hello world!
    Ciao world!
I’m now trying to add docker in the game, will let you know
@Mokok and @skptic
Evan Floden
@evanfloden
Sep 23 2016 08:37
@mfoll Amazing! Good job. We will have to get the boss to a get a windows license now so we can do some testing.
Paolo Di Tommaso
@pditommaso
Sep 23 2016 08:39
Cool. @mfoll what your plan for Docker? I mean to install it in the Linux subsystem or is there are way to access the Windows native Docker version ?
Matthieu Foll
@mfoll
Sep 23 2016 08:40
install docker in the Linux subsystem but connect it to the native Windows one
Paolo Di Tommaso
@pditommaso
Sep 23 2016 08:41
ah, this is a nice idea
:+1:
If you manage in doing that I would be very happy to guest a post about that in the NF blog
Mokok
@Mokok
Sep 23 2016 08:58
wew ! thanks @mfoll !
(have you ever tried with soft like cygwin or similar ? not to be forced to use win10)
so...there could be a way to define 'windows specific' task in a task-chain... really promising
Matthieu Foll
@mfoll
Sep 23 2016 09:03
I tried some time ago with cygwin but couldn’t get java to work properly. But I didn’t spend too much time on it, so it might be possible.
Paolo Di Tommaso
@pditommaso
Sep 23 2016 09:05
I think the way that is proposing @mfoll would allow a pipeline to mix Linux/Windows tasks in the proper container isolation
Mokok
@Mokok
Sep 23 2016 09:06
i ran into similar issue in my 'free' time, i didn't spend much time either
Matthieu Foll
@mfoll
Sep 23 2016 12:57
A further note on Windows 10 Linux: the java bug currently existing is still creating issues (java and threfore nextfllow hangs) even with the -Xint fix. Hopefuly it will be fixed in a future update.
Paolo Di Tommaso
@pditommaso
Sep 23 2016 12:58
I see, thanks for investigating that Matthieu
Matthieu Foll
@mfoll
Sep 23 2016 12:58
Regarding docker: we can install docker in Windows and docker in Linux and use the Windows Host in Linux:
wget https://get.docker.com/builds/Linux/x86_64/docker-1.12.0.tgz
tar -xzvf docker-1.12.0.tgz
mkdir ~/bin
mv docker/docker ~/bin
mv nextflow ~/bin
export PATH=$PATH:~/bin
export DOCKER_HOST=localhost:2375
Paolo Di Tommaso
@pditommaso
Sep 23 2016 12:59
yes, you waere mentioning this. it's a great idea
Matthieu Foll
@mfoll
Sep 23 2016 12:59
But there are issues for mounting volumes between Linux and Windows
Paolo Di Tommaso
@pditommaso
Sep 23 2016 12:59
ah
Matthieu Foll
@mfoll
Sep 23 2016 13:00
But maybe there is a workaroud: when you mount the volume, this works:
docker run -i -v c:/Users/m_fol/AppData/Local/Lxss/home/mfoll/work/01/3649c747b2ea81505e2e8d86207363:/home/mfoll/work/01/3649c747b2ea81505e2e8d86207363 -w "$PWD" --entrypoint /bin/bash nextflow/examples -c "/bin/bash -ue /home/mfoll/work/01/3649c747b2ea81505e2e8d86207363/.command.sh”
We need to add c:/Users/m_fol/AppData/Local/Lxss before mounting the volume in my case
Paolo Di Tommaso
@pditommaso
Sep 23 2016 13:01
because I guess the path have a different format between the linux and the native windows env ?
how is seen the Windows FS from BASH ?
Matthieu Foll
@mfoll
Sep 23 2016 13:35
Yes exactly. The windows FS from Bash is /mnt/c for the C drive for example
But I don’t think nextflow can work in this FS
Paolo Di Tommaso
@pditommaso
Sep 23 2016 13:36
um no, but it would be possible to handle it in a special manner as you were suggesting
Matthieu Foll
@mfoll
Sep 23 2016 13:37
When the Windows docker execute the jobs he can see the linux filesystem in c:/Users/m_fol/AppData/Local/Lxss/
Paolo Di Tommaso
@pditommaso
Sep 23 2016 13:38
I see
Mike Smoot
@mes5k
Sep 23 2016 21:23
@pditommaso Hi Paolo, I finally took a crack at this ticket: nextflow-io/nextflow#211. Can you take a look at the branch I've got here https://github.com/mes5k/nextflow/tree/log_input_hashes to see if my approach is acceptable? The basic idea is to add an input_hashes field to TraceRecord as a string where the string is just what was being logged before. I've got two concerns with the approach. First, the inputHashesTrace string is being produced every time the TaskProcessor.createTaskHashKey is called, something that was previously only happening when -trace was enabled. I don't have a good sense of whether there might be performance concerns with this. Second, the inputHashesTrace string adds a bunch of newlines, which will mess up the log or trace output. The output actually looks OK as it is, it's just not nicely tab delimited. Maybe this is OK since the input_hashes is an optional field, but maybe not? Apart from that, I think this would make debugging caching issues really helpful. Obviously, no rush on this!