Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jun 10 19:58
    BoPeng opened #320
  • May 28 15:18

    BoPeng on test

    Skip a test (compare)

  • May 28 14:36
    BoPeng commented #319
  • May 28 14:36
    BoPeng closed #319
  • May 28 14:35

    BoPeng on master

    Release sos notebook 0.22.5 (compare)

  • May 28 14:26

    BoPeng on master

    Add a dependency Add toggle class with_console_p… Add toggle class active to pane… and 7 more (compare)

  • May 28 14:25

    BoPeng on console_css

    Remove with_console_panel, whic… (compare)

  • May 28 10:41
    csemken commented #319
  • May 28 10:40
    csemken commented #319
  • May 27 22:17

    BoPeng on stash

    Stash and restore output files (compare)

  • May 27 18:31
    BoPeng commented #319
  • May 27 18:30

    BoPeng on console_css

    Keep displayed settings in note… (compare)

  • May 27 14:22
    BoPeng commented #319
  • May 27 14:21
    BoPeng commented #319
  • May 27 08:46
    csemken commented #319
  • May 27 08:36
    csemken commented #319
  • May 26 23:17
    BoPeng commented #319
  • May 26 23:16

    BoPeng on console_css

    Add a dependency Add toggle class with_console_p… Add toggle class active to pane… and 5 more (compare)

  • May 26 14:39
    BoPeng commented #319
  • May 26 12:54
    csemken commented #319
D. R. Evans
@N7DR

OK... I was thinking that sos_step had to be involved somehow, but I just couldn't see how to make it work. Thank you. I just ran a quick much-simplified test, and it's now working. Marvellous!

I think I should make a couple of comments: the first is that I am very impressed with what SoSW can do... I have replaced about 80% of a complicated and fragile workflow with SoSW, and it's all much faster and cleaner. I'm expecting I'll get 100% replaced, but this problem blocked me for a couple of days.

However, I do think you greatly underestimate the difficulty for someone coming to this never having seen anything like it before. Most of what I do is multithreaded, realtime C++, with a bit of R -- often, although not exclusively for its graphical capabilities -- and then whatever kludgy bash and python scripts are needed to make the programs that do the real work hang together. I believe that when you get down to it, that's a very common paradigm: the baling wire and tape that holds it all together is generally not really designed at all, and one just cobbles together a bunch of scripts to run things in the right order and pass data between programs. While it's a great idea to force the imposition of order and design on the all that, it's not something that's very easy, especially for people who don't use python or bash except in the manner I've just described. So I'm sure that SoSW is all very obvious and even "simple" to you, but to me all this is MUCH harder than writing error-free multithreaded C++ code. It's not that I don't know what I want to do, it's that trying to find out exactly how to do it s very hard. The closest parallel I can think of is writing C++ templates. Basically, if you don't already know how to do it, it's really very difficult to learn, and made more so (as in SoSW also) by the difficulty of debugging things.

Bo
@BoPeng
Thanks for the comment. I am not sure if you are familiar with other workflows systems such as make, snakemake, nextflow, ... and there are 200+ of them just in the biomedical field. They are HARD to lean and require specific format etc so I designed SoS to be "easy", close to "daily workflow", although I might have failed to deliver the goal. I do value your feedback and will continue to improve the documentation of SoS, and hopefully newcomers will benefit from your experience.
D. R. Evans
@N7DR
I learned make in the mid 1980s, when it was dead simple. When I look at a modern makefile, it's incomprehensible to me. I just use the old, original style, and -- fortunately - that still works. I also use a 1984 editor with WordStar keybindings :-) Like old-style makefiles, it ain't broke, does everything I need, and I don't have to think about it. On the other hand, I use C++17 with all the bells and whistles; I suppose I'm just very discriminating on where I decide to spend my time learning something new, because it's all time taken away from getting things done. A new technology really has to look like it will be worth the pain of learning it properly. I do think that there are some fundamental design flaws in Jupyter, which SoSW is stuck with, but SoSW tips the scales so that it's worth learning despite the flaws, which bare Jupyter really doesn't seem to be (to me) -- but I suppose they serve different purposes: Jupyter is very linear and probably fine for expository material, whereas SoSW is something else entirely.
Luca De Cicco
@ldecicco
Hi all. First of all thanks for the amazing job SoS developers have put together, it's really impressive. I am using SoS in lecturing and I sometimes need to mix Matlab and other languages. I have some issues with MATLAB kernel. In particular, if I try to get a quite long array from matlab (%get x --from MATLAB) I got the error "Failed to evaluate 'np.array([0,0.10000000000000000555,0.2000000000000000111,0.30000000000000004441,0.4000000000000000222,0.5,0.60000000000000008882,0.70000000000000006661,0.80000000000000004441,0.9000000000000000222,1,1.100000000000000088 ... " : unexpected EOF while parsing (<string>, line 1)". This does not happen with smaller arrays. In the example above the array has around 300 elements.
Luca De Cicco
@ldecicco
The issue is that the get_response function getting called in kernel.py (sos_matlab) in the put_vars function returns a response['text'] that is not complete (the array gets truncated). I have tried to debug the sos_notebook kernel.py get_response function, but the implementation it is not so simple to understand
D. R. Evans
@N7DR

Am working on a different flow than before, but still having problems understanding parameters and dependencies. Two steps:

[A]
input:
output: f'constant-file-name.txt'
parameter: n_mx = int
bash: expand=True

echo {n_mx} > constant-file-name.txt

[B]
depends: sos_step('A')
input: f'constant-file-name.txt'
output:
parameter: n_mx = int
bash: expand=True

echo cat {_input}

What I expected was that I could call:
%sosrun B --n_mx 60
and SoS would say:

  1. [B] says that it depends on [A], so I'll look at [A].
  2. The parameter is 60, but the last time I ran [A] the value of the parameter was 96, so I must run [A] first with the new value, and then run [B]

But that's not what happens at all. In fact, once [A] has run once, I don't seem to be able to force it to run again no matter how I call [B].

So what do I need to change to the signatures to force [A] and [B] to run twice, once with the values of n_mx = to 96 and once with the value 60? The output filename must stay the same for the two runs, so I can't embed the value of {n_mx} in it (well, I suppose I could create a dummy file in /tmp with the value of {n_mx} in its name, but that's a horrible hack; what is the correct, elegant way to deal with this situation?).

D. R. Evans
@N7DR
I hope it's obvious, but in case not, I guess I should say that the above steps are not the actual ones in the flow; I tried to distill the issue and invent a simple case that demonstrates the issue over which I'm stumbling.
joannfeng
@joannfeng
Since python does not support the char/character type, should I exclude it from the python/javascript tests and update the documentation, or would it be safe to have it as a string of length 1?
Bo
@BoPeng
@ldecicco Sorry that gitter did not send me any notification for whatever reason.
Your report looks a lot like vatlab/sos-matlab#9 , could you update to the latest release of sos-matlab and check again?
@joannfeng When we map from javascript, we map single character and string both to python strings.
And we map python string only to JS string.
Bo
@BoPeng
@N7DR Let us see, with your script
[A]
input:
output: f'constant-file-name.txt'
parameter: n_mx = int
bash: expand=True

  echo {n_mx} > constant-file-name.txt

[B]
depends: sos_step('A')
input: f'constant-file-name.txt'
output:
parameter: n_mx = int
bash: expand=True

  echo cat {_input}
The first time it seems to be ok
 ✗ sos run test B  --n-mx 60
INFO: Running A:
INFO: A is completed.
INFO: A output:   constant-file-name.txt
INFO: Running B:
cat constant-file-name.txt
INFO: B is completed.
INFO: Workflow B (ID=36b8cf2aec411212) is executed successfully with 2 completed steps.
Second time with the same parameter, ok
✗ sos run test B  --n-mx 60
INFO: Running A:
INFO: A (index=0) is ignored due to saved signature
INFO: A output:   constant-file-name.txt
INFO: Running B:
INFO: B (index=0) is ignored due to saved signature
INFO: Workflow B (ID=36b8cf2aec411212) is ignored with 2 ignored steps.
With a different parameter, there seems to be a bug
✗ sos run test B  --n-mx 90
INFO: Running A:
INFO: A (index=0) is ignored due to saved signature
INFO: A output:   constant-file-name.txt
INFO: Running B:
INFO: B (index=0) is ignored due to saved signature
INFO: Workflow B (ID=0403969ddecaa791) is ignored with 2 ignored steps.
which you can for now get around with option -s force (-s signature)
✗ sos run test B  --n-mx 90 -s force
INFO: Running A:
INFO: A is completed.
INFO: A output:   constant-file-name.txt
INFO: Running B:
cat constant-file-name.txt
INFO: B is completed.
INFO: Workflow B (ID=0403969ddecaa791) is executed successfully with 2 completed steps.
Bo
@BoPeng

The case that has been implemented and tested is for parameter: to be defined in the global section,

[global]
parameter: n_mx=int

[A]
input:
output: f'constant-file-name.txt'
bash: expand=True

  echo {n_mx} > constant-file-name.txt

[B]
depends: sos_step('A')
input: f'constant-file-name.txt'
output:
bash: expand=True

  echo cat {_input}

and the signatures are correctly handled

(sos) ➜  demo git:(master) ✗ sos run test B  --n-mx 90 -s force
INFO: Running A:
INFO: A is completed.
INFO: A output:   constant-file-name.txt
INFO: Running B:
cat constant-file-name.txt
INFO: B is completed.
INFO: Workflow B (ID=d91e38138c007847) is executed successfully with 2 completed steps.
(sos) ➜  demo git:(master) ✗ sos run test B  --n-mx 60 -s force
INFO: Running A:
INFO: A is completed.
INFO: A output:   constant-file-name.txt
INFO: Running B:
cat constant-file-name.txt
INFO: B is completed.
INFO: Workflow B (ID=efc47f8da932e0f3) is executed successfully with 2 completed steps.
Bo
@BoPeng
This is now #1372
Bo
@BoPeng
@N7DR @1372 is fixed and I have released sos 0.21.7 for it.
D. R. Evans
@N7DR
When I get a moment I'll figure out how to update sos (easy, I expect, but I haven't done it before); sounds like you've done exactly what's needed, though. Thanks very much.
Bo
@BoPeng
pip install sos -U should do the trick.
D. R. Evans
@N7DR
Thank you! So helpful, sir.
D. R. Evans
@N7DR

I am running afoul of the new warning: WARNING: Embedding script "..." without indentation is error-prone and will be deprecated in the future.

How do I reformat the following step so as not to receive the warning?

[plot_classical_phase_at_same_height]
depends: sos_step('calculate_classical_phase_at_same_height')
input: f'phase-0.txt'
output: f'phase-0.png', f'phase-0.gplt', f'phase-0-diff.png'
bash: expand=True

cat << 'EOF' > phase-0.gplt
set terminal png
set output "phase-0.png"

set key off

#set xrange [180:0]

set title "Direct/Reflected Phase Difference"

set xlabel "log10(d(λ))"
set ylabel "Phase Difference(°)"

#set errorbars small

plot "phase-0.txt" using (log10($6)):14 with lines

set output "phase-0-diff.png"
set ylabel "abs(Phase Difference - 180°)"

plot "phase-0.txt" using (log10($6)):(abs($14-180)) with lines

EOF

gnuplot phase-0.gplt

[I note in passing that I would use the gnuplot kernel directly, except: (1) I don't think SoS supports that kernel; (2) there is a bug somewhere that causes the gnuplot kernel, at least on debian stable, not to handle Unicode correctly: https://github.com/has2k1/gnuplot_kernel/issues/21). In the absence of the gnuplot kernel, I think I have to do something like the above.]

Bo
@BoPeng
Add indentation will suppress the warning. Depending on the editor you use, this can be trivial or quite troublesome.
The warning was added due to vatlab/sos#1363
Bo
@BoPeng
SoS supports all kernels, just that there is no variable exchange for kernels of unsupported languages. https://vatlab.github.io/sos-docs/doc/user_guide/expand_capture_render.html
D. R. Evans
@N7DR
I really don't understand this whole indentation issue, even though you've tried to explain it :-( The cognitive dissonance I experience when having to indent shell code not at all trivial :-( Isn't there some other way to write bash code so that it doesn't have to be indented... it feels like going back to FORTRAN IV :-) How about allowing something like:
D. R. Evans
@N7DR

Stupid thing was in chat mode... I hate the way it does that, basically requiring one to remember to check which mode it's in... anyway, as I was saying, how about allowing something like:

bash: expand=True, end='FRED'

cat << 'EOF' > phase-0.gplt
set terminal png
set output "phase-0.png"

set key off

#set xrange [180:0] etc. etc.

plot "phase-0.txt" using (log10($6)):(abs($14-180)) with lines

EOF
gnuplot phase-0.gplt
FRED

so that one can write the bash script without indentation up until one hits the string 'FRED' on a line by itself. ['FRED', of course, could be any string one likes, defined by the parameter to "end=" on the "bash:" line.]

D. R. Evans
@N7DR

SoS supports all kernels, just that there is no variable exchange for kernels of unsupported languages

I had tried the following:

[C]
input:
output:
parameter:
gnuplot:

plot "phase-0.txt" using (log10($6)):14 with lines

But running the step produced: NameError: name 'gnuplot' is not defined, so I thought that that meant that I couldn't use the gnuplot kernel. What do I need to change in order for it to work? (Although it still wouldn't be usable for many of my plots anyway, because of the Unicode problem.)

Bo
@BoPeng
You are using sos kernel, not gnuplot kernel if there is such a thing. The sh: stuff is a sos function/action. Since did does not provide one, you can use
script:
Bo
@BoPeng
There is actually a gnuplot kernel. https://github.com/has2k1/gnuplot_kernel If you use it, you can use SoS Notebook, and run the script directly in the kernel.

If you are using SoS workerlow, in a SoS cell, gnuplot: action is not defined, but you can try

run:
    #!/bin/env gnuplot
    script...

which will use gnuplot command to run the script. Or you can do

script: args='gnuplot {filename}'
    script ...

to specify the interpreter. see https://vatlab.github.io/sos-docs/doc/user_guide/sos_actions.html#Option--args for details.

D. R. Evans
@N7DR

There is actually a gnuplot kernel. https://github.com/has2k1/gnuplot_kernel

Yes, that's part of what I was trying to get across :-)

I'll try your first suggestion above; that seems cleaner in the absence of explicit support for the gnuplot kernel.

Bo
@BoPeng
I saw a message that was later deleted. I think https://vatlab.github.io/sos-docs/doc/user_guide/sos_actions.html#Option-template-and-template_name was what was asked.
Patrick Cudahy
@pgcudahy

Hello, I'm trying to set up a remote host for my university cluster but having trouble getting started
My hosts.yml is

localhost: macbook
hosts:
  yale_farnam:
    address: farnam.hpc.yale.edu
    paths:
      home: /home/pgc29/scratch60
  macbook:
    address: 127.0.0.1
    paths:
      home: /Users/pgcudahy

But when I run something simple like

%run -r yale_farnam -c ~/.sos/hosts.yml
sh:
    echo Working on `pwd` of $HOSTNAME

I get ERROR: Failed to connect to yale_farnam: pgcudahy@farnam.hpc.yale.edu: Permission denied (publickey).

I also tried sos remote setup but get the error INFO: scp -P 22 /Users/pgcudahy/.ssh/id_rsa.pub farnam.hpc.yale.edu:id_rsa.pub.yale_farnam ERROR: Failed to copy public key to farnam.hpc.yale.edu

Perhaps the issue is that my usernames for my computer and the cluster are different. A simple ssh pgc29@farnam.hpc.yale.edu works for me, but ssh farnam.hpc.yale.edu does not. The scp command referenced in the error message doesn't prepend a username to the cluster's domain name. Any help on how to move forward would be great. Thanks

Bo
@BoPeng
Could you try to change address: farnam.hpc.yale.edu to address: pgc29@farnam.hpc.yale.edu
Patrick Cudahy
@pgcudahy
Ah, thanks. Of course the cluster just went down for maintenance until Wednesday! I'll try it as soon as it's back up.
Bo
@BoPeng
It is a cluster, then you will have to install sos on it, set sos: /path/to/sos in the config file (to avoid changing $PATH on server), and then add a template to submit jobs. I usually have two hosts for headnode and cluster in case I want to run something on the headnode. Let me know if you encounter any problem.
Bo
@BoPeng
BTW -c ~/.sos/hosts.yml is not needed. That file is always read.
Patrick Cudahy
@pgcudahy
Adding my username to the address worked well, thanks! My cluster recommends installing pip modules like sos with anaconda, but I couldn't figure out how to get my local notebook to access sos in the remote conda environment. Instead I was able to install sos with pip install --user sos and then change my PYTHONPATH in ~/.bash_profile
Patrick Cudahy
@pgcudahy

Okay, so now that I can directly run remote jobs, I'm trying to get SLURM set up. The cluster documentation has this example task that I want to replicate

#!/bin/bash
#SBATCH --job-name=example_job
#SBATCH --out="slurm-%j.out"
#SBATCH --time=01:00
#SBATCH --nodes=1 --ntasks=1 --cpus-per-task=2
#SBATCH --mem-per-cpu=5G
#SBATCH --mail-type=ALL

mem_bytes=$(</sys/fs/cgroup/memory/slurm/uid_${SLURM_JOB_UID}/job_${SLURM_JOB_ID}/memory.limit_in_bytes)
mem_gbytes=$(( $mem_bytes / 1024 **3 ))

echo "Starting at $(date)"
echo "Job submitted to the ${SLURM_JOB_PARTITION} partition, the default partition on ${SLURM_CLUSTER_NAME}"
echo "Job name: ${SLURM_JOB_NAME}, Job ID: ${SLURM_JOB_ID}"
echo "  I have ${SLURM_CPUS_ON_NODE} CPUs and ${mem_gbytes}GiB of RAM on compute node $(hostname)"

My updated hosts.yml is now

%save ~/.sos/hosts.yml -f

localhost: ubuntu
hosts:
  ubuntu:
    address: 127.0.0.1
    paths:
      home: /data2
  yale_farnam:
    address: pgc29@farnam1.hpc.yale.edu
    paths:
      home: /home/pgc29/scratch60/
  yale_hpc_slurm:
    address: pgc29@farnam.hpc.yale.edu
    paths:
      home: /home/pgc29/scratch60/
    queue_type: pbs
    submit_cmd: sbatch {job_file}
    submit_cmd_output: "Submitted batch job {job_id}"
    status_cmd: squeue --job {job_id}
    kill_cmd: scancel {job_id}
    status_check_interval: 120
    max_running_jobs: 100
    max_cores: 200 
    max_walltime: "72:00:00"
    max_mem: 1280G
    task_template: |
        #!/bin/bash
        #SBATCH --time={walltime}
        #SBATCH --nodes=1
        #SBATCH --ntasks-per-node={cores}
        #SBATCH --job-name={task}
        #SBATCH --output=/home/{user_name}/.sos/tasks/{task}.out
        #SBATCH --error=/home/{user_name}/.sos/tasks/{task}.err
        cd {workdir}
        {command}

But when I run

%run -q yale_hpc_slurm 

bash:
    mem_bytes=$(</sys/fs/cgroup/memory/slurm/uid_${SLURM_JOB_UID}/job_${SLURM_JOB_ID}/memory.limit_in_bytes)
    mem_gbytes=$(( $mem_bytes / 1024 **3 ))

    echo "Starting at $(date)"
    echo "Job submitted to the ${SLURM_JOB_PARTITION} partition, the default partition on ${SLURM_CLUSTER_NAME}"
    echo "Job name: ${SLURM_JOB_NAME}, Job ID: ${SLURM_JOB_ID}"
    echo "  I have ${SLURM_CPUS_ON_NODE} CPUs and ${mem_gbytes}GiB of RAM on compute node $(hostname)"

It tries to run on my local machine and returns

INFO: Running default:

/tmp/tmp9o84b7hs.sh: line 1: /sys/fs/cgroup/memory/slurm/uid_/job_/memory.limit_in_bytes: No such file or directory
/tmp/tmp9o84b7hs.sh: line 2: / 1024 **3 : syntax error: operand expected (error token is "/ 1024 **3 ")
Starting at Wed Oct  7 04:14:57 EDT 2020
Job submitted to the  partition, the default partition on 
Job name: , Job ID: 
  I have  CPUs and GiB of RAM on compute node ubuntu

INFO: Workflow default (ID=187c6b10c86049c7) is executed successfully with 1 completed step.

I tried

%run -r yale_hpc_slurm

bash:
    mem_bytes=$(</sys/fs/cgroup/memory/slurm/uid_${SLURM_JOB_UID}/job_${SLURM_JOB_ID}/memory.limit_in_bytes)
    mem_gbytes=$(( $mem_bytes / 1024 **3 ))

    echo "Starting at $(date)"
    echo "Job submitted to the ${SLURM_JOB_PARTITION} partition, the default partition on ${SLURM_CLUSTER_NAME}"
    echo "Job name: ${SLURM_JOB_NAME}, Job ID: ${SLURM_JOB_ID}"
    echo "  I have ${SLURM_CPUS_ON_NODE} CPUs and ${mem_gbytes}GiB of RAM on compute node $(hostname)"

and got

ERROR: No workflow engine or invalid engine definition defined for host yale_hpc_slurm

Workflow exited with code 1

Where have I messed up my template? Thanks

Patrick Cudahy
@pgcudahy

Okay, got it to work with

%run -q 'yale_hpc_slurm'

task: walltime='00:05:00'
bash:
    mem_bytes=$(</sys/fs/cgroup/memory/slurm/uid_${SLURM_JOB_UID}/job_${SLURM_JOB_ID}/memory.limit_in_bytes)
    mem_gbytes=$(( $mem_bytes / 1024 **3 ))

    echo "Starting at $(date)"
    echo "Job submitted to the ${SLURM_JOB_PARTITION} partition, the default partition on ${SLURM_CLUSTER_NAME}"
    echo "Job name: ${SLURM_JOB_NAME}, Job ID: ${SLURM_JOB_ID}"
    echo "  I have ${SLURM_CPUS_ON_NODE} CPUs and ${mem_gbytes}GiB of RAM on compute node $(hostname)"

Not sure why I had to quote the remote host name. Also it does not produce a .out file in ~/.sos/tasks/ on the remote computer or my local computer, just job_id, pulse, task and sh files.

Bo
@BoPeng
ok, first, if you add sos: /path/to/sos, you do not need to set up PATH or PYTHONPATH on the cluster. In our experience it is better to leave $PATH on the cluster alone because your job might be using a different Python than the one sos uses.
Then, task is needed to define a portion of the step as external tasks. Using only bash would not work. Basically the template executes the task with sos execute, which can be bash, python, R and any other scripts...
third, the "Starting at $(data)" stuff usually belongs to the template as your notebook would focus on "real" stuff, not anything directly related to the cluster. There is a problem with interpolation of ${ } since sos expands { }, so you will have to use ${{ }} to avoid that.
mem_bytes is read from you job file. Actually sos provides variable mem which is exactly that.
Bo
@BoPeng
Finally, I do not see where you specify mem in your template, is it needed at all for your system?