Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
Peter Amstutz
@tetron
and give the imported data a "time to live" so it gets cleaned up automatically
we have some workflows that incorporate steps for data transfer into/out of Arvados, I can check around and see if any of them could be cleaned up and shared
NagarajBM
@NagarajBM

Hi @tetron

do you have shellinabox installed and enabled on shell.b3f2m.zifohpc.com?
Yes I have installed it in shell and it is running.

But still unable to connect to shell. The error was same as mentioned here.

2021/05/12 16:43:27 [error] 22548#22548: 64 connect() failed (111: Connection refused) while connecting to upstream, client: 192.164.1.200, server: webshell.b3f2m.zifohpc.com, request: "POST /shell.b3f2m.zifohpc.com? HTTP/1.1", upstream: "http://192.164.1.133:4200/shell.b3f2m.zifohpc.com?", host: "webshell.b3f2m.zifohpc.com", referrer: "https://workbench.b3f2m.zifohpc.com/".*

Trevor Tanner
@tantrev
I'm trying to download some files from a PGP participant that appears to be on Arvados. Is there any way to download the collection using arvbox? The http links are very slow (i.e. single-digit kb/s) - I just can't figure out how the public shareable link translates to Arvados' Collection ID
Peter Amstutz
@tetron
hi @tantrev
you probably want the Python SDK, not arvbox: https://doc.arvados.org/v2.1/sdk/python/sdk-python.html
SWZ
@swzCuroverse
@tantrev what data in particular were you looking for? The data is hosted on arvados but not in a single collection, and you don't have direct access to the collections via the Arvados APIs. However, I have made some collections that collect all of certain types of PGP data (whole genome, 23 and me, etc) and I can make that available via wget, etc - and that should be the fastest to download.
Ward Vandewege
@cure
@tantrev I see the problem, looking at a cause
Ward Vandewege
@cure
@tantrev configuration problem found and fixed, can you try again?
Trevor Tanner
@tantrev
@cure - that fix did the trick, thank you! @swzCuroverse - I should be good now but thanks for being willing to help, and thanks @tetron for the SDK tip.
NagarajBM
@NagarajBM
Hi
while executing the below command
"arv-keepdocker --pull arvados/jobs latest --project-uuid $project_uuid"
The following error is being displayed. It tries to pull image but after 64MB the job is getting killed
"latest: Pulling from arvados/jobs
Digest: sha256:5ad1019f39bb3284c56e2c084455b582f9a2c2899e56891efdbcedf360a9120e
Status: Image is up to date for arvados/jobs:latest
docker.io/arvados/jobs:latest
2021-05-26 17:26:50 arvados.arv_put[23775] INFO: Resuming upload from cache file /root/.cache/arvados/arv-put/cd68a955ecda6da2e91e9ec31325439f
64M / 281M 22.7% Killed"
Lucas Di Pentima
@ldipenti
Hi @NagarajBM , how much RAM does the machine has where this is running? Maybe it’s being OOMed?
NagarajBM
@NagarajBM
I'm trying that on API server machine. i.e., 1 RAM and 1vCPU (t2.micro)
Ward Vandewege
@cure
@NagarajBM you're unlikely to have a good time on something with 1 vCPU and 1G of RAM, can you get a node that's a little bigger?
(arv-keepdocker is still a python program, and Python... likes RAM. A lot.)
Peter Amstutz
@tetron
@cosmicexplorer howdy
Danny McClanahan
@cosmicexplorer
hello!! this project is incredibly interesting to me
NagarajBM
@NagarajBM
Thanks @cure After increasing the size of the RAM it worked

Using the arvados-cli (arv container delete) command I deleted all the container that was there in my workbench. After that when I try to connect to the workbench I'm getting the below error. Could you please help me with this?

"Oh... fiddlesticks.
Sorry, I had some trouble handling your request.

Path not found (req-4c08pidf4sv5z2qyiy75) [API: 404]"

Ward Vandewege
@cure
@NagarajBM it sounds like workbench is asking for an object that does not exist (any longer?). The req-4c08pidf4sv5z2qyiy75 identifier is a unique identifier for the request that you can use to look in the logs of workbench, arvados-controller, and the api server to see where it went wrong. In this case, if you look in the API server logs and grep for that req-... string, you will find the uuid that is being requested and that no longer exists. What is that uuid?
NagarajBM
@NagarajBM
Thanks for the reply @cure
I was able to solve this by looking in the API server logs as you mentioned.
Peter Amstutz
@tetron
@/all Arvados 2.2.0 is released https://arvados.org/release-notes/2.2.0/
NagarajBM
@NagarajBM

While running a workflow, its getting failed.
In the log these are the error
"socket.gaierror: [Errno -3] Temporary failure in name resolution

httplib2.ServerNotFoundError: Unable to find the server at b3f2m.zifohpc.com

except httplib2.SSLHandshakeError as e:
2021-06-07T05:32:25.364193594Z AttributeError: module 'httplib2' has no attribute 'SSLHandshakeError'"

Peter Amstutz
@tetron:matrix.org
[m]
hmm, I thought we fixed the module 'httplib2' has no attribute 'SSLHandshakeError'" that was a transition from python2 -> 3 mistake, but your real problem is that you don't have working DNS
NagarajBM
@NagarajBM
Thanks for the reply, But while trying to run sample container request mentioned in this document '''https://doc.arvados.org/main/install/crunch2-cloud/install-dispatch-cloud.html'''. Its executes without any error
Peter Amstutz
@tetron
you mean the "confirm working installation" part
?
NagarajBM
@NagarajBM
Yes
joepio
@joepio:matrix.org
[m]
Hi everyone, im looking into arvados as an option for a national scale genomics repository that can integrate with clinical data, i was directes here for 2 questions i have.
  1. How well supported is metadata exploration/visualization supported? (I.e. thinking portals like tcga etc.)
  1. Are people also doing (dynamic) consenting in arvados or what would be a good companion suite to do that?
Peter Amstutz
@tetron
hi @joepio:matrix.org
you can attach metadata to collections of files, and perform searches on it. the base Arvados workbench does not do data visualization, but it is possible to create UI plugins that to add that kind of functionality
9 replies
I am not sure what you mean by dynamic consentinting?
Peter Amstutz
@tetron
*consenting
Popescu Mihai
@mhpopescu
hi! I tried to get used to cwl and Arvados and I run the tutorial from here
https://doc.arvados.org/main/user/tutorials/wgs-tutorial.html
on Arvados playground using workbench
https://workbench.pirca.arvadosapi.com/
I run it with the default values and it seems that it is way bigger than I expected.
SWZ
@swzCuroverse
By default that is the WGS tutorial for full genome, I pointed you to the chromosome 19 version which should be way faster. By default the tutorial should reruse old run jobs, so I am unsure why it ran from scratch. How did you run it?
Popescu Mihai
@mhpopescu

I tried to run first this single chromosome version.
https://github.com/arvados/arvados-tutorial/blob/master/make-singlechromfastq/bam2singlechromfastq.cwl
But I received some errors in the command line and then I tried the tutorial on the workbench.
I set up a new workflow and a collection and then tried to run the process. I recreated some steps (but did not run anything this time) and I believe that here was my mistake and I did not run my workflow, but "WGS processing workflow scattered over samples". Searching for my workflow name did not return anything. This is step 4a.1. I saw now that I can select my workflow from the "All projects" menu, but I can not click on "Next: choose inputs"

image.png

Popescu Mihai
@mhpopescu

The errors that I received in the command line seemed to be because I did not have --local

mhpopescu@shell:~/arvados-tutorial/make-singlechromfastq$ arvados-cwl-runner --api=containers --no-wait --project-uuid pirca-j7d0g-cxau3eh8nkzfegx ./bam2singlechromfastq.cwl ./yml/bam2singlechromfastq-test.yml
INFO /usr/bin/arvados-cwl-runner 2.2.0, arvados-python-client 2.2.0, cwltool 3.0.20210319143721
INFO Resolved './bam2singlechromfastq.cwl' to 'file:///data/home/mhpopescu/arvados-tutorial/make-singlechromfastq/bam2singlechromfastq.cwl'
INFO Using cluster pirca (https://workbench2.pirca.arvadosapi.com/)
INFO Using collection cache size 256 MiB
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/json?all=1: dial unix /var/run/docker.sock: connect: permission denied
ERROR Unhandled error, try again with --debug for more information:
  Docker image arvados/jobs:2.2.0 is not available
Command '['docker', 'images', '--no-trunc', '--all']' returned non-zero exit status 1.

Then the input file seemed to be different and I changed it to another bam file and it worked

mhpopescu@shell:~/arvados-tutorial/make-singlechromfastq$ arvados-cwl-runner --local --api=containers --no-wait --project-uuid pirca-j7d0g-cxau3eh8nkzfegx ./bam2singlechromfastq.cwl ./yml/bam2singlechromfastq-test.yml
INFO /usr/bin/arvados-cwl-runner 2.2.0, arvados-python-client 2.2.0, cwltool 3.0.20210319143721
INFO Resolved './bam2singlechromfastq.cwl' to 'file:///data/home/mhpopescu/arvados-tutorial/make-singlechromfastq/bam2singlechromfastq.cwl'
INFO Using cluster pirca (https://workbench2.pirca.arvadosapi.com/)
INFO Using collection cache size 256 MiB
INFO [container Convert one chromosome of bam to fastqs] pirca-xvhdp-wl80jlfls615628 state is Committed


INFO [container Convert one chromosome of bam to fastqs] pirca-xvhdp-wl80jlfls615628 is Final
ERROR [container Convert one chromosome of bam to fastqs] (pirca-xvhdp-wl80jlfls615628) error log:

  2021-06-09T13:42:07.411181034Z crunch-run crunch-run 2.2.0 (go1.16.3) started
  2021-06-09T13:42:07.411208858Z crunch-run Executing container 'pirca-dz642-vvgabz9f8lm5eco'
  2021-06-09T13:42:07.411216748Z crunch-run Executing on host 'ip-10-254-254-61'
  2021-06-09T13:42:07.525140928Z crunch-run Fetching Docker image from collection '075939ff3efa89db025903129e17fd5e+766'
  2021-06-09T13:42:07.550600109Z crunch-run Using Docker image id 'sha256:54ccdfdac4551c65e07a4c27b1f7a6dd1e092a10f262c2b3636578c73bc6726e'
  2021-06-09T13:42:07.551970897Z crunch-run Docker image is available
  2021-06-09T13:42:07.552160000Z crunch-run Running [arv-mount --foreground --allow-other --read-write --crunchstat-interval=10 --file-cache 9663676416 --mount-tmp tmp0 --mount-by-pdh by_id /tmp/crunch-run.pirca-dz642-vvgabz9f8lm5eco.308699705/keep762011716]
  2021-06-09T13:42:17.089277274Z crunch-run error in Run: While setting up mounts: while checking that input files exist: stat /tmp/crunch-run.pirca-dz642-vvgabz9f8lm5eco.308699705/keep762011716/by_id/df46f3dd9eb77e1cfda3fa2b5c165858+60837: no such file or directory
  2021-06-09T13:42:17.132744568Z crunch-run Cancelled
ERROR Overall process status is permanentFail
INFO Final output collection 282fbb8a925551343e98e762fa1ec1be+57 "Output of bam2singlechromfastq.cwl" (pirca-4zz18-rqsaafyx8mv8ozr)
{}
WARNING Final process status is permanentFail
NagarajBM
@NagarajBM

Which parameter in 'config.yml' describes the EBS storage volume to be allocated for compute nodes that are newly launched in AWS. Is it "IncludedScratch"?

I have mentioned 'includedScratch' as '50GB' in the config.yml. But still when a compute node is launched only 8GB is attached.

Peter Amstutz
@tetron
@NagarajBM no, IncludedScratch is how much disk space is included with the node type (which is 0 for many EBS-only nodes), you want AddedScratch
NagarajBM
@NagarajBM
Thank you @tetron
joepio
@joepio:matrix.org
[m]
Yes fully understand, it's fairly specific, that's why I initially asked if anyone was oing consenting on top of Arvados to see if there are apps out there that integrate
Peter Amstutz
@tetron
we have a project we're working on to develop a lab sample tracking application on top of Arvados, which has a notion of patients and studies, but doesn't currently have separate user interface for consent, it is being designed from the researcher's point of view and assumes consent has already been obtained
Alexander (Sasha) Wait Zaranek
@awz
And, separately to the sample tracker, we have an application that manages the Personal Genome Project (and the consent process there) so this is an area where we do have interest.
joepio
@joepio:matrix.org
[m]
@awz where can I find a demo/code or more about that application?
Peter Amstutz
@tetron
@joepio:matrix.org perhaps the best way for us to answer all your Arvados / Personal Genome Project / consent / etc questions would be to have a call? if you email info@curii.com we can find a time that works