Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Apr 09 00:16

    BenLangmead on master

    embed parsed_md (compare)

  • Apr 08 21:26

    BenLangmead on master

    make more portable (compare)

  • Apr 08 21:18

    BenLangmead on master

    some attempts to make this scri… (compare)

  • Dec 03 2018 15:25
    gianmaz edited #88
  • Dec 03 2018 14:25
    gianmaz opened #88
  • Mar 15 2018 20:05
    ChristopherWilks opened #87
  • Mar 15 2018 20:01

    ChristopherWilks on master

    switched to use sratoolkit 2.8.… (compare)

  • Mar 04 2018 22:50

    nellore on master

    patches bowtie2-build in travis… (compare)

  • Mar 04 2018 22:11

    nellore on master

    uses bowtie2 2.3.4.1 (compare)

  • Mar 04 2018 21:57

    nellore on master

    specifies samtools version to i… (compare)

  • Mar 04 2018 21:47

    nellore on master

    updates dependencies Merge branch 'master' of https:… (compare)

  • Mar 04 2018 21:39

    nellore on master

    quote rules Merge pull request #86 from Ben… (compare)

  • Mar 04 2018 21:39
    nellore closed #86
  • Feb 04 2018 01:11

    nellore on master

    enables --keep-alive for covera… (compare)

  • Jan 10 2018 13:09
    BenLangmead opened #86
  • Dec 29 2017 05:47

    nellore on master

    fixes unit test in bed_pre (compare)

  • Dec 29 2017 03:11

    nellore on master

    writes coverage tsv, optionally… (compare)

  • Dec 28 2017 20:59

    nellore on master

    fixes issue wipassing temp dir … Merge branch 'master' of https:… (compare)

  • Dec 28 2017 20:53

    nellore on master

    fixes issue wipassing temp dir … (compare)

  • Dec 23 2017 00:30

    nellore on master

    fixes misplaced joins in except… (compare)

abhinav
@nellore
ok, good to know
can you try creating a new bucket in the AWS console
and adding one dud lifecycle rule?
Julia di Iulio
@juliadiiulio_twitter
Let me check with our devops team (but I think that's what they did). I'll come back to you as soon as I hear from them!
abhinav
@nellore
great!
Julia di Iulio
@juliadiiulio_twitter
Okay, they confirmed that they did that already
abhinav
@nellore
alright, can i just walk you through hacking rail so it doesn't do this?
Julia di Iulio
@juliadiiulio_twitter
sure! I'll do my best to follow:)
abhinav
@nellore
edit the file /usr/local/raildotbio/rail-rna/dooplicity/ansibles.py so L356 reads if 'NoSuchLifecycleConfiguration' not in errors: rather than if 'NoSuchLifecycleConfiguration' in errors:
save and rerun, and hopefully that particular error will be gone
but then you'll still face the issue that's potentially overtrimming
one way to handle that is that if a read is 100% trimmed, replace it with a single-character read sequence "N" with quality sequence "#"
on the other hand, you may find you don't really need to trim since rail-rna soft-clips alignments
Julia di Iulio
@juliadiiulio_twitter

hmm looks like I still get the error

Parameter validation failed:
Missing required parameter in LifecycleConfiguration.Rules[0]: "Prefix"
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/raildotbio/rail-rna/main.py", line 975, in <module>
ec2_slave_security_group_id=args.ec2_slave_security_group_id
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 6435, in init
secure_stack_name=secure_stack_name)
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 2217, in init
days=intermediate_lifetime)
File "/usr/local/raildotbio/rail-rna/dooplicity/ansibles.py", line 386, in expire_prefix
' '.join(aws_command)
RuntimeError: Error encountered changing lifecycleparameters with command "aws --profile default s3api put-bucket-lifecycle --bucket rail-rna --lifecycle-configuration {"Rules":[{"Status": "Enabled", "ID": "something", "NoncurrentVersionExpiration": {"NoncurrentDays": 365}, "Expiration": {"ExpiredObjectDeleteMarker": true}, "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}}, {"Status": "Enabled", "Prefix": "jdiiulio/storage/analysis/RNAseq/HNX/GNE/ElasticMode/20170816.intermediate/", "Expiration": {"Days": -1}}]}".

abhinav
@nellore
alright better idea
add a new line at the beginning of the function
right before L315
write return
i think AWS changed its API
Julia di Iulio
@juliadiiulio_twitter
just tried and still got the same error :(
abhinav
@nellore
the error didn't change when you put return right before L315?
Julia di Iulio
@juliadiiulio_twitter

It looks the same to me but maybe I am missing something, here it is

Parameter validation failed:
Missing required parameter in LifecycleConfiguration.Rules[0]: "Prefix"
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/raildotbio/rail-rna/main.py", line 975, in <module>
ec2_slave_security_group_id=args.ec2_slave_security_group_id
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 6435, in init
secure_stack_name=secure_stack_name)
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 2217, in init
days=intermediate_lifetime)
File "/usr/local/raildotbio/rail-rna/dooplicity/ansibles.py", line 387, in expire_prefix
' '.join(aws_command)
RuntimeError: Error encountered changing lifecycleparameters with command "aws --profile default s3api put-bucket-lifecycle --bucket rail-rna --lifecycle-configuration {"Rules":[{"Status": "Enabled", "ID": "something", "NoncurrentVersionExpiration": {"NoncurrentDays": 365}, "Expiration": {"ExpiredObjectDeleteMarker": true}, "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}}, {"Status": "Enabled", "Prefix": "jdiiulio/storage/analysis/RNAseq/HNX/GNE/ElasticMode/20170816.intermediate/", "Expiration": {"Days": -1}}]}".

abhinav
@nellore
oh
Julia di Iulio
@juliadiiulio_twitter
oh wait! the "return" was indented... my fault
abhinav
@nellore
no no
forget return for now
remove --intermediate-lifetime -1
from the command
return is supposed to be indented, it's just that return is supposed to exit the function, so my guess is you didn't save
but don't worry about the return now
keep your last change
the addition of the not
and just remove the --intermediate-lifetime -1 from your rail-rna command
i think the trouble is it's now sending an invalid number of days as expiration
Julia di Iulio
@juliadiiulio_twitter
wait it looks like without the --intermediate-lifetime -1 and keeping the "return" and the "not", it woks !!!! youhouhouhouh well I got another error, but at least it started :)
abhinav
@nellore
yay!
what's the other error?
Julia di Iulio
@juliadiiulio_twitter
NameError: global name 'Url' is not defined
but I might have messed the manifest file... let me check
abhinav
@nellore
wow
where are you getting that?
can you paste the full traceback?
Julia di Iulio
@juliadiiulio_twitter
Loading...
Checked all files listed in manifest file.
Copied Rail-RNA and bootstraps to S3.
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/raildotbio/rail-rna/main.py", line 975, in <module>
ec2_slave_security_group_id=args.ec2_slave_security_group_id
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 6477, in init
profile=base.profile))
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 3938, in init
if not Url(assembly).is_s3:
NameError: global name 'Url' is not defined
abhinav
@nellore
oh this is happening because you're not selecting one of hg19, hg38, mm9, mm10, dm3, dm6 as the assembly
can you paste me your rail-rna command
(it's also a bug in the code i'm just making a commit to fix now_
)
Julia di Iulio
@juliadiiulio_twitter
rail-rna go elastic -m s3://rail-rna-pdx/data/RNAseq/manifest_20170808_2samples_railrna.txt -a s3://rail-rna-pdx/Genome/Index/RailRNA/hg38_ERCC92.tgz -o s3://rail-rna-pdx/jdiiulio/storage/analysis/RNAseq/ElasticMode/20170816 --core-instance-type c3.2xlarge --master-instance-type c3.2xlarge -c 8 --region us-west-2 -d idx,tsv,bed,bam,bw,jx --verbose --action-on-failure "TERMINATE_CLUSTER" --master-instance-bid-price 0.15 --core-instance-bid-price 0.15
abhinav
@nellore
ahhh okay you want to use your own assembly
Julia di Iulio
@juliadiiulio_twitter
ya :)