zenodo/zenodo

Zenodo - Research. Shared. Problem running Zenodo? https://github.com/zenodo/zenodo/wiki/Troubleshooting

Lars Holm Nielsen

@lnielsen

Hi @graeme-winter

Does it happen immediately

Graeme Winter

@graeme-winter

No, after quite a while - makes me think a lot of the data has been uploaded

I acknowledge that this is quite a lot of data to HTTP PUT

Lars Holm Nielsen

@lnielsen

can you provide me with one of the following: user-agent string, bucket id and filename (e.g. file upload url) or ip address

then i can locate the requests in the webserer logs more easily

Graeme Winter

@graeme-winter

sure thing, will see what I can do

OK, so this was the upload which included a failed file https://zenodo.org/deposit/3724425

I think the file was "INS1_3.tar"

Lars Holm Nielsen

@lnielsen

Ok….tahnks…..

will check

Graeme Winter

@graeme-winter

I was running inside a screen session so the logging vanished off the top 🙄

I should add some actual logging to the application...

Lars Holm Nielsen

@lnielsen

Ok….I’ve found the failed request in the logs

Graeme Winter

@graeme-winter

Cool - there is a very good chance that the issue was on the interconnect or at my end... but adding some more resilience would be a good thing I think

Lars Holm Nielsen

@lnielsen

I can see it send a 502 bad gateway from our end

and that it managed to go to ~5GB uploaded first

does it fit that it was uploading for some 15-20 minutes before failing?

Graeme Winter

@graeme-winter

Yeh that fits

anything I can do at my end to be more resilient e.g. restart the upload at the failed point?

Lars Holm Nielsen

@lnielsen

unfortunately we don’t do resumable uploads :S

we’re looking into it and trying to debug it…..right now,it’s just to try again….which is not very nice for a 20 minute upload :(

Graeme Winter

@graeme-winter

I'll leave with you for a little while then - for the moment it seems to be fine with files <= ~ 1GB or so which should cover most of our use case. Thanks for looking into it :)

Graeme Winter

@graeme-winter

@lnielsen we are going now - looks like people are appreciating this too https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=ind2003&L=CCP4BB&D=0&P=190211 - thanks again for your help. Quick question, probably for Monday - currently reading API is in development according to your pages - would it be possible for you to give me some hints how I could make a script which will perform a keyword search and then grab the title / doi / file list for each? This will mean I can liaise with the other data archive people (the protein data bank) to allow linking back to the data sources...

Lars Holm Nielsen

@lnielsen

great news!!! Thanks for keeping us in the loop. If I get a moment during the weekend, I’ll try to send you info on the api

Https://zenodo.org/api/records

Our entire front end goes through this interface

https://help.zenodo.org/guides/search/

https://zenodo.org/api/records?q=

The q Parma is the query from the search guide

Graeme Winter

@graeme-winter

👍 thanks for the starting point - I can figure things out from here I think 🙂 appreciate the quick feedback too!

Will try this out a bit later - got other things on right now ...

Lars Holm Nielsen

@lnielsen

Actually, the easiest is - make your search query on zenodo

e.g. https://zenodo.org/search?page=1&size=20&q=%22COVID-19%22

change https://zenodo.org/search to https://zenodo.org/api/records

https://zenodo.org/api/records/?page=1&size=20&q=%22COVID-19%22

and you got your search query…...

Graeme Winter

@graeme-winter

OK, followed up on this finally - all good, got what I was looking for 🙂 thank you

Graeme Winter

@graeme-winter

Good morning 🙂 now trying to update automatically some of these records to (i) fix a typo in a keyword and (ii) add a couple of extra words to the descriptions - weird thing I am finding is that it works sometimes but now appears to be failing with an invalid action

{'message': 'Invalid action', 'status': 403}
Traceback (most recent call last):
File "/Users/graeme/git/zenodo-uploader/zenodo_updater.py", line 203, in <module>
updater()
File "/Users/graeme/git/zenodo-uploader/zenodo_updater.py", line 198, in updater
zenodo_updater.update()
File "/Users/graeme/git/zenodo-uploader/zenodo_updater.py", line 132, in update
self._update()
File "/Users/graeme/git/zenodo-uploader/zenodo_updater.py", line 106, in _update
raise RuntimeError("in update: HTTP status %d" % r.status_code)
RuntimeError: in update: HTTP status 403

At around 07:13 UTC

Lars Holm Nielsen

@lnielsen

Checking

Graeme Winter

@graeme-winter

👍 thank you

Lars Holm Nielsen

@lnielsen

Do you have the deposit id that it happened to?

Graeme Winter

@graeme-winter

Located deposition: id = 3730327

Lars Holm Nielsen

@lnielsen

Usually this happens if e.g. a record was not put in editing mode, or is already in editing mode…..

e.g. the normal flow would be

/deposit/depositions/:id/actions/edit

Where communities thrive

People

Repo info

Activity