zenodo/zenodo

Zenodo - Research. Shared. Problem running Zenodo? https://github.com/zenodo/zenodo/wiki/Troubleshooting

Thomas Morrell

@tmorrell

Not really, see zenodo/zenodo#1606. You can pull from some successful examples like https://github.com/swcarpentry/shell-novice/blob/gh-pages/.zenodo.json or use a tool like https://github.com/citation-file-format/cff-converter-python

Lordjette Leigh Lecaros

@lordjette

Hello, I just finished publishing Journal (pdf) and Presentation doc in with meta data in my local. But how how can I expose this in frontend to search a result? I tried to run:

zenodo index run

Kindly help, thanks. Please see image
https://ibb.co/t2ZyjKf
https://ibb.co/5T174NT

Lars Holm Nielsen

@lnielsen

only things accepted in the zenodo community appear on the frontpage

Lordjette Leigh Lecaros

@lordjette

hello @lnielsen , is this means only accepted/published can be seen in the frontpage? I have published a journal doc in zenodo community. but, it can't be seen in recent upload frontpage. see images:
https://ibb.co/9hbYQ2Z
https://ibb.co/t2ZyjKf

Lars Holm Nielsen

@lnielsen

yes, only the frontpage

search shows everything

Lordjette Leigh Lecaros

@lordjette

I have 3 published article in a community but still doesn't show in front https://ibb.co/5T174NT

Lars Holm Nielsen

@lnielsen

the search query is being performed by this code: https://github.com/zenodo/zenodo/blob/da91693ef7e3f857668a0d2ffae9089d9c097849/zenodo/modules/frontpage/api.py#L33-L45

Lordjette Leigh Lecaros

@lordjette

@lnielsen thanks.

Kai Woerner

@Herrner

Deleting records and files through the api is only allowed for unpublished depositions (so say the docs) - is there any other way to delete published records and their files programatically (for an admin user)?

<context>we have a sandbox(staging machine that we would like to "clean up" from time to time - so no DOI harm done.</context>

Lordjette Leigh Lecaros

@lordjette

Hello, I have found there is no "stats":{} shows in page after published and accept record in a community. I tried to run celery along zenodo run - h 1.2.3.4

Lordjette Leigh Lecaros

@lordjette

I tried 3 different users to download and view record but it does not appear in page.

Graeme Winter

@graeme-winter

Hi Folks - talked to Lars last week - Graeme from Diamond Light Source who is working on a tool to allow folks to automatically upload COVID-19 related data sets - for small files works fine using the new stream-type API, but for larger files I am getting network errors sometimes - ``` File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20200323/conda_base/lib/python3.6/site-packages/requests/api.py", line 134, in put
return request('put', url, data=data, kwargs)
File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20200323/conda_base/lib/python3.6/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, kwargs)
File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20200323/conda_base/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, send_kwargs)
File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20200323/conda_base/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, kwargs)
File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20200323/conda_base/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', OSError("(32, 'EPIPE')",))

Lars Holm Nielsen

@lnielsen

Hi @graeme-winter

Does it happen immediately

Graeme Winter

@graeme-winter

No, after quite a while - makes me think a lot of the data has been uploaded

I acknowledge that this is quite a lot of data to HTTP PUT

Lars Holm Nielsen

@lnielsen

can you provide me with one of the following: user-agent string, bucket id and filename (e.g. file upload url) or ip address

then i can locate the requests in the webserer logs more easily

Graeme Winter

@graeme-winter

sure thing, will see what I can do

OK, so this was the upload which included a failed file https://zenodo.org/deposit/3724425

I think the file was "INS1_3.tar"

Lars Holm Nielsen

@lnielsen

Ok….tahnks…..

will check

Graeme Winter

@graeme-winter

I was running inside a screen session so the logging vanished off the top 🙄

I should add some actual logging to the application...

Lars Holm Nielsen

@lnielsen

Ok….I’ve found the failed request in the logs

Graeme Winter

@graeme-winter

Cool - there is a very good chance that the issue was on the interconnect or at my end... but adding some more resilience would be a good thing I think

Lars Holm Nielsen

@lnielsen

I can see it send a 502 bad gateway from our end

and that it managed to go to ~5GB uploaded first

does it fit that it was uploading for some 15-20 minutes before failing?

Graeme Winter

@graeme-winter

Yeh that fits

anything I can do at my end to be more resilient e.g. restart the upload at the failed point?

Lars Holm Nielsen

@lnielsen

unfortunately we don’t do resumable uploads :S

we’re looking into it and trying to debug it…..right now,it’s just to try again….which is not very nice for a 20 minute upload :(

Graeme Winter

@graeme-winter

I'll leave with you for a little while then - for the moment it seems to be fine with files <= ~ 1GB or so which should cover most of our use case. Thanks for looking into it :)

Graeme Winter

@graeme-winter

@lnielsen we are going now - looks like people are appreciating this too https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=ind2003&L=CCP4BB&D=0&P=190211 - thanks again for your help. Quick question, probably for Monday - currently reading API is in development according to your pages - would it be possible for you to give me some hints how I could make a script which will perform a keyword search and then grab the title / doi / file list for each? This will mean I can liaise with the other data archive people (the protein data bank) to allow linking back to the data sources...

Lars Holm Nielsen

@lnielsen

great news!!! Thanks for keeping us in the loop. If I get a moment during the weekend, I’ll try to send you info on the api

Https://zenodo.org/api/records

Our entire front end goes through this interface

https://help.zenodo.org/guides/search/

https://zenodo.org/api/records?q=

The q Parma is the query from the search guide

Graeme Winter

@graeme-winter

👍 thanks for the starting point - I can figure things out from here I think 🙂 appreciate the quick feedback too!

Will try this out a bit later - got other things on right now ...