Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Colin Sippl
@cs-ubr

@ntarocco No worries :). Now I understand your approach much better. Thanks a lot! I'll see if I can recreate this based on the ILS approach. So, basically this should work without touching the DB as problematic circular reference errors are avoided within the resolver by stripping the problematic $ref field which points into the wrong direction.

However, storing relations between records in the DB allows you to describe them with "relation types" (i.e. attributes), e.g. "multipart_monograph" or "series" etc. Therefore, I guess, it would make sense to incorporate this implementation as well. But I‘d better wait until Invenio-PIDRelations is released, right?

As for the indexing, I'll handle this later, but many thanks for the hint! It's good to keep that in mind :)

Nicola
@ntarocco

@cs-ubr well, I don't think there are plans to release invenio-pidrelations as official module, even if we are using it in ILS and it is also used in Zenodo. invenio-pidrelations tries to solve the problem of representing a graph of relations between records in a data structure that can be stored somewhere, in the db. In ILS, we had to decide for what to use pid relations and for what not to use it. It is a bit long to explain, but quickly with an example: I have a record A that has language en, and I have a very similar record (but it must be another record) that has language es. I can actually have multiple languages, let's say 10 similar records that are very similar but the language is different (and therefore some metadata). When resolving the $ref relations, you can:

  1. choose to store somehow all related record ids in record A and know which records to fetch, but this can problematic to mantain and keep up-to-date (on every relation change, you will have to update n records and relations)
  2. if you don't store related ids, you can decide to instead search in ES, with a query that allows you to find all related records. I would consider this feasible but probably weak and fragile
  3. if you don't store related ids, you can instead decide to store them in the DB to have a single and reliable source of truth

Basically, here was our choice: when we have n <-> n relations, we decided to and it might make sense for you too to go for something similar to invenio-pidrelations (we called this siblings), when we have 1<->n (more a parent-child relation) that you can simly store in the parent id in the child record. You can for sure come up with more solutions, we decided to go with this to balancing performance and data integrity.
I hope it helps!

Bigyan
@bigyan_gitlab
I’m getting the following error when installing invenio-migrator[loader] with invenio 3.1.1. Any ideas how to fix this?
(invenio-test) ➜  invenio-test git:(develop) pip install "invenio-migrator[loader]"
...
ERROR: invenio 3.1.1 has requirement invenio-celery<1.1.0,>=1.0.1, but you'll have invenio-celery 1.1.1 which is incompatible.
ERROR: invenio-celery 1.1.1 has requirement celery<4.3.0,>=4.2.1, but you'll have celery 3.1.26.post2 which is incompatible.
…
Bigyan
@bigyan_gitlab
(…continuing from above) I successfully installed it using pip install "invenio-migrator[loader]=1.0.0a9"
However dumps command is not installed (is not listed when running invenio —help)for me to run $ invenio dumps loadrecords /path/to/records_dump_0.json?
Olivier DOSSMANN
@blankoworld
@diegodelemos : 2 month ago we spoke about docker-invenio in debian-slim version and cookiecutter-invenio-instance to add lighter images for "get started" tutorial, I created a PR here: inveniosoftware/cookiecutter-invenio-instance#207 .
Nevertheless I encount some problem while building the "docker-compose.full.yml" image. I'm working on it ;)
Pablo Panero
@ppanero
Does anybody have some sort of deployment with different nodes for indexing and search? Like having dedicated nodes for records creation, so in case the load is quite high, the search nodes will still keep serving? (Assuming the ES cluster keeps up and the bottleneck is in the web apps)
mb-wali
@mb-wali

hello guys. i have configured sendmail with ssmtp in my virtual server i can send mails with this command . so it means the SMTP is working.

echo "Testing" | mail -s "Test Email" test@fomain.com

how can i sendmails from my invenio instance using this Mail client of sendmail???

Zacharias Zacharodimos
@zzacharo
@mb-wali invenio is using invenio-mail which is a wrapper of flask-mail. So in the latter's docs you have a config https://github.com/mattupstate/flask-mail/blob/1709c70d839a7cc7b1f7eeb97333b71cd420fe32/flask_mail.py#L560 to define your server...
I think this will work...
mb-wali
@mb-wali
@zzacharo thanks i got it.
Sébastien Délèze
@sebastiendeleze

Hi all,
after upgrading Invenio from 3.1 to 3.2, I receive this error when I run the tests:

FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/w6/cscc82qj1wn6ngbf32v19hf00000gq/T/tmp8bczybr0/static/dist/manifest.json'

It's related to webpack and seems to look to manifest.json file in the wrong folder. Someone already faced this issue ?

Zacharias Zacharodimos
@zzacharo
@sebastiendeleze did you run the webpack build? I would guess there is an updated bootstrap script insise your scripts folder?
Sébastien Délèze
@sebastiendeleze

@zzacharo Thanks for you answer. Yes I did it and it generates the files in the right folder (in my virtual env), but when I launch the tests the directive :

{{ webpack['theme.css'] }}

is looking into a temporary folder as I mentioned before.

Bigyan
@bigyan_gitlab

Hi. I’m currently trying out invenio-records-files.

The code from https://invenio-records-files.readthedocs.io/en/latest/usage.html#creating-a-record says

>>> from invenio_records_files.api import Record
>>> record = Record.create({})
>>> len(record.files)
0

But when I run it I get a

In [21]: from invenio_records_files.api import Record

In [22]: record = Record.create({})

In [23]: len(record.files)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-23-b6239e85b022> in <module>
----> 1 len(record.files)

TypeError: object of type 'NoneType' has no len()

I already have a default location set.

In [11]: from invenio_files_rest.models import Location

In [12]: Location.all()
Out[12]: [default]

What am I doing wrong? Currently running invenio==3.1.1 and invenio-records-files==1.0.0a11

Nicola
@ntarocco
@bigyan_gitlab it is not easy to understand from the code you have copy/pasted, because the commands you run are exactly the same :)
In any case, for file support, the invenio 3.2 release which should happen very soon will have files support and related documentation.
You can already try it by using invenio=3.2.0a9 (inveniosoftware/invenio@3cea80c) until is released as official
Zacharias Zacharodimos
@zzacharo
@sebastiendeleze sorry for the late response, did you fix your problem or you still have troubles launching the tests?
Bigyan
@bigyan_gitlab

I am unable to load fonts in the browser. The browser console says

Refused to load the font 'data:font/woff;base64,d09GRgABAAAAAGz8ABEAAAAA09gAAQABAAAAAAAAAAAAAAAAAAAAAAAAAABHREVGAAABgAAAAC8AAAA0AsQC9UdQT1MAAAGwAAATuAAANLwBEyF1R1NVQgAAFWgAAAIWAAAEZqfk0PVPUy8yAAAXgAAAAFAAAABgaNCCw2NtYXAAABfQAAABkwAAAkQk8AV7Y3Z0IAAAGWQAAABiAAAAugGiQq9mcGdtAAAZyAAABZcAAAvNb3/BHGdhc3AAAB9gAAAACAAAAAgAAAAQZ2x5ZgAAH2gAAESvAAB8yu28l3FoZWFkAABkGAAAADYAAAA2BmibVWhoZWEAAGRQAAAAIAAAACQHMQRzaG10eAAAZHAAAAJDAAAEImBmMbxsb2NhAABmtAAAAhoAAAIaflxdR21heHAAAGjQAAAAIAAAACACjgzgbmFtZQAAaPAAAACdAAABKBQEL8lwb3N0AABpkAAAAsMAAAS9pi3QFXByZ...w76a3jVVUpJzXkBsRtNQoHWTV2mt2UusrulbnIrkvAXNBDFtTVIB8Uoau4pSruq4q7qq2dHpQADUAT0IJ5ra0yPUAfMACMFY6pOtegV/9D7UtTZx72tTeXI4JdcUXh7Pb67D7I/S05AwjAAiYsNie6WOwc4MiYCORSEx+ZExuCvQpiNSRmAdL8wDs2AslUOgp8HfnSYyfCYjrE7w8QDucyS0aXjH0zGk7FX991RgON6L7Qma6pQ+SzA0Qw1x9+HgNFtrBk+F9RsmDpTShvNJL4BDWtP8IAeAFj8N7BcCIoYiMjY1/kBsadHAwcDMkFGxnYnLZFMFgwMLAyaIE4DjzeHPYs+mzKLOIsrBxQoVA2VyZzFk0mWSawELfTPmEGAQYeBk4GNpBGTqCYgNM+BgcYhIgxM7hsVGHsCIzY4NARsZE5xWWjGoi3i6OBgZHFoSM5JAKkJBIIHHh8ORxZDNlUWSRZWHm0djD+b93A0ruRicFlA1vcRtYUFwBQJimV' because it violates the following Content Security Policy directive: "default-src 'self' 'unsafe-inline' 'unsafe-inline'". Note that 'font-src' was not explicitly set, so 'default-src' is used as a fallback.

Any way to fix this?

Nicola
@ntarocco
@bigyan_gitlab by default invenio comes with strict CSP policies. You can tune it by changing its config: https://github.com/inveniosoftware/invenio-app/blob/master/invenio_app/config.py#L115
This config is the one provided by flask-talisman
You can read more about CSP here
Here how we are going to provide this config in Invenio 3.2 (hopefully released soon) to fix this issue
Sébastien Délèze
@sebastiendeleze
@zzacharo Hi, no unfortunately the problem is not yet solved.... I will continue to investigate
Zacharias Zacharodimos
@zzacharo
@sebastiendeleze just to understand you migrated an existing instance in 3.2 or you cookiecuttered a new instance?
Mirek Simek
@mesemus
Hello folks, I would like to ask if you are planning any hands-on meeting the next year - the spring meeting was extremely helpful and I and my colleagues would definitely like to come for another one. Thanks. (the reason I am asking so early is because we have to plan the expenses for the next year)
Jose Benito Gonzalez Lopez
@jbenito3

Hello folks, I would like to ask if you are planning any hands-on meeting the next year - the spring meeting was extremely helpful and I and my colleagues would definitely like to come for another one. Thanks. (the reason I am asking so early is because we have to plan the expenses for the next year)

Hi @mesemus ! Thanks for reaching out. It would be great to do another similar hands-on meeting, it was a very rich experience for us at CERN as well. We haven’t discussed anything related yet because at the moment our full focus is on delivering InvenioRDM by summer 2020. We will discuss internally…and if this is a shared feeling among many people in the community, we should really see when/how/where/what we could do ASAP (but probably after InvenioRDM release).

Sébastien Délèze
@sebastiendeleze
@zzacharo We migrated an existing instance from 3.1 to 3.2. But we found the problem. After removing some instructions on setup.py, it works well, now:
    setup_requires=[
        'pytest-runner>=3.0.0,<5',
    ],
    tests_require=[
        'pytest-invenio>=1.0.0,<1.1.0',
    ]
Zacharias Zacharodimos
@zzacharo
ok I was about to tell you to check the differences with the latest cookiecutter module...:)
Sébastien Délèze
@sebastiendeleze
That was I did ;-)
Zacharias Zacharodimos
@zzacharo
ok! But did you understand what was the problem?
also what is your actual invenio version that you try to migrate to?
Sébastien Délèze
@sebastiendeleze
Yes, the problem was if I ran the tests with pytest it worked correctly, but if I ran sh run-tests.sh, setup.py affected the version of pytest-invenio, due to instructions above.
The current version we use is 3.2.0a9
mb-wali
@mb-wali

Hello guys, i am trying to achieve file upload with invenio-files-rest, i have installed invenio-files-rest==1.0.0 on invenio 3 instance. after installing the module, it was able to create db tables and blueprints for admin. BUT no blueprints are registered for

http://localhost:5000/files
`

can't upload or retrieve file with REST. could anyone guide me please how to register these rest endpoints?
and i am confused how to point the buckets to the records. should i create a new field of array in jsonschema of records like files:[] and add the buckets?
and if i got it correct it should be at least one default location to create a bucket, and one bucket can hold multiple objects(files)? True?

Nicola
@ntarocco
@mb-wali you can read more about it in the documentation: https://invenio-files-rest.readthedocs.io/en/latest/usage.html#rest-apis
since you are directly using files-rest, this is registered under the API endpoint
so, your url will something like http://localhost:5000/api/files
you have example in the documentation
mb-wali
@mb-wali
@ntarocco thanks!
Jose Benito Gonzalez Lopez
@jbenito3
@mesemus we’ve been discussing internally and we will orgnise another bootcamp-style event in the second half of 2020; so secure your budget for it :)
Pablo Panero
@ppanero

Hello!

There is been some initiative to define, design and implement some files processing for Invenio. The design of how it might look can be found on inveniosoftware/rfcs#14. I recall some of you were interested in full-text extraction and indexing, and that is exactly one of the first use cases being considered.

@mhaya I beleive you had this use case?
Suyoj Man Tamrakar
@suyojman
Hello all,
I am using invenio 3.1.1 and I want to show the recent uploads in the homepage.How do i implement it? I am new so please be kind :)
Bigyan
@bigyan_gitlab

@bigyan_gitlab by default invenio comes with strict CSP policies. You can tune it by changing its config: https://github.com/inveniosoftware/invenio-app/blob/master/invenio_app/config.py#L115
This config is the one provided by flask-talisman
You can read more about CSP here
Here how we are going to provide this config in Invenio 3.2 (hopefully released soon) to fix this issue

@ntarocco I added the APP_DEFAULT_SECURE_HEADERS to my config.py and added a font-src key in content_security_policy.

APP_DEFAULT_SECURE_HEADERS = {
    'force_https': True,
    'force_https_permanent': False,
    'force_file_save': False,
    'frame_options': 'sameorigin',
    'frame_options_allow_from': None,
    'strict_transport_security': True,
    'strict_transport_security_preload': False,
    'strict_transport_security_max_age': 31556926,  # One year in seconds
    'strict_transport_security_include_subdomains': True,
    'content_security_policy': {
        'default-src': ["'self'"],
        'object-src': ["'none'"],
        'font-src': ["'self'", "data:", "https://fonts.gstatic.com", "https://fonts.googleapis.com"]
    },
    'content_security_policy_report_uri': None,
    'content_security_policy_report_only': False,
    'session_cookie_secure': True,
    'session_cookie_http_only': True
}

But I’m still getting a

Refused to load the stylesheet 'https://fonts.googleapis.com/css?family=Source+Sans+Pro|Source+Serif+Pro&display=swap' because it violates the following Content Security Policy directive: "default-src 'self' 'unsafe-inline' 'unsafe-inline'". Note that 'style-src-elem' was not explicitly set, so 'default-src' is used as a fallback.` in the browser console.

Just to try out things, I replaced ’self’ in default-src with * and it works:

    'content_security_policy': {
        'default-src': ["'*'"],
        'object-src': ["'none'"],
        'font-src': ["'self'", "data:", "https://fonts.gstatic.com", "https://fonts.googleapis.com"]
    },

Is there a syntax error with 'font-src': ["'self'", "data:", "https://fonts.gstatic.com", "https://fonts.googleapis.com”]? Do I have to write a policy for stylesheet (Refused to load the stylesheet)? How do I do that?

Bigyan
@bigyan_gitlab

@ntarocco It is now solved. I did a

    'content_security_policy': {
        'default-src': ["'self'"],
        'object-src': ["'none'"],
        'font-src': ["'self'", "data:", "https://fonts.gstatic.com", "https://fonts.googleapis.com"],
        'style-src': ["'unsafe-inline'","'self'", "https://fonts.gstatic.com", "https://fonts.googleapis.com"]
    },

It does not work with just the style-src… I’m still not clear why I need both the style-src and font-src?

Bigyan
@bigyan_gitlab

.
I’m facing a weird problem!!

I created a bucket and used invenio-files-rest to upload a file to the bucket. The url to access the resource is now https://localhost:5000/api/files/bee41da0-d59c-42e9-8e9d-ff5294e70543/cover.png. I can access the url and all is going well. Then all of a sudden I start getting a 404 Not Found for the same url. Then after some random interval I can access the url again! ...Not just that, sometimes the url is accessible from a browser and not from a different browser at the exact same time! Any ideas? This is happening both on my local machine and on a server.

(Running Inveino 3.1.1 with docker-compose.yml)

Nicola
@ntarocco

@ntarocco It is now solved. I did a

    'content_security_policy': {
        'default-src': ["'self'"],
        'object-src': ["'none'"],
        'font-src': ["'self'", "data:", "https://fonts.gstatic.com", "https://fonts.googleapis.com"],
        'style-src': ["'unsafe-inline'","'self'", "https://fonts.gstatic.com", "https://fonts.googleapis.com"]
    },

It does not work with just the style-src… I’m still not clear why I need both the style-src and font-src?

@bigyan_gitlab style src vs font-src. I know this can be quite annoying, but that's the CSP standard checked by the browser, nothing to do with Invenio....

.
I’m facing a weird problem!!

I created a bucket and used invenio-files-rest to upload a file to the bucket. The url to access the resource is now https://localhost:5000/api/files/bee41da0-d59c-42e9-8e9d-ff5294e70543/cover.png. I can access the url and all is going well. Then all of a sudden I start getting a 404 Not Found for the same url. Then after some random interval I can access the url again! ...Not just that, sometimes the url is accessible from a browser and not from a different browser at the exact same time! Any ideas? This is happening both on my local machine and on a server.

(Running Inveino 3.1.1 with docker-compose.yml)

Do you have any permission set? Are you logging it / logging out? There is a "security" measure in invenio-files-rest: if you are not authorized you will get a 404 and not a 403 to hide the fact that the file exists in your system

Bigyan
@bigyan_gitlab

@ntarocco Thanks! I was indeed logged in as an admin and logged out respectively during the 200 and 404 responses! :) Turns out invenio-files-rest, by default, only allows access when logged in as an admin. I added a quick

allow_all = lambda *args, **kwargs: type('Allow', (), {'can': lambda self: True})()
FILES_REST_PERMISSION_FACTORY = allow_all

in my config.py and now the files are accessible without logging in.

I will look at Invenio-Files-REST permissions and Invenio-Access properly and see if I can figure out how to grant Read-Only access to non-admin/logged-out users.
Thanks again.

Masaharu Hayashi
@mhaya
@ppanero Thanks a lot. Yes, we have this use case. I’ll check it.
Nicola
@ntarocco
@bigyan_gitlab you can find the answer to your questions in the doc that will be released with the files bundle here and here
Bigyan
@bigyan_gitlab
Are there any reasons why the elasticsearch container would restart when querying elasticsearch?
Pablo Panero
@ppanero
Hi @bigyan_gitlab sometimes if your machine is not "poweful" enough the container dies by OOM