Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    wgresshoff
    @wgresshoff
    I'm in home office now with a freshly installed notebook. kubectl, helm and docker installed. I've got a namespace in our Kubernetes Cloud now, so I can test helm installations. I just don't know where to start. I think it's best to start with some standard components. Where do I find information what images are used by Invenio? It's missing from the invenio-helm documentation.
    Pablo Panero
    @ppanero
    This message was deleted
    @wgresshoff by images you mean the services (redis, rabbitmq, postgresql, etc.) images?
    wgresshoff
    @wgresshoff
    Yes, I think they are the best to start development of the helm charts. In the openshift version they are documented as provided (not in cluster), but for me they are needed cause they are missing here.
    Pablo Panero
    @ppanero

    Documentation for all this is not yet finished (will be for this release, tho we do not have yet a full description of the services). It will improve over time. The services, even if they are marked as "not in cluster" they are in the services folfer here. For example, you can find ES and its version/image here, the same for PostgreSQL here

    As for the versions, we are compatible with the same than Invenio: ES 6 or 7, PostgreSQL 9.4+

    Wow, nice markdown there on quoting + link on Invenio :joy:
    wgresshoff
    @wgresshoff
    Ah, great, thanks! I had a look in one file or two, and when I saw they are services I closed them again :) This is a good starting point.
    wgresshoff
    @wgresshoff
    I started with PostgreSQL. Defined a secret (just replaced oc by kubectl). worked. In the Deployment I had to change the type from Openshift to Kubernetes and add a selector. So PostgreSQL get's installed and is working.
    That's it for today :)
    fenekku
    @fenekku

    Daily Notes
    Today

    Tomorrow

    • Iterate on feedback
      • need to make a script to fill out the default resource_types
    • Implement template (filters)
    • Review Sara's article on InvenioRDM user testing we did
    Pablo Panero
    @ppanero

    In the Deployment I had to change the type from Openshift to Kubernetes and add a selector. So PostgreSQL get's installed and is working.

    @wgresshoff Nice news! Are you storing that code somewhere? Would be nice to check it and potentially move it to the helm-invenio repo :D

    wgresshoff
    @wgresshoff
    @ppanero I'm far away even from a beta. But I forked the invenio-helm-repo to my account. It should be accessible. In the moment I try to get nginx and haproxy running, I don't know what helm does when it finds unknown tags in the yaml (it's not enabled (filebeat), so I hope helm accepts it). I must find a way to get around the BuildConfig.
    Pablo Panero
    @ppanero
    so should not cause any troubles
    Perfect for the fork, thanks :)
    wgresshoff
    @wgresshoff
    Actually I'm having trouble with the PersistentVolumeClaim. That's fine when defining the non default storageClass (the default one in our cluster ist not capable of accessMode ReadWriteMany).
    Pablo Panero
    @ppanero
    Ah I see

    Sorry forgot to notify we changed that today, if you check master now its just an emptyDir volume with nginx running as sidecar (simpler) --> https://github.com/inveniosoftware/helm-invenio/blob/master/invenio/templates/application.yaml#L179

    we plan to release the helm-chart with the the march release so from then it will be stable. Sorry about this, I do not expect bigger changes from now. That was the last one.

    wgresshoff
    @wgresshoff
    There are quite a lot of dependencies. nginx can't start without web. The image is defined in values.yaml as your/invenio-image. I guess it's an image with invenio installed, which I should create before I can call the helm chart?!
    @ppanero I will merge the commits of the upstream repository and have a look.
    Pablo Panero
    @ppanero
    Yeah the image is the image of your RDM instance, we are still seeinf if/how to provide a default one. On how to build your own, there are a few PRs in cookiecutter and the docs. If you can wait till the release, we will make all this clearer so is less painfull for you with all this dependencies and blockings
    wgresshoff
    @wgresshoff
    I'd like to be as near to the release as possible. You're right, I should wait for the release (when is it, btw?). In the meantime I'll try to find solutions for the Openshift specialities.
    @ppanero If you want to have a look, my fork is merged now.
    wgresshoff
    @wgresshoff
    After a look in the Openshift documentation I think there is no need for the imagestream in the kubernetes version, because the container user is special to Openshift. I'll try filebeat without it.
    wgresshoff
    @wgresshoff
    @ppanero The last missing pieces now are the route and the images for web and worker deployments. I think I can replace the route by an ingress definition but first I must read some documentation.
    Pablo Panero
    @ppanero
    @wgresshoff great news! Thanks a lot for the efforts! I will check the fork :)
    about the release, end of the month, so latest before the next week telecon :D
    Pablo Panero
    @ppanero

    Daily Notes
    Today

    • Review and worked a bit more on Helm charts. Should be stable now.
    • Improvements in UI of RDM demo site and default installation site (submitted a few PRs)
    • Merged documentation.

    Later on/tomorrow

    • Release what's possible
    • Start testing
    fenekku
    @fenekku

    Daily Notes
    Today

    • PR reviews (per usual)
    • metadata working group meeting
    • custom vocabulary updates inveniosoftware/invenio-rdm-records#82
      • Fill out the default resource_types.csv
      • display resource type on template
      • validate resource type as follows:
        if resource type has subtypes, then a subtype must be passed and be part of them.
        (adding an "Other" subtype should fix cases where only a type was allowed before.)
      • what is missing now is a full manual test which will uncover issues I am sure
    • Review Sara's article on InvenioRDM user testing we did

    Tomorrow

    • full manual test
    wgresshoff
    @wgresshoff
    @ppanero I added an Ingress definition and created a pull request. Possible next steps:
    • add documentation
    • integrate the charts, perhaps it's possible to replace Openshift DeploymentConfig by pure K8s Deployment? It's easier to maintain if there is only one definition.
    • add a value to decide on Openshift / K8s installation
    Pablo Panero
    @ppanero
    @wgresshoff Amazing! so it is working on K8?
    Yes, for this release is a bit tight. But for the next one we can evaluate if there is a need for DeploymentConfig or we can just do with Deployment and consolidate the templates
    Thanks!
    :rocket:
    wgresshoff
    @wgresshoff
    @ppanero The last bits are not tested as I don't know how to create the images for web and workers. The (outconfigured) part logstash is not tested as well, cause in the docs the needed secret is missing. So the K8s part should be considered as preview, not as stable.
    Nicola
    @ntarocco
    :rocket: very good news! @ppanero congrats for the successfull Invenio sprint then (send congrats to Diego too ;))
    Pablo Panero
    @ppanero

    @wgresshoff how to build the image is documented in the repo (will be available in the official docs site when we release). However, you can "build" te future docs with the steps of the readme:

    $ mkvirtualenv docs-invenio-rdm
    $ pip install -r requirements.txt
    $ mkdocs serve
    $ firefox http://127.0.0.1:8000

    It might be easier to navigate through them :)

    The (outconfigured) part logstash is not tested as well, cause in the docs the needed secret is missing.

    Sorry, I didn't get this part. Which secret do you mean?

    Once again thanks for the efforts!

    @ntarocco Thanks a lot, to you too!
    wgresshoff
    @wgresshoff
    @ppanero Sorry, should have looked to make sure. It's not a secret but the logstash-configmap which I din't find in the docs.
    And thanks for the link :) Kubernetes doesn't provide a registry, but WWU.IT offers a nexus repository which supports docker.
    Pablo Panero
    @ppanero

    @wgresshoff the logstash configmap is not documented true. However, the config is provided: https://github.com/inveniosoftware/helm-invenio/blob/master/invenio/templates/configuration.yaml#L157

    I think it will be more intuitive to split the configuration per service/object affected, no? The same way we do for deployments of services? Something like

    /services
        - logstash.yaml
        -haproxy.yaml
        -postgresql.yaml
    
    /configutration
        - logstrash.yaml
        - nginx.yaml
        - invenio.yaml

    WDYT? Created an issue for that inveniosoftware/helm-invenio#13

    About Nexus, could you do a PR to the docs (after we release so you can do it simple in the UI) with the steps to push to nexus (or a simple link to official doc if there is, as in dockerhub)? - this latest request of mine can be revisited/more clear once we release :)

    Daily Notes
    Today

    • Merges, reviews & co.
    • Full test with latest masters. Deployed a QA instance with those, using the helm chart!

    Later on/tomorrow

    • Keep testing, prepare release!
    wgresshoff
    @wgresshoff
    That's a good idea! Perhaps with helm it's even possible to generate the passwords that are now provided by command line, so the secrets can be added in the same way?! Or is it possible to directly generate them into values.yaml?!
    Regarding Nexus: I surely can put a PR to the docs on how to add a docker image to a (private) registry.
    Sorry, I missed the logstash configmap in configuration.yaml. So everything is there to start testing (tomorrow).
    fenekku
    @fenekku

    Daily Notes
    Today

    Tomorrow

    • Still todo:
      • custom vocab + metadata extension docs
      • vocabulary_title filter
      • check the CSV encoding situation
    wgresshoff
    @wgresshoff
    After some thinking I'll postpone my helm/k8s testings till after the march release. When it works we could make it available with the april release.
    @ppanero What do you think?
    Kai Woerner
    @Herrner
    Re concept DOIs: " in the future we will change this to create a landing page specifically representing the concept behind the record and all of its versions" - is this future going to be the present in RDM?
    Pablo Panero
    @ppanero
    @wgresshoff it's perfect! thanks a lot for the work on the PR! lets revisit this in April :D
    Lars Holm Nielsen
    @lnielsen
    @Herrner Yes, but not in the first release - still to be seen.
    fenekku
    @fenekku

    Daily Notes
    Today

    • released invenio-cli v0.11.0
      • uses updated cookiecutter
      • more test coverage!
      • symlinks app_data in development mode
    • merged + released invenio-rdm-records v0.10.0
    • merged + released invenio-app-rdm v0.7.0
    • other calls
    • documentation work
    • development mode test: all good for me

    Tomorrow

    • metadata extension docs
    • lots of administrative tasks that slipped me by
    Dan Granville
    @dfdan
    Nice, look forward to trying it out!
    fenekku
    @fenekku
    We should have an official release next week. The cookiecutter needs to be updated again for sure :D