Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Dec 13 22:13
    amercader commented #5107
  • Dec 13 22:10
    amercader ready_for_review #5107
  • Dec 13 18:35
    Chealer commented #5115
  • Dec 13 14:45

    smotornyuk on py3

    IClick support (compare)

  • Dec 13 14:41
    amercader synchronize #5107
  • Dec 13 14:41

    amercader on py3-general-support-2

    [#4801] Fix url_for_static logic (compare)

  • Dec 13 14:31
    smotornyuk closed #5122
  • Dec 13 14:21
    smotornyuk synchronize #5121
  • Dec 13 14:19
    smotornyuk opened #5122
  • Dec 13 14:03
    smotornyuk edited #5121
  • Dec 13 14:01
    smotornyuk opened #5121
  • Dec 13 12:54
    amercader synchronize #5107
  • Dec 13 12:54

    amercader on py3-general-support-2

    [#4801] Simplify even more the … Coding standards (compare)

  • Dec 13 12:28

    smotornyuk on master

    [#5108] Alloe extensions to def… add error_shot to toolkit Merge branch 'master' into 5108… and 6 more (compare)

  • Dec 13 12:28
    smotornyuk closed #5112
  • Dec 13 12:28
    smotornyuk commented #5112
  • Dec 13 12:23
    smotornyuk unlabeled #5112
  • Dec 13 12:16
    davidread commented #4803
  • Dec 13 12:11

    davidread on diff-styles-in-less

    (compare)

  • Dec 13 12:11

    davidread on 4800-remove-formencode-support-2

    (compare)

Adrià Mercader
@amercader
Thanks for your review on ckan/ckan#5096 @davidread !
Sicco van Sas
@siccovansas
@davidread as often is the case; the problem was on my end where I imported the databases incorrectly. It all works now. Thx for the help though!
Patricio Del Boca
@pdelboca
@smotornyuk thanks for that guideline on migrating extensions! I have one question: Is it possible to start migrating extensions without being able to run CKAN in Python 3? (ckan/ckan#4801) If so, how can we proceed or at least start doing some work?
Adrià Mercader
@amercader
@pdelboca you can do prep work like migrating from pylons to flask (iblueprint, use toolkit etc), syntax changes for py3 (or py2/py3) etc but to test the actual functionality of course we need to wait until core runs py3
Patricio Del Boca
@pdelboca
Thanks @amercader! :)
Abed khooli
@abedkhooli
image.png
How to fix CKAN 2.8.3 datapusher (push_to_datastore)? Using csv, both link and upload are stuck: pending, fetching in jobs module, push_to_datastore func, ln 346.
Jari Voutilainen
@Zharktas
https://github.com/Zharktas/CKAN-in-Vagrant did a thing during the long weekend, this basically enables developing ckan in windows. Its not python3 based yet as some the commands in ckan core depends on pylons still
Sergey
@smotornyuk
:wave:
I've created first revision of extension's migration guid in repo wiki: https://github.com/ckan/ckan/wiki/Migration-guide-for-extensions
Brett
@kowh-ai
@smotornyuk -- Superb effort! Well done on the first revision!
Sivan Grünberg
@sivang
hey all, does anybody know the faceting code ? I have a custom vocab tag field that does not get into the facets sidebar, despite having specified in the facets config in the interface override in plugin.py nothing shows. I know that tags are stored onto the one tags field and then either expanded or folded back there depending on the context, so was wondering how should that work..
Adrià Mercader
@amercader
@wardi @tino097 meeting?
Cody Boyko
@boykoc
In CKAN there is a pylons response object that I can set something like response.set_cookie(). Is there a flask equivalent or a typical way to do that for the current response?
Konstantin Sivakov
@tino097
Cody Boyko
@boykoc

@tino097 appreciate the help. I was looking at that but from what I understand it needs to be called on a response object. Grabbing the current response object doesn't seem to be as easy as in pylons.

in pylons I was able to from common import response then do response.set_cookie(...)

But when dealing with flask handled pages I was trying flask.Response.set_cookie(...) without much luck. I'll keep tinkering

Cody Boyko
@boykoc
problem is actually described here for those that stumble on it later. https://github.com/ckan/ckan/wiki/Migration-from-Pylons-to-Flask#response-object
David Read
@davidread
@smotornyuk I got https://github.com/davidread/ckanext-hierarchy transferred from data.gov.uk's organization in the summer so that the CKAN community can own it. I did a bit of modernization then and I'm pleased to see you've done some great work making it python3 ready. I'm keen to see it move to the ckan organization on github - do you think that is a good idea? If so, perhaps you can ask for approval at the meeting?
Sergey
@smotornyuk
Yep, i'll mention it tomorrow.
Sergey
@smotornyuk
@boykoc , as far as i know, we don't have anything like global response, available during request. Currently vanilla-flask behavior is used, so anything you'll return from view(blueprint, etc) should be either ready-to-use response object, or something that can be converted to it. In most cases, views return rendered template(string) and rely on default flask conversion rules. If you want to set particular headers(cookies), you have to wrap view's result into manually created response object with any additional configuration. For example, API blueprint is setting custom headers to each request in such manner - https://github.com/ckan/ckan/blob/master/ckan/views/api.py#L77
This means, you cannot set cookie in random helper - you need to do it inside view function. Or, create initial response object inside view function and explicitely pass it to helper. The only other way i can imagine - you can define middleware, that will set cookies, but it is likely too low-level solution
Sergey
@smotornyuk
PS and that's correct way of using flask.Response.set_cookie: response = make_response(...); response.set_cookiet(...)
David Read
@davidread
Thanks
David Read
@davidread
@smotornyuk I've suggested a 'modernizations' section in the extension guide, although I'm not totally hot on the controller -> IBlueprints stuff - can you check this makes sense? https://github.com/ckan/ckan/wiki/Migration-guide-for-extensions
Cody Boyko
@boykoc
@smotornyuk awesome, thanks for the explanation, I appreciate it.
Adrià Mercader
@amercader
@smotornyuk @boykoc yes, that is quite a paradigm shift from the Pylons way of adding cookies, issue redirects etc from anywhere in the code. It also means that we need to adapt some of our plugin hooks to take this into account. For instance is quite common for IAuthenticator.login() or IAuthenticator.logout() to need to clear cookies, issue redirects etc. The current code just calls the plugin methods and carries on with the normal view function execution, so extensions can't do any of these actions. The way I've solved it for this particular case is checking if extensions are returning a response object, and if so just returning this directly on the view function, as it assumes that the extension has done all that's needed, see eg:
diff --git a/ckan/views/user.py b/ckan/views/user.py
index eae9b32dd..975f31096 100644
--- a/ckan/views/user.py
+++ b/ckan/views/user.py
@@ -3,6 +3,8 @@ import logging

 from flask import Blueprint
 from flask.views import MethodView
+from werkzeug import Response as werkzeug_response
+from flask import Response as flask_response
 from paste.deploy.converters import asbool
 from six import text_type

@@ -360,7 +362,9 @@ class RegisterView(MethodView):
 def login():
     # Do any plugin login stuff
     for item in plugins.PluginImplementations(plugins.IAuthenticator):
-        item.login()
+        resp = item.login()
+    if isinstance(resp, (werkzeug_response, flask_response)):
+        return resp

     extra_vars = {}
     if g.user:
@@ -391,7 +395,9 @@ def logged_in():
 def logout():
     # Do any plugin logout stuff
     for item in plugins.PluginImplementations(plugins.IAuthenticator):
-        item.logout()
+        resp = item.logout()
+    if isinstance(resp, (werkzeug_response, flask_response)):
+        return resp
     url = h.url_for(u'user.logged_out_page')
     return h.redirect_to(
         _get_repoze_handler(u'logout_handler_path') + u'?came_from=' + url,
Would love your views on this and potential alternatives
Cody Boyko
@boykoc

@amercader at first review I kind of like the above. I think i've experienced this as some of my h.redirect_to's where getting "ignored".

I'm definitely a fan of moving things into more of a structure (e.g. responses are only modified in Views).

As I'm working on my other piece i'll keep this in mind and share any thoughts I have.

Ian Ward
@wardi
@davidread +1 to ckanext-hierarchy in ckan github org
@boykoc ran into that too. We should call out the redirect helper brokenness in the upgrade docs
Adrià Mercader
@amercader
@davidread we did a bunch of improvements on our fork of ckanext-hierarchy. If you are working on it and want to pick them up go ahead: https://github.com/davidread/ckanext-hierarchy/compare/master...okfn:master If you can't I'll try to port them upstream when the extension is in the ckan org (which I'm +1 of course)
Brett
@kowh-ai
Hey everyone - here is the link to the latest CKAN 2.9 (py3) Status Report on Google Drive (Page 5 is the latest status - Nov 12th)
https://drive.google.com/open?id=15W2iR6mzrRJ_EVAR_1fGp7t4ST6OH_2l
  • thanks
Adrià Mercader
@amercader
Thanks @kowh-ai
Unfortunately I won't be able to join the call today
Michael Polidori
@mpolidori
@amercader I left a few comments on ckan/ckan#4803 (just a few questions I had). Just mentioning it here in case you didn't see them, thanks!
Adrià Mercader
@amercader
@mpolidori thanks for flagging it, I'll get back to you
Ian Ward
@wardi
@davidread There's general agreement to move ckanext-hierarchy to the ckan org, do you want to merge some of the okfn changes first or just move the repo?
Michael Polidori
@mpolidori
@amercader Thanks!
Brett
@kowh-ai
@wardi - Issue 4801 continues to be worked on by Adria. There is a lot of detailed work here and as far as I know Adria is pushing through this…the latest was, he had past starting the server, initialising the Middleware milestones and was working on rendering issues. Sergey has been working on migrating the Google Analytics CKAN extension Other extensions will be worked on…Link Digital and Dataopian will be helping with some of these. The CKAN 2.9 Status Report (link above) can be used to get an idea of the other work being done at the moment. HTH
Ian Ward
@wardi
@kowh-ai thanks again
Eric
@eric1599
Hi, I try to run docker-compose build for ckan, following instruction from https://docs.ckan.org/en/2.8/maintaining/installing/install-from-docker-compose.html#build-docker-images and I am getting dpkg: error processing package python-minimal (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
E: Sub-process /usr/bin/dpkg returned an error code (1)
ERROR: Service 'ckan' failed to build: The command '/bin/sh -c apt-get -q -y update && DEBIAN_FRONTEND=noninteractive apt-get -q -y upgrade && apt-get -q -y install python-dev python-pip python-virtualenv python-wheel libpq-dev libxml2-dev libxslt-dev libgeos-dev libssl-dev libffi-dev postgresql-client build-essential git-core vim wget && apt-get -q clean && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100
any ideas? thanks
Eric
@eric1599
I found the problem. I was using an older version of Dockerfile which use debian:jessie as base image, I changed it to debian:stretch it's working now.
David Read
@davidread
Thanks @amercader @smotornyuk @wardi , I've moved https://github.com/ckan/ckanext-hierarchy into the ckan organization now.
Ian Ward
@wardi
nice
David Read
@davidread
@amercader I'll see if I can work through the changes on https://github.com/ckan/ckanext-hierarchy/compare/master...okfn:master and get them in
닥짱
@darkzzang
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1104)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:585)
... 53 more
Caused by: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
does somebody know how to solve this issue?
it is about solr-jetty
Abed khooli
@abedkhooli
Anyone got datapusher to work in 2.8.3?