The Open Source platform for Open Data: https://ckan.org | *NOT* related to Kerbal Space Program | Come to our dev meetings: https://hack.allmende.io/ckan-meeting
@smotornyuk thanks for that guideline on migrating extensions! I have one question: Is it possible to start migrating extensions without being able to run CKAN in Python 3? (ckan/ckan#4801) If so, how can we proceed or at least start doing some work?
How to fix CKAN 2.8.3 datapusher (push_to_datastore)? Using csv, both link and upload are stuck: pending, fetching in jobs module, push_to_datastore func, ln 346.
https://github.com/Zharktas/CKAN-in-Vagrant did a thing during the long weekend, this basically enables developing ckan in windows. Its not python3 based yet as some the commands in ckan core depends on pylons still
:wave:
I've created first revision of extension's migration guid in repo wiki: https://github.com/ckan/ckan/wiki/Migration-guide-for-extensions
I've created first revision of extension's migration guid in repo wiki: https://github.com/ckan/ckan/wiki/Migration-guide-for-extensions
hey all, does anybody know the faceting code ? I have a custom vocab tag field that does not get into the facets sidebar, despite having specified in the facets config in the interface override in plugin.py nothing shows. I know that tags are stored onto the one tags field and then either expanded or folded back there depending on the context, so was wondering how should that work..
@tino097 appreciate the help. I was looking at that but from what I understand it needs to be called on a response object. Grabbing the current response object doesn't seem to be as easy as in pylons.
in pylons I was able to from common import response then do response.set_cookie(...)
But when dealing with flask handled pages I was trying flask.Response.set_cookie(...) without much luck. I'll keep tinkering
problem is actually described here for those that stumble on it later. https://github.com/ckan/ckan/wiki/Migration-from-Pylons-to-Flask#response-object
@smotornyuk I got https://github.com/davidread/ckanext-hierarchy transferred from data.gov.uk's organization in the summer so that the CKAN community can own it. I did a bit of modernization then and I'm pleased to see you've done some great work making it python3 ready. I'm keen to see it move to the ckan organization on github - do you think that is a good idea? If so, perhaps you can ask for approval at the meeting?
@boykoc , as far as i know, we don't have anything like global response, available during request. Currently vanilla-flask behavior is used, so anything you'll return from view(blueprint, etc) should be either ready-to-use response object, or something that can be converted to it. In most cases, views return rendered template(string) and rely on default flask conversion rules. If you want to set particular headers(cookies), you have to wrap view's result into manually created response object with any additional configuration. For example, API blueprint is setting custom headers to each request in such manner - https://github.com/ckan/ckan/blob/master/ckan/views/api.py#L77
This means, you cannot set cookie in random helper - you need to do it inside view function. Or, create initial response object inside view function and explicitely pass it to helper. The only other way i can imagine - you can define middleware, that will set cookies, but it is likely too low-level solution
@smotornyuk I've suggested a 'modernizations' section in the extension guide, although I'm not totally hot on the controller -> IBlueprints stuff - can you check this makes sense? https://github.com/ckan/ckan/wiki/Migration-guide-for-extensions
@smotornyuk @boykoc yes, that is quite a paradigm shift from the Pylons way of adding cookies, issue redirects etc from anywhere in the code. It also means that we need to adapt some of our plugin hooks to take this into account. For instance is quite common for
IAuthenticator.login() or IAuthenticator.logout() to need to clear cookies, issue redirects etc. The current code just calls the plugin methods and carries on with the normal view function execution, so extensions can't do any of these actions. The way I've solved it for this particular case is checking if extensions are returning a response object, and if so just returning this directly on the view function, as it assumes that the extension has done all that's needed, see eg:diff --git a/ckan/views/user.py b/ckan/views/user.py
index eae9b32dd..975f31096 100644
--- a/ckan/views/user.py
+++ b/ckan/views/user.py
@@ -3,6 +3,8 @@ import logging
from flask import Blueprint
from flask.views import MethodView
+from werkzeug import Response as werkzeug_response
+from flask import Response as flask_response
from paste.deploy.converters import asbool
from six import text_type
@@ -360,7 +362,9 @@ class RegisterView(MethodView):
def login():
# Do any plugin login stuff
for item in plugins.PluginImplementations(plugins.IAuthenticator):
- item.login()
+ resp = item.login()
+ if isinstance(resp, (werkzeug_response, flask_response)):
+ return resp
extra_vars = {}
if g.user:
@@ -391,7 +395,9 @@ def logged_in():
def logout():
# Do any plugin logout stuff
for item in plugins.PluginImplementations(plugins.IAuthenticator):
- item.logout()
+ resp = item.logout()
+ if isinstance(resp, (werkzeug_response, flask_response)):
+ return resp
url = h.url_for(u'user.logged_out_page')
return h.redirect_to(
_get_repoze_handler(u'logout_handler_path') + u'?came_from=' + url,
Would love your views on this and potential alternatives
@amercader at first review I kind of like the above. I think i've experienced this as some of my h.redirect_to's where getting "ignored".
I'm definitely a fan of moving things into more of a structure (e.g. responses are only modified in Views).
As I'm working on my other piece i'll keep this in mind and share any thoughts I have.
@boykoc ran into that too. We should call out the redirect helper brokenness in the upgrade docs
@davidread we did a bunch of improvements on our fork of ckanext-hierarchy. If you are working on it and want to pick them up go ahead: https://github.com/davidread/ckanext-hierarchy/compare/master...okfn:master If you can't I'll try to port them upstream when the extension is in the ckan org (which I'm +1 of course)
Hey everyone - here is the link to the latest CKAN 2.9 (py3) Status Report on Google Drive (Page 5 is the latest status - Nov 12th)
https://drive.google.com/open?id=15W2iR6mzrRJ_EVAR_1fGp7t4ST6OH_2l
https://drive.google.com/open?id=15W2iR6mzrRJ_EVAR_1fGp7t4ST6OH_2l
- thanks
Unfortunately I won't be able to join the call today
@amercader I left a few comments on ckan/ckan#4803 (just a few questions I had). Just mentioning it here in case you didn't see them, thanks!
@wardi - Issue 4801 continues to be worked on by Adria. There is a lot of detailed work here and as far as I know Adria is pushing through this…the latest was, he had past starting the server, initialising the Middleware milestones and was working on rendering issues. Sergey has been working on migrating the Google Analytics CKAN extension Other extensions will be worked on…Link Digital and Dataopian will be helping with some of these. The CKAN 2.9 Status Report (link above) can be used to get an idea of the other work being done at the moment. HTH
Hi, I try to run docker-compose build for ckan, following instruction from https://docs.ckan.org/en/2.8/maintaining/installing/install-from-docker-compose.html#build-docker-images and I am getting dpkg: error processing package python-minimal (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
E: Sub-process /usr/bin/dpkg returned an error code (1)
ERROR: Service 'ckan' failed to build: The command '/bin/sh -c apt-get -q -y update && DEBIAN_FRONTEND=noninteractive apt-get -q -y upgrade && apt-get -q -y install python-dev python-pip python-virtualenv python-wheel libpq-dev libxml2-dev libxslt-dev libgeos-dev libssl-dev libffi-dev postgresql-client build-essential git-core vim wget && apt-get -q clean && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100
dependency problems - leaving unconfigured
Errors were encountered while processing:
E: Sub-process /usr/bin/dpkg returned an error code (1)
ERROR: Service 'ckan' failed to build: The command '/bin/sh -c apt-get -q -y update && DEBIAN_FRONTEND=noninteractive apt-get -q -y upgrade && apt-get -q -y install python-dev python-pip python-virtualenv python-wheel libpq-dev libxml2-dev libxslt-dev libgeos-dev libssl-dev libffi-dev postgresql-client build-essential git-core vim wget && apt-get -q clean && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100
any ideas? thanks
Thanks @amercader @smotornyuk @wardi , I've moved https://github.com/ckan/ckanext-hierarchy into the ckan organization now.
@amercader I'll see if I can work through the changes on https://github.com/ckan/ckanext-hierarchy/compare/master...okfn:master and get them in
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1104)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:585)
... 53 more
Caused by: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1104)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:585)
... 53 more
Caused by: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
does somebody know how to solve this issue?
it is about solr-jetty