Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 22:55
    jrobind starred potatolondon/djangae
  • Jan 30 12:50
    bvwells starred potatolondon/djangae
  • Jan 29 19:44
    mattdoughty closed #1159
  • Jan 29 19:38
    mattdoughty opened #1159
  • Jan 29 12:06
    pmutua starred potatolondon/djangae
  • Jan 25 17:02
    mattbis starred potatolondon/djangae
  • Jan 23 16:26
    davidwtbuxton commented #1017
  • Jan 23 14:02
    michal-macioszczyk edited #1158
  • Jan 23 14:02
    michal-macioszczyk edited #1158
  • Jan 23 14:01
    michal-macioszczyk synchronize #1158
  • Jan 23 11:41
    michal-macioszczyk assigned #1158
  • Jan 23 11:41
    michal-macioszczyk edited #1158
  • Jan 23 11:39
    michal-macioszczyk edited #1158
  • Jan 23 11:39
    michal-macioszczyk opened #1158
  • Jan 23 10:43
    michal-macioszczyk edited #1150
  • Jan 23 10:43
    michal-macioszczyk edited #1150
  • Jan 22 13:41
    michal-macioszczyk assigned #1017
  • Jan 17 17:14
    adamalton commented #1150
  • Jan 17 17:08
    adamalton commented #1151
  • Jan 17 17:06

    michal-macioszczyk on master

    Increase the initial wait time … Don't lose the original error s… Always log the tracebacks if we… and 7 more (compare)

Luke Benstead
@Kazade
as in, when we write to the database, not read
jacobg
@jacobg
hmmm i’ll explore further. the unit test i wrote to demonstrate my issue fails on read not saving.
Luke Benstead
@Kazade
class MyModel(models.Model):
    some_field = IntegerField(null=False)


instance = MyModel.objects.create(some_field=1)

from google.appengine.api.datastore import Get, Put, Key

entity = Get(Key.from_path(MyModel._meta.db_table, instance.pk, namespace=connection.settings_dict["NAMESPACE"]))
del entity["some_field"]
Put(entity)

instance.refresh_from_db()

self.assertIsNone(instance.some_field)
something like that ^ is the test we need
although some_field would need a default
jacobg
@jacobg
ok thanks i’ll add that as a test
jacobg
@jacobg
@Kazade
Pull requests ready:
potatolondon/djangae#1090
potatolondon/djangae#1084
potatolondon/djangae#1082
Cloud Firestore is the next generation of Cloud Datastore. Cloud Firestore now supports all Cloud Datastore APIs and client libraries when created in Datastore mode. When Cloud Firestore reaches general availability, all Cloud Datastore customer projects will be automatically upgraded to Cloud Firestore in Datastore mode. This upgrade will require no code changes and will be completed with no scheduled downtime.
Luke Benstead
@Kazade
Well that's all very cool
We're trying to work out our python 3 strategy just so you know
jacobg
@jacobg
yeah i’m working on making my gae app python 3 compatible: using python-future and caniusepython3
Luke Benstead
@Kazade
For what it's worth, the python 3 runtime has big gaps
You're not going to be able to just upgrade
No memcache, no mail, no search, no task queues built in...
No users API
jacobg
@jacobg
the Firestore is more of a surprise for me, especially with consistency. i’m surprised that after all this time they still don’t support more than one inequality field filter. datastore used to be the best nosql, but it looks like azure’ cosmo db is ahead of them. i would seriously consider azure now for a Greenfield project. they’ve made huge progress.
oh really, i didn’t dive into their python 3 runtime stuff. that’s crazy.
i didn’t see anything about the app engine pipelines and mapreduce updated to python 3.
i
ve recently started to break up my gae app, using functions and pub-sub. was going to look at dataflow too.
gae sdk made it a lot easier for development. it’s a pain to do integrated development with functions and other gcp services.
Jun Wang
@wj1918_twitter
I want to migrate data from google cloud sql to google cloud datastore, any suggestion?
do i have to write a script to copy the data?
Or can I use admin command dumpdata from sql database and loaddata into datastore?
Luke Benstead
@Kazade
You'll probably need to write your own script that runs on the App Engine production site
(Rather than locally)
jacobg
@jacobg
Hi @Kazade Any chance you can merge my pull request for map-reduce slicing?
potatolondon/djangae#1090
Thanks!
Luke Benstead
@Kazade
@jacobg sorry, in a bit of crunch mode at the moment - I'll get to it when I can
jacobg
@jacobg
@Kazade no worries, thanks
jacobg
@jacobg

The following commit is causing startup failure in GAE production:
potatolondon/djangae@36974d0

Exception:

2018-09-27 20:16:35.804 EDT
    from djangae import checks
2018-09-27 20:16:35.804 EDT
  File "/base/data/home/apps/s~/1.412878905325365371/sitepackages/prod/djangae/checks.py", line 8, in <module>
2018-09-27 20:16:35.804 EDT
    from google.appengine.tools.sdk_update_checker import GetVersionObject, _VersionList
2018-09-27 20:16:35.804 EDT
ImportError: No module named sdk_update_checker

Looks like sdk_update_checker only exists in local dev sdk, not in production. Is this commit correct?

jacobg
@jacobg

Just sharing my thoughts on next-gen Djangae (I didn't see an issue specific to this)… I’m gradually breaking apart my Djangae monolith server:
1) All my REST API’s now use JWT (on DRF), and session auth is limited to an auth app which issues the JWT tokens.
2) I’m planning to soon switch the auth itself over to Firebase Auth so I can get rid of auth completely from Djangae (except for obviously validating JWT)
3) Moved html and some basic backend code for it over to Firebase Hosting and Functions. Uses PubSub to communicate between Djangae and these Functions.
4) Have a SPA on Firebase Hosting.
5) May at some point start using Dataflow instead of Pipelines/MR tasks in Djangae.

So I’m sort of left with rest API’s (protected with JWT) that read/write from datastore, do some caching, transactions, and some background tasks and mapping over querysets. I find the real Djangae value is in the ORM, and in bootstrapping Django WSGI to run in App Engine (including url handlers for the _ah stuff). Most of my code is actually interfacing with DRF, and it would be completely impossible without Djangae.

Thanks guys!

Luke Benstead
@Kazade
The plan definitely involves moving the ORM connector into its own repo - for use with any Django setup
anywhere you can use Cloud Datastore
we'll likely be dropping the caching layer (which exists largely to fight eventual consistency - which doesn't exist on Datastore on Firestore)
jacobg
@jacobg
Ah neat, so using it in Dataflow or Functions will be a breeze
Luke Benstead
@Kazade
As GAE is moving to a microservice platform (for better or worse...) Djangae will need to break up too
jacobg
@jacobg
Yeah exactly
Luke Benstead
@Kazade
Things like djangae.deferredare intended as abstractions so a future move to Cloud Tasks won't be so disruptive
jacobg
@jacobg
Yeah I find the biggest disruption is actually the configuration and the development environment. Once that’s set up the code is not really different.
jacobg
@jacobg

Suppose I have two models like this:

class A(models.Model):
    pass # fields not important here

class B(models.Model):
    a = models.OneToOneField(A)

I want to build an API that serves A, with B where it exists. Simple code might looks like:

    a_list = A.objects.all() # for example purpose, ignore filter and paging, etc
    b_list = B.objects.filter(a__in=a_list) # a_list is always within pk__in max
    results = json_serialize(a_list) # json_serialize automatically follows reverse relation
    # should be list like this: { id: 1, b: { ... } }

My question is: In a single request, if I’ve already fetched B entities, will following reverse relation from A to one of those B's invoke a new datastore query, or does Djangae caching pull it out of the in-process context cache matching by pk?

Michael Sander
@speedplane
Hi Folks, is there any documentation or examples on how one would port a project from the older django-nonrel to djangae?
Adam Alton
@adamalton
@jacobg to answer your questions about caching, I'm pretty sure that Djangae will add entities to the cache and will use the cache to fetch entities if the fetching is using some kind of get() type thing. E.g.
A.objects.get(pk=x)
Adam Alton
@adamalton
Erm... not sure what happened there. I wrote a long reply, and then when I hit submit Gitter decided to revert it to a previous version.
I need to go home, but will try to re-write my reply tomorrow.
Quick answer, I think that it will try to use the cache, but it won't find anything because the first query won't have added anything to it.
Adam Alton
@adamalton
@speedplane I don't think we have any specific documentation for this. I'm just trying to think what the main things will be...
  • I would recommend taking Djangae Scaffold as you starting point in terms of project configuration, libraries, etc, and porting your own code into that setup, rather than trying to convert your existing setup.
  • There are some custom fields in djangoappengine (e.g. ListField), which have now been replaced with new (better!) versions in djangae.fields. So you'll need to swap those model fields out. I think that the underlying data structures will still work (certainly for ListField), so I don't think you'll need to change the data for that, but don't rely on my word for that!
  • If you were using dbindexer then you'll no longer need that as Djangae automatically detects the queries that you use and then generates the extra fields/indexes for them for you. The only gotchas here are that (1) you'll need to make sure you run all the queries that you need to use locally with the SDK first so that Djangae can detect which indexes need to be built, and (2) you'll need to then re-save all of your existing objects so that the new index data can be saved.
  • If I remember rightly, djangoappengine/django-nonrel stopped being updated around Django 1.4, so you'll need to update your code to deal with the changes in Django itself from 1.4 to 1.11.
  • I think that in general you'll just find that Djangae provides a whole load of additional features and functionality (e.g. the ability to do __contains queries; new fields such as SetField, RelatedListField, RelatedSetField and ComputedXFields; support for transactions (Djangae's custom transactions, not Django transactions); utilities such as retry and an improved version of defer; automatic caching of things (see above conversation!); and other things that I can't remember off the top of my head).
jacobg
@jacobg
@adamalton @Kazade Are you able to review my PR for the gcloud branch?
potatolondon/djangae#1172
What do you think about merging into master?
jacobg
@jacobg
@Kazade had been working on v2 datastore connector, but @adamalton indicated that he went away on parental leave. Will he be coming back, or is someone else going to be taking the reins on that? Thanks!