These are chat archives for translate/dev

13th
Sep 2016
Julen Ruiz Aizpuru
@julen
Sep 13 2016 07:23
hi there good morning
phlax
@phlax
Sep 13 2016 07:34
morning
Dwayne Bailey
@dwaynebailey
Sep 13 2016 08:06
hello
Taras Semenenko
@ta2-1
Sep 13 2016 08:09
hi @/all
Leandro Regueiro
@unho
Sep 13 2016 08:30
Hi
Taras Semenenko
@ta2-1
Sep 13 2016 08:34
hey @julen what do you think re https://github.com/translate/pootle/pull/5202/commits/7f31ac3ab327e571764d41fa593bb03dc15f5e82 ?
is it ok to use __html here or I should think re creating a new component for translatin-text div?
Julen Ruiz Aizpuru
@julen
Sep 13 2016 11:25
hi ta2-1. Unless highlightRW returns elements, that's the only way to go as it returns an string which needs to be treated as HTML. Just make sure input is escaped before outputting
Julen Ruiz Aizpuru
@julen
Sep 13 2016 11:37
phlax: re. figures, I am aware of the WIP nature and that these are more or less continuously changing, although it might not hurt to share some from time to time especially for people busy looking at other stuff
I'm also unclear on the stats data-related migrations, on whether these mirror refresh stats (I see it's gone) and on its expected execution time
phlax
@phlax
Sep 13 2016 11:45
the data related migrations are fairly time-consuming
re test stats - the new system obviously adds stuff to a unit.save - so my focus has been on finding other optimizations around this
for vfolder users - this was the big pain point
so we have addressed this and this is no longer an issue - or at least wont be when we are finished
in terms of the remaining cost of unit.save
by far the majority of it is scorelog related
Julen Ruiz Aizpuru
@julen
Sep 13 2016 11:51
Good to hear other stuff is being optimized. I'm still unclear on refresh stats bits though. Let's say for instance someone changes the wordcount function. This means that automatically, all word-related stats are not accurate. How can one refresh these without something like refresh_stats?
phlax
@phlax
Sep 13 2016 11:51
@julen any thoughtss on this translate/pootle#5235 - its a minor cleanup - but i wanted to check before removing
@julen no stats will always be accurate
there will be no refresh_stats
Julen Ruiz Aizpuru
@julen
Sep 13 2016 11:53
I see there is no refresh stats, but how is accuracy ensured in the example above?
phlax
@phlax
Sep 13 2016 11:53
by saving them in an aggregation table as requried
this is then mined to produce accurate stats
quickly
@julen how is accuracy assured now?
Julen Ruiz Aizpuru
@julen
Sep 13 2016 11:55
maybe accuracy is not the right term — I'm wondering how is refreshing triggered to ensure recalculation has happened in the example above
phlax
@phlax
Sep 13 2016 11:56
no accuracy is defo the right term
Julen Ruiz Aizpuru
@julen
Sep 13 2016 11:56
currently one changes the setting for the wordcount function, and runs refresh_stats
phlax
@phlax
Sep 13 2016 11:56
we will ensure accuracy because a) we will have tests b) the stats will be set immediately
well re wordcounts - we can look at an equiv to calculate_checks
Julen Ruiz Aizpuru
@julen
Sep 13 2016 11:57
cool, thanks for answering
phlax
@phlax
Sep 13 2016 11:57
np
any thoughts on translate/pootle#5235 - shall i just land it?
Julen Ruiz Aizpuru
@julen
Sep 13 2016 11:58
even if changing, some preliminary numbers would help too
phlax
@phlax
Sep 13 2016 11:58
they really are changing
but i can tell you we have gone from ~12000 db calls in the test runs to ~80000
Julen Ruiz Aizpuru
@julen
Sep 13 2016 11:58
I believe that was used with the auto-terminology feature (which was removed)
phlax
@phlax
Sep 13 2016 11:58
altho a lot of that was vfolder related
in my current PR db calls are ~82000
Julen Ruiz Aizpuru
@julen
Sep 13 2016 11:59
and perhaps with monolingual file formats too (to perform update against templates without an on-disk store)
phlax
@phlax
Sep 13 2016 11:59
but we have some further optmizations around this
@julen if you want figures for this try this
py.test -vv -k tp_tool --debug-tests mytest.log
this tool can be greatly optimized with a slightly diff pattern
but its a v good indicator of the cost of db cloning/updates
Julen Ruiz Aizpuru
@julen
Sep 13 2016 12:03
thank you, will try on my own spare time (as said, busy with other stuff and since you are actively working on it, you probably had the numbers)
phlax
@phlax
Sep 13 2016 12:14
@julen im gonna land that PR - this is what grepping gives
(pootle)phlax@aio:~/.virtualenvs/pootle/pootle$ grep -r "\._units" pootle/ | grep -v log | grep -v Binary | grep -v [#~]
pootle/apps/pootle_store/models.py:            return self._units
pootle/apps/pootle_store/models.py:            for i, unit in enumerate(self._units):
pootle/apps/pootle_store/models.py:                self._units = FakeQuerySet()
pootle/apps/pootle_store/models.py:            self._units.append(newunit)
Taras Semenenko
@ta2-1
Sep 13 2016 12:35
so @julen any chance to get GTM on translate/pootle#5202 ?
Julen Ruiz Aizpuru
@julen
Sep 13 2016 12:44
ta2-1: I believe I did last week already, didn't I? unless you have changed a lot...
*changed it
Taras Semenenko
@ta2-1
Sep 13 2016 12:49
hmm... there were some your comments which weren't addressed. I believe that now they are. And I wasn't 100% sure re commit which I shared today morning. Though all my tests work fine.
Dwayne Bailey
@dwaynebailey
Sep 13 2016 12:57
@/all scratchpad for the Pootle hack get-together, please add your objectives and goals https://github.com/translate/pootle/wiki/PootleConf-2016
Julen Ruiz Aizpuru
@julen
Sep 13 2016 15:34
cheers everyone