These are chat archives for damianh/Cedar

27th
Jul 2015
Dan Barua
@danbarua
Jul 27 2015 08:54
projecting to SQL -> 3NF or denormalized?
Adam Ralph
@adamralph
Jul 27 2015 09:07
whatever is most efficient for your view requirements
Dan Barua
@danbarua
Jul 27 2015 09:07
we need to do quite a bit of filtering/reporting on it
was pondering whether to stuff certain value types in as strings (as we did in Redis)
or break out to a lookup table
Adam Ralph
@adamralph
Jul 27 2015 09:07
capture that filtering and reporting and project up front?
Dan Barua
@danbarua
Jul 27 2015 09:08
i wish
they'll tell us the reporting doesn't tell them what they want but they won't say what they want
Adam Ralph
@adamralph
Jul 27 2015 09:08
is it the old "it needs to be fully flexible... [6 months passes] ... same old 3 reports being hammered daily" applecart?
Dan Barua
@danbarua
Jul 27 2015 09:08
sorta
tbh I'm thinking project out to 3nf db and slap third party reporting suite (we sell one already) in front of it -> done
Adam Ralph
@adamralph
Jul 27 2015 09:09
this is something we considered doing for some our stuff once
we slapped sql analysis service or whatever its called on top of it, as a POC
even just let people write SQL against it (the original purpose of SQL!)
someone writes a cartesian, meh, who cares. write model and UI read model all still happy
Dan Barua
@danbarua
Jul 27 2015 09:11
yeah, this
i've managed to escape building a generic reporting engine
so far
Adam Ralph
@adamralph
Jul 27 2015 09:11
a queue somewhere might be getting backed up... but so long as the queue infra can handle it...
Dan Barua
@danbarua
Jul 27 2015 09:11
and we have one in our portfolio
some other mug can support it ha
i'm looking at getting rid of the queues
on the fence re: polling + checkpoints vs push via queues
easynetq's error handling is nice
apart from base64encoding the headers it republishes onto the bus which I've found buried in my event store, grrrr
Dan Barua
@danbarua
Jul 27 2015 09:44
well, at least i can query for them:
select distinct json_metadata->>'UserId'
from events
where 
(json_metadata->>'UserId')::text is not null
 and (json_metadata->>'UserId')::text not like '%-%-%'
limit 1000;
Damian Hickey
@damianh
Jul 27 2015 09:55

is it the old "it needs to be fully flexible... [6 months passes] ... same old 3 reports being hammered daily" applecart?

That shit is in full effect here.

Cube this, 3NF that
Dan Barua
@danbarua
Jul 27 2015 09:58
Npgsql.NpgsqlException: Npgsql.NpgsqlException : FATAL: 53300: sorry, too many clients already
sorry, not Cedar related, I've got nCrunch running tests in serial, why am I still getting socket issues
socket/connection issues in general
Damian Hickey
@damianh
Jul 27 2015 10:00
We have our customers asking for a Sql store of some sort - they don't know what they want to report on. They just want to explore. I'm fine with that oc.
the number of times i've had to deploy this in response to "what table is XXX in so I can just go and do this thing the software was not designed to do" cases
Dan Barua
@danbarua
Jul 27 2015 10:03
this is the source of most of our problems - everything got whacked with the MSSQL hammer
Adam Ralph
@adamralph
Jul 27 2015 10:10
^^ 90% of enterprise apps, everywhere
Damian Hickey
@damianh
Jul 27 2015 10:11
@adamralph What are you working on at PS? Doing event sourcing at all?
Adam Ralph
@adamralph
Jul 27 2015 10:11
no, not at all
Damian Hickey
@damianh
Jul 27 2015 10:12
Fixing their interface naming conventions? :trollface:
Dan Barua
@danbarua
Jul 27 2015 10:55
for the record, nCrunch + npgsql -> MaxPoolSize=100;
SHOW max_connections; then put that in your connection string
Damian Hickey
@damianh
Jul 27 2015 10:55
mentally noted
parallel tests?
fuppin statics
Dan Barua
@danbarua
Jul 27 2015 10:55
no, in serial
i've got some dumbass tests where i use EmbeddedSmtpServer to catch and assert on emails
i think i can run the postgres ones in parralel i'll give it a go
Dan Barua
@danbarua
Jul 27 2015 11:01
nah, running in parallel seems to hit PGSQL's MaxConnections
and duh, of course i can't run tests in parallel since they drop and rebuild the schema..
Damian Hickey
@damianh
Jul 27 2015 11:03
Use a different, but deterministic, database name for each test collection?
Dan Barua
@danbarua
Jul 27 2015 11:04
I've been toying with that idea, I'd need to find a reliable way of cleaning them up
Damian Hickey
@damianh
Jul 27 2015 11:04
Clean up on CI server or on dev?
Dan Barua
@danbarua
Jul 27 2015 11:04
both
I've cheated - on TeamCity only one build can be running integration tests at a time
it's on my TODO backlog
Damian Hickey
@damianh
Jul 27 2015 11:05
Our build agents are spun up on-demand from an AMI.
Dan Barua
@danbarua
Jul 27 2015 11:05
oh nice
Damian Hickey
@damianh
Jul 27 2015 11:05
So no probs there
Dan Barua
@danbarua
Jul 27 2015 11:05
we own all our metal here
none of that cloud nonsense
banks seem to like photos of racks with a £5 combination padlock on the door
Damian Hickey
@damianh
Jul 27 2015 11:06
For local dev - a script that selects all databases that start with a prefix, iterates and drops them
lol
Dan Barua
@danbarua
Jul 27 2015 11:06
that's a neat idea, i'll give that a go
Damian Hickey
@damianh
Jul 27 2015 11:06
I'll never use a CI build agent that isn't an AMI (or equivalent) again
Dan Barua
@danbarua
Jul 27 2015 11:06
we've..
erm
Damian Hickey
@damianh
Jul 27 2015 11:06
The AMI has 1 thing installed, choco
Everything can be installed from there.
Dan Barua
@danbarua
Jul 27 2015 11:06
we've got visual studio, delphi, all kinds of shit installed on TC
Damian Hickey
@damianh
Jul 27 2015 11:06
yeah fuck that
Dan Barua
@danbarua
Jul 27 2015 11:07
hard coded dependencies on file system paths etc
Damian Hickey
@damianh
Jul 27 2015 11:07
non-repeatable junk
Dan Barua
@danbarua
Jul 27 2015 11:07
yep
still better than it was without CI
ish
ok so that works, now postgres has locked out pgadmin because nCrunch is hammering it ha
Damian Hickey
@damianh
Jul 27 2015 11:11
Your test fixture could always pool out the connections itself - if you wanted to reserve some.
or limit it
Dan Barua
@danbarua
Jul 27 2015 11:18
DROP DATABASE cannot be executed from a function or multi-command string urgh
Damian Hickey
@damianh
Jul 27 2015 11:19
Wrap in a BEGIN-END
er scratch that
Dan Barua
@danbarua
Jul 27 2015 11:19
select -> clipboard -> notepad++ -> alt -> insert "DROP" before db name
there's got to be a better way but meh
oh lol ffs
wont let me do it
Dan Barua
@danbarua
Jul 27 2015 11:26
aaaah hang on yes i remember why i held off on doing db per test run
drop and recreate schema is about 800ms-1s
create db is about 3s
Damian Hickey
@damianh
Jul 27 2015 11:28
So... we're exploring the concept of a db 'pool'. Create several DB async, hand them out and when returned do the clean up async.
First test run is always expensive though.
Dan Barua
@danbarua
Jul 27 2015 11:28
for sure
that sounds interesting
I feel like i'm about to start an epic yak shaving exercise
Damian Hickey
@damianh
Jul 27 2015 11:28
yeah :|
Dan Barua
@danbarua
Jul 27 2015 11:28
tbh
migrate event store mssql -> postgres
migrate read model redis -> postgres
shitloads of yak shaving
Damian Hickey
@damianh
Jul 27 2015 11:29
part of the job
Dan Barua
@danbarua
Jul 27 2015 12:49
as I suspected, struggling to get these databases cleaned up
(doing a DROP in Dispose() on the fixture)
Dan Barua
@danbarua
Jul 27 2015 14:42
var nextCheckpoint = !events.IsEnd ? events.NextCheckpoint : events.StreamEvents.Last().Checkpoint;
if we're at the end, .NextCheckpoint probably should not be null right?
Damian Hickey
@damianh
Jul 27 2015 14:42
I need to check the ES api
It's prob be -1 or something
or whatever their vector clock equivalent is
Dan Barua
@danbarua
Jul 27 2015 14:43
await dbConnection.ExecuteAsync("UPDATE projections.checkpoint SET checkpoint = @checkpoint;", new { checkpoint = nextCheckpoint.ToString() }); i'm just doing this at the moment
Damian Hickey
@damianh
Jul 27 2015 14:43
Can't context switch to this right now :(
Dan Barua
@danbarua
Jul 27 2015 14:44
np, i'll figure something out
Dan Barua
@danbarua
Jul 27 2015 15:00
doh, i'm an idiot
var nextCheckpoint = events.IsEnd ? Checkpoint.End : events.NextCheckpoint;
actually no that wont work
shit..
Dan Barua
@danbarua
Jul 27 2015 15:05
can't persist Checkpoint.End because next time you read the end of the stream might have moved
Dan Barua
@danbarua
Jul 27 2015 15:52
we need a mechanism to return checkpoint.next when at the end of the stream
i'm guessing (max(ordinal) + 1).ToString() should be ok...
will need to see how GES does it
 return new AllEventsPage(
                 allEventsSlice.FromPosition.ToString(),
                 allEventsSlice.NextPosition.ToString(),
                 allEventsSlice.IsEndOfStream,
                 GetReadDirection(allEventsSlice.ReadDirection),
                 events);
so we're assuming that allEventsSlice.NextPosition is never null
if that's the case then i'll add an impl and test to the sql providers
GES tests don't assert either way