projecting to SQL -> 3NF or denormalized?
whatever is most efficient for your view requirements
we need to do quite a bit of filtering/reporting on it
was pondering whether to stuff certain value types in as strings (as we did in Redis)
or break out to a lookup table
capture that filtering and reporting and project up front?
i wish
they'll tell us the reporting doesn't tell them what they want but they won't say what they want
is it the old "it needs to be fully flexible... [6 months passes] ... same old 3 reports being hammered daily" applecart?
sorta
tbh I'm thinking project out to 3nf db and slap third party reporting suite (we sell one already) in front of it -> done
this is something we considered doing for some our stuff once
we slapped sql analysis service or whatever its called on top of it, as a POC
even just let people write SQL against it (the original purpose of SQL!)
someone writes a cartesian, meh, who cares. write model and UI read model all still happy
yeah, this
i've managed to escape building a generic reporting engine
so far
a queue somewhere might be getting backed up... but so long as the queue infra can handle it...
and we have one in our portfolio
some other mug can support it ha
i'm looking at getting rid of the queues
on the fence re: polling + checkpoints vs push via queues
easynetq's error handling is nice
apart from base64encoding the headers it republishes onto the bus which I've found buried in my event store, grrrr
well, at least i can query for them:
select distinct json_metadata->>'UserId'
from events
where
(json_metadata->>'UserId')::text is not null
and (json_metadata->>'UserId')::text not like '%-%-%'
limit 1000;
is it the old "it needs to be fully flexible... [6 months passes] ... same old 3 reports being hammered daily" applecart?
That shit is in full effect here.
Cube this, 3NF that
Npgsql.NpgsqlException: Npgsql.NpgsqlException : FATAL: 53300: sorry, too many clients already
sorry, not Cedar related, I've got nCrunch running tests in serial, why am I still getting socket issues
socket/connection issues in general
We have our customers asking for a Sql store of some sort - they don't know what they want to report on. They just want to explore. I'm fine with that oc.
the number of times i've had to deploy this in response to "what table is XXX in so I can just go and do this thing the software was not designed to do" cases
http://devlicio.us/blogs/casey/archive/2009/05/14/commercial-suicide-integration-at-the-database-level.aspx
^ one of my fav blog posts of all time
^ one of my fav blog posts of all time
this is the source of most of our problems - everything got whacked with the MSSQL hammer
^^ 90% of enterprise apps, everywhere
@adamralph What are you working on at PS? Doing event sourcing at all?
no, not at all
Fixing their interface naming conventions? :trollface:
for the record, nCrunch + npgsql -> MaxPoolSize=100;
SHOW max_connections; then put that in your connection string
mentally noted
parallel tests?
fuppin statics
no, in serial
i've got some dumbass tests where i use EmbeddedSmtpServer to catch and assert on emails
i think i can run the postgres ones in parralel i'll give it a go
nah, running in parallel seems to hit PGSQL's MaxConnections
and duh, of course i can't run tests in parallel since they drop and rebuild the schema..
Use a different, but deterministic, database name for each test collection?
I've been toying with that idea, I'd need to find a reliable way of cleaning them up
Clean up on CI server or on dev?
both
I've cheated - on TeamCity only one build can be running integration tests at a time
it's on my TODO backlog
Our build agents are spun up on-demand from an AMI.
oh nice
So no probs there
we own all our metal here
none of that cloud nonsense
banks seem to like photos of racks with a £5 combination padlock on the door
For local dev - a script that selects all databases that start with a prefix, iterates and drops them
lol
that's a neat idea, i'll give that a go
I'll never use a CI build agent that isn't an AMI (or equivalent) again
we've..
erm
The AMI has 1 thing installed, choco
Everything can be installed from there.
we've got visual studio, delphi, all kinds of shit installed on TC
yeah fuck that
hard coded dependencies on file system paths etc
non-repeatable junk
yep
still better than it was without CI
ish
ok so that works, now postgres has locked out pgadmin because nCrunch is hammering it ha
Your test fixture could always pool out the connections itself - if you wanted to reserve some.
or limit it
DROP DATABASE cannot be executed from a function or multi-command string urgh
Wrap in a BEGIN-END
er scratch that
select -> clipboard -> notepad++ -> alt -> insert "DROP" before db name
there's got to be a better way but meh
oh lol ffs
wont let me do it
aaaah hang on yes i remember why i held off on doing db per test run
drop and recreate schema is about 800ms-1s
create db is about 3s
So... we're exploring the concept of a db 'pool'. Create several DB async, hand them out and when returned do the clean up async.
First test run is always expensive though.
for sure
that sounds interesting
I feel like i'm about to start an epic yak shaving exercise
yeah :|
tbh
migrate event store mssql -> postgres
migrate read model redis -> postgres
shitloads of yak shaving
part of the job
as I suspected, struggling to get these databases cleaned up
(doing a DROP in Dispose() on the fixture)
var nextCheckpoint = !events.IsEnd ? events.NextCheckpoint : events.StreamEvents.Last().Checkpoint;
if we're at the end,
.NextCheckpoint probably should not be null right?
I need to check the ES api
It's prob be -1 or something
or whatever their vector clock equivalent is
await dbConnection.ExecuteAsync("UPDATE projections.checkpoint SET checkpoint = @checkpoint;", new { checkpoint = nextCheckpoint.ToString() }); i'm just doing this at the moment
Can't context switch to this right now :(
np, i'll figure something out
doh, i'm an idiot
var nextCheckpoint = events.IsEnd ? Checkpoint.End : events.NextCheckpoint;
actually no that wont work
shit..
can't persist Checkpoint.End because next time you read the end of the stream might have moved
we need a mechanism to return checkpoint.next when at the end of the stream
i'm guessing (max(ordinal) + 1).ToString() should be ok...
will need to see how GES does it
return new AllEventsPage(
allEventsSlice.FromPosition.ToString(),
allEventsSlice.NextPosition.ToString(),
allEventsSlice.IsEndOfStream,
GetReadDirection(allEventsSlice.ReadDirection),
events);
if that's the case then i'll add an impl and test to the sql providers
GES tests don't assert either way