These are chat archives for DevrexLabs/OrigoDB

4th
Nov 2016
Alan Hemmings
@goblinfactory
Nov 04 2016 13:33
Hi Robert; there's two broken links on the github homepage : https://github.com/DevrexLabs/OrigoDB
in this sentence To learn more, visit the project web site and read the online documentation
Robert Friberg
@rofr
Nov 04 2016 14:02
DOH! I recently updated the links in the bottom of the page but missed these two, thanks for the heads up.
Alan Hemmings
@goblinfactory
Nov 04 2016 14:02
listening to your talk at NDC...I'm sold! ;-D
Are you sticking with the idea of making the server open source as well?
Robert Friberg
@rofr
Nov 04 2016 14:03
Yes, but I ran out of funding so had to slow down development
Alan Hemmings
@goblinfactory
Nov 04 2016 14:03
will dm you
ah, sorry, can't, can you follow me on twitter please? @snowcode
Robert Friberg
@rofr
Nov 04 2016 14:08
followed!
Harry McIntyre
@mcintyre321
Nov 04 2016 15:38
@rofr re: transactions. I'm not sure the command is sufficient protection for the model - if lots of writers start throwing NullRefExceptions due to a bug in a command the deployed code, and the time between bad writes is less than the time to rebuild a clean model, then the system is going to grind to a halt...
Alan Hemmings
@goblinfactory
Nov 04 2016 15:38
sounds testable? i.e. provable.
Harry McIntyre
@mcintyre321
Nov 04 2016 15:40
Well it happened on my app (not on OrigoDB, but from what I understand using the same approach to deal with an unexpected exception)
Harry McIntyre
@mcintyre321
Nov 04 2016 15:48

For a while I had two models, one for reads and one for writes, which doubled memory usage, but allowed the app to mostly continue to work in case of an unhandled exception, but my writes had to wait for the write model to be rebuilt/cloned from the read model.

For N-bad-writers I could have had N-backup models, but that's expensive in RAM terms.

It's not a problem if your code is bug-free, but if it isn't... Or am i missing something?
Harry McIntyre
@mcintyre321
Nov 04 2016 15:59
@goblinfactory BTW I'm not trying to be a downer on the MemoryImage approach, I'm a massive fan!
Robert Friberg
@rofr
Nov 04 2016 16:14
@mcintyre321 Origo has a RoyalFoodTaster kernel which also has double models, sounds exactly what your describing but with a much cool name :smile:
cooler
I think protecting against bad code is impossible
how many relational db's have you seen full of garbage due to bugs?
Robert Friberg
@rofr
Nov 04 2016 16:20
Halting and rebuilding the state from scratch following an unhandled exception will guarantee consistency at the expense of availiability
Robert Friberg
@rofr
Nov 04 2016 16:28
Also, you could choose to ignore exceptions altogether if availability is more important than consistency
Harry McIntyre
@mcintyre321
Nov 04 2016 16:54
So the ORM approach is to clone(rehydrate)-whats-expected-from-the-db-and-lazily-fetch-whats-unexpected and run the command/session against that clone
Would it be so hard to have a writeable proxy that 'lazily fetches' against the read model? I guess the ORM sessions do dirty state tracking also to write back again...
I want to have my cake and eat it, rather than using a taster
Alan Hemmings
@goblinfactory
Nov 04 2016 16:59
would akka.net supervisor pattern (or just use Akka.net + OrigoDB ) avoid a flood of errors taking down a system? (please excuse me...I'm noobie, just started looking at OrigoDB today.)
+1 for RoyalFoodTaster ;-D
Harry McIntyre
@mcintyre321
Nov 04 2016 17:15
I think the supervisor would be akin to the kernel strategy
Robert Friberg
@rofr
Nov 04 2016 17:23
I'm not familiar with akka.net supervisor pattern so can't comment on that one
@mcintyre321 It seems like you are assuming a CRUD pattern, where you have distinct objects that can be modified
But origo is more general than that so it would be impossible (or extremely difficult) to derive what parts of the model will be/have been modified by any given command
Harry McIntyre
@mcintyre321
Nov 04 2016 17:32
Not impossible, merely very very difficult! I rewriting or a custom compiler works
Not 'works', 'might work'
Or a very clever proxy with field interception
Alan Hemmings
@goblinfactory
Nov 04 2016 20:26
just going through the 'getting started' and had some issues with 'cannot access file' ... 'in use' and tracked this down to possibly Resharper shadow copying the assemblies to some temp App_data folder. After telling Resharper to stop doing that, the problem persisted, and dissasembling Origo EngineConfiguration it looks like it might be caused by this method,
    public static string GetBaseDirectory()
    {
      string str = Directory.GetCurrentDirectory();
      string data = AppDomain.CurrentDomain.GetData("DataDirectory") as string;
      if (data != null && data.EndsWith("App_Data"))
        str = data;
      return str;
    }
I might be wrong, but should that be using AppDomain.CurrentDomain.BaseDirectory instead of Directory.GetCurrentDirectory?
the reason I say this, and again... this is just a quick first look, so ... not familiar with the expected behavior, but I believe that GetCurrentDirectory is dangerous to use because it's a mad-arsed global that anyone can set? (and I think Resharper is setting it)
I believe can override this by manually setting configuration at start, but this gives some extra friction for new developers using it. i.e. I'd expect the default behavior (without configuring anything) reading the docs, would be to see a new folder appear in my bin directory, named after the main engine model?
Alan Hemmings
@goblinfactory
Nov 04 2016 20:31
Screen Shot 2016-11-04 at 20.31.15.png
above is the type of error you get if you have resharper shadow copy enabled, and don't override the location setting. I'm on shaky ground here, quite easily something silly I'm missing here.
Alan Hemmings
@goblinfactory
Nov 04 2016 20:40

Also, I think there's a small bug in the quick start guide, this line under executing queries
http://origodb.com/docs/core-0.18/getting-started/quick-start-guide/

// can't serialize lambdas, need local engine
var localEngine = (ILocalEngine<TaskModel>) engine;

I couldnt get that to compile, I had to change it to (LocalEngine<TaskModel>) without the I. Base class instead of interface?

Alan Hemmings
@goblinfactory
Nov 04 2016 21:18
sorry, correction, I had to change to LocalEngineClient<TaskModel>
Robert Friberg
@rofr
Nov 04 2016 22:22
Thanks for the feedback
Robert Friberg
@rofr
Nov 04 2016 22:29
I usually use a non-persistent journal when running automated tests
Also, it's probably important to shutdown properly during teardown
Alan Hemmings
@goblinfactory
Nov 04 2016 22:32
Hi Robert; I see that, been looking at the source and the OrigoDB.Test.NUnit
following the getting started though I can't see a way to 'shutdown' the engine.
Robert Friberg
@rofr
Nov 04 2016 22:33
There is some info on engine lifecycle here: http://origodb.com/docs/core-0.18/client-api/basics/
Alan Hemmings
@goblinfactory
Nov 04 2016 22:34
  public interface IEngine<TModel> where TModel : Model
  {
    TResult Execute<TResult>(Query<TModel, TResult> query);
    void Execute(Command<TModel> command);
    TResult Execute<TResult>(Command<TModel, TResult> command);
    object Execute(Command command);
    object Execute(Query query);
  }
I think the getting started needs to be updated?
I made up my own test classes, will create a new solution and follow the getting started exactly as it's written and see if I still get the same error. The getting started code works, and the test will pass, provided you don't make another method call. The second method call and you get the error, file is in use.
ah, thanks for that link, I missed the Config.Engines.CloseAll();
Robert Friberg
@rofr
Nov 04 2016 22:39
But even better, use the EngineConfiguration.ForIsolatedTests() extension method:
Alan Hemmings
@goblinfactory
Nov 04 2016 22:40
Naw... ForIsolatedTests is in memory; the error ..exception that's being thrown is a real one, The process cannot access the file; doing the test in memory doesnt prove you can use the software!
;-D
sorry, that was meant to be a wink
Robert Friberg
@rofr
Nov 04 2016 22:41
No offense :)
I'm just happy for the feedback
The InMemoryStorage mimics the behavior of the FileStorage, so it serializes commands and appends to
a list of byte arrays, a kind of virtual file system if you will
Alan Hemmings
@goblinfactory
Nov 04 2016 22:43
I see that, and the tests look great; except (possibly?) not enough around real integration "file" tests.
I can help with that.
(possibly ... hopefully, lol!)
here's a small spike project with a failing test
no build needed, just checkout the code and open the solution
the failing test , minimal code that gives the error, is in Tests/HelloOrigoSmokeTests.cs -> Happy_path()
Alan Hemmings
@goblinfactory
Nov 04 2016 22:48
should we take this offline? Or start a new discussion so that other folk can ask questions and not be drowned out by this thread?
Robert Friberg
@rofr
Nov 04 2016 22:50
No, I think it's ok here
Alan Hemmings
@goblinfactory
Nov 04 2016 22:50
updating the test, adding Config.Engines.CloseAll(); and try .. finally { Config.Engines.CloseAll(); still throws the exception.
Robert Friberg
@rofr
Nov 04 2016 22:55
I recall an issue where the file system does not have time to sync
so lock is not released before next test commences
Alan Hemmings
@goblinfactory
Nov 04 2016 22:57
the branch I gave you the link for and Happy_path is changed to run everything in 1 test only, so that removes that from the equation.
Busy re-creating the exact code from getting started, almost finished
Robert Friberg
@rofr
Nov 04 2016 22:59
ok, got it. So looks like maybe the resharper shadow copy may be causing the problem them
then
Alan Hemmings
@goblinfactory
Nov 04 2016 23:04
I'm not convinced.
this test throws System.IO.Exception cannot access the file
        [Test]
        public void Happy_path()
        {
            Config.Engines.CloseAll();

            try
            {
                Console.WriteLine("Hosting the engine");
                //------------------------------------
                var path = Path.Combine(AppDomain.CurrentDomain.BaseDirectory);
                Environment.CurrentDirectory = path;
                var config = new EngineConfiguration()
                {
                    JournalPath = path
                };
                IEngine<VetModel> engine = Engine.For<VetModel>();


                Console.WriteLine("Executing commands");
                // ------------------------------------
                var cat = new Cat()
                {
                    DOB = DateTime.UtcNow,
                    Name = NameGenerator.Generate(Gender.Female)
                };
                engine.Execute(new RegisterBirthCommand(cat));


                Console.WriteLine("Executing query");
                // ------------------------------------
                var localEngine = (LocalEngineClient<VetModel>)engine;
                var cats = localEngine.Execute(db => db.Cats.Where(c => true)).ToArray();
                foreach (var c in cats) Console.WriteLine(c);
            }
            finally
            {
                Config.Engines.CloseAll();
            }
        }
i.e. I've gone overboard and set Environment.CurrentDirectory as well as JournalPath plus ensuring Engines.CloseAll before and after.
adding in thread.Sleeps does not fix either.
Robert Friberg
@rofr
Nov 04 2016 23:11
So which line throws the exception?
Alan Hemmings
@goblinfactory
Nov 04 2016 23:11
oops, code above ..config is not passed to Engine.For
Robert Friberg
@rofr
Nov 04 2016 23:12
That's a classic mistake, it as happened more than just a few times :)
Alan Hemmings
@goblinfactory
Nov 04 2016 23:13
Give me 10 minutes so that I can give you a solid failing test that's well written without being rushed, and will save us both much time and folks reading this (curiously) will not curse me for pasting bollocks code ;-D
quick question, what exact namespace, what exact 'serializable' attribute must be used?
I'm now getting class Foo (Cat) is not marked as serializable, and it is. mmm!
Robert Friberg
@rofr
Nov 04 2016 23:21
That would be System.Serializable
Alan Hemmings
@goblinfactory
Nov 04 2016 23:21
just checking I wasnt going crazy.
quickest way to give you the feedback is for you to clone this 'gist' / spike repo and run the test
clone that, and checkout branch fail2
the failing test is '\Tests\HelloOrigoSmokeTests.cs`
Alan Hemmings
@goblinfactory
Nov 04 2016 23:29
ah gosh, I copied the walkthrough exactly, with just the 2 fixes required, and that works a treat, passing no problem!
Robert Friberg
@rofr
Nov 04 2016 23:30
So the reason for the fail is that the query passed to Execute returns an IQueryable which is not serializable
Put the ToArray() inside the Execute instead and it will work
Alan Hemmings
@goblinfactory
Nov 04 2016 23:31
<blush!>
Robert Friberg
@rofr
Nov 04 2016 23:34
Nah, that's an easy mistake to make
Alan Hemmings
@goblinfactory
Nov 04 2016 23:34
that fixes the serialisation, but still get the process cannot access the file, because it is being used by another process.