These are chat archives for Snaipe/Criterion

14th
Apr 2016
Dominik
@kaidowei
Apr 14 2016 12:44
@Snaipe hi, do you already have concrete plans for the embedded unit tests? (Snaipe/Criterion#69)
Franklin Mathieu
@Snaipe
Apr 14 2016 12:46
No implementation yet, but I was thinking of releasing something twofold
Dominik
@kaidowei
Apr 14 2016 12:46
I'm working at a company that makes home router and maybe we can work something out together? :)
Franklin Mathieu
@Snaipe
Apr 14 2016 12:46
sure
Dominik
@kaidowei
Apr 14 2016 12:46
please elaborate
Franklin Mathieu
@Snaipe
Apr 14 2016 12:46
The first component should probably be some kind of macro, like ExternalTest
which would provide some function hooks to start and communicate with an external test
the second component would be a lightweight version of criterion that would run on embedded device
You'd have your criterion runner on your machine, declare some external tests, and implement those tests with that lightweight library
Dominik
@kaidowei
Apr 14 2016 12:48
ah okay. The second component came to my mind first. I envisioned, that we could compile the tests we already have to run on our embedded device.
without much (any) extra effort
Franklin Mathieu
@Snaipe
Apr 14 2016 12:49
the thing is that criterion tries its best to survive should anything happen
so there should be, somewhere, a supervisor that tracks all the tests
Dominik
@kaidowei
Apr 14 2016 12:50

would be ideal, if we don't have to rewrite any tests...

sure, can't that be done on embedded devices?

Franklin Mathieu
@Snaipe
Apr 14 2016 12:50
if you'd run the tests without the supervisor, then if the test crashes somewhere, you get incomplete/inconsistent results
that actually can't really be done on bare metal
because criterion needs some kind of virtual address space
or the abstraction of process isolation in general
(it relies mostly on fork to isolate each test)
Dominik
@kaidowei
Apr 14 2016 12:52
we have a small linux on our systems
Franklin Mathieu
@Snaipe
Apr 14 2016 12:52
so sure, you can run criterion (the runner) itself on embedded devices, granted you have fork() and all
Dominik
@kaidowei
Apr 14 2016 12:52
with processes, fork and everything
Franklin Mathieu
@Snaipe
Apr 14 2016 12:52
in that case everything should be fine
the most important part would be to reduce the memory footprint for those cases
Dominik
@kaidowei
Apr 14 2016 12:53
yes
Franklin Mathieu
@Snaipe
Apr 14 2016 12:55
I'm measuring about ~200kB of allocated memory (total, not accounting for intermediate frees), it could have less allocations I'd say
Although currently I think that this is mostly the work of nanomsg
Dominik
@kaidowei
Apr 14 2016 12:55
Mem: 113564K used, 125536K free :)
Franklin Mathieu
@Snaipe
Apr 14 2016 12:56
how many tests on your end ?
(the numbers I was giving was for 1 empty test)
Dominik
@kaidowei
Apr 14 2016 12:56
oh okay
at the moment about 100
I have a hard time introducing unit testing to the whole team
Franklin Mathieu
@Snaipe
Apr 14 2016 12:58
I think the first big culprit here is cr_assert and its derivatives
because it allocates each message on the heap, and sends it through the wire whenever the assertion passed or failed
I could probably make a configurable variation that only sends the message if it failed, and use a fixed stack space
maybe check if CRITERION_EMBEDDED is defined
Dominik
@kaidowei
Apr 14 2016 13:00
sounds good
so we don't have to change anything, just compile with different parameter, like that
Franklin Mathieu
@Snaipe
Apr 14 2016 13:01
yeah
I was thinking of maybe introducing a criterion/embedded.h
but this will probably be more for bare metal
or maybe both
it could define CRITERION_EMBEDDED and include criterion.h
another problem is stats
Dominik
@kaidowei
Apr 14 2016 13:03
we currently use a selfmade "criterion superheader" with logging, hooks, and so on
so for us there is no problem.
Franklin Mathieu
@Snaipe
Apr 14 2016 13:03
I see
Dominik
@kaidowei
Apr 14 2016 13:04
what is the problem with stats?
Franklin Mathieu
@Snaipe
Apr 14 2016 13:04
well, they also take a large space
you have stats for each tests, each suite, and each assertions
I'm not sure how I should handle that, because they are used by output reporters
Dominik
@kaidowei
Apr 14 2016 13:05
hmm I guess for embedded, it's okay to only track the result and the first failing assertion
or "forbid" output reporters
or dump that to a file maybe?
oh hmm maybe there is no file system :D
Franklin Mathieu
@Snaipe
Apr 14 2016 13:07
I guess we could strip down some functionality if needed
One thing is sure: if stats are to be scaled down memory-wise, assertion stats needs to go away
So sure, output reporters would produce less accurate reports
(basically it would say that something failed, but would not include the assertion message)
but it would produce at least something
Dominik
@kaidowei
Apr 14 2016 13:10
something like Test(embedded, test1, .verbose=true){...} could be handy
you run all tests, see that one fails
and recompile with verbose to true for that test
and criterion only tracks stats for that verbose test :)
Franklin Mathieu
@Snaipe
Apr 14 2016 13:11
That's a good idea actually
You can even put that on a test suite
or cherry pick tests as you said
And for the rest, should anyone want all the functionalities of a full runner, they still can make the tests on the embedded device report to a supervising runner
The nanomsg refactor completely allows that anyways
Dominik
@kaidowei
Apr 14 2016 13:13
via the embedded test extension you mentioned?
Franklin Mathieu
@Snaipe
Apr 14 2016 13:13
If you want the process to be bound/automated, yes
Dominik
@kaidowei
Apr 14 2016 13:13
btw. how do you envision the communication between embedded client and unittest-server
Franklin Mathieu
@Snaipe
Apr 14 2016 13:13
otherwise you can simply specify --client and --server on the respective ends
Dominik
@kaidowei
Apr 14 2016 13:13
ah okay
tcp?
Franklin Mathieu
@Snaipe
Apr 14 2016 13:14
anything nanomsg supports
the code isn't there yet, but you'll be able to put --client tcp://192.168.x.x:port and --server tcp://*:port
(or any nanomsg url, so ipc:// for pipes, ws:// for websockets, ...)
Dominik
@kaidowei
Apr 14 2016 13:16
wow, I see... didn't look at nanomessage before, but it seems quite awesome
Franklin Mathieu
@Snaipe
Apr 14 2016 13:17
Yeah, it had a bit of a hiatus period (with the maintainer leaving its job), but since the maintainer came back very recently, I don't have to maintain my own fork
Dominik
@kaidowei
Apr 14 2016 13:21
the client/server option is quite exciting.
Franklin Mathieu
@Snaipe
Apr 14 2016 13:21
It's currently working well for ipc, but I still have some usability issues on the matter
mostly because the runner doesn't really know what to expect
so currently the runner waits indefinitely and reports anything a client sends
and does its summary if SIGINT is caught
Dominik
@kaidowei
Apr 14 2016 13:23
you don't have a protocol yet, right?
Franklin Mathieu
@Snaipe
Apr 14 2016 13:23
we have, in fact
Dominik
@kaidowei
Apr 14 2016 13:23
like:
client sends all tests, it has
server says: start test1
client reports result
...
so why do you have to wait indefinitely
Franklin Mathieu
@Snaipe
Apr 14 2016 13:24
well, because the server doesn't know what (or how many) clients there are
hence ExternalTest
which would basically tell at compile time "hey, expect these remote tests"
Dominik
@kaidowei
Apr 14 2016 13:26
if you start the same program with --client and --server on the other side, it should know all tests, right?
Franklin Mathieu
@Snaipe
Apr 14 2016 13:26
ah, yes, but the purpose of --server is to also serve tests that are not in the scope of the runner itself
which is why i'm using it on a runner with no tests
but with the same executable, it could be possible
although hardly convenient
Dominik
@kaidowei
Apr 14 2016 13:27
okay, so terminating is a problem for the general purpose runner, BUT
if the client first describes all tests, it has, the runner can output a statistic after that specific client finished
Franklin Mathieu
@Snaipe
Apr 14 2016 13:30
it kinda does that, but what you're telling me is to make the server only expect one client, right?
So the server stops after the client say it's done
Dominik
@kaidowei
Apr 14 2016 13:31
well yeah. that's like running local tests except they're not
Franklin Mathieu
@Snaipe
Apr 14 2016 13:31
It could work
but I'd have to make some more options, because there are many ways something could go wrong
The first (big) problem I encountered while developping --client and --server is the case where the client crashes
Dominik
@kaidowei
Apr 14 2016 13:33
I'd wrap that in a script for my environment:
1) compile both programs (server for host, client for embedded)
2) upload client
3) start server
4) run client on embedded
5) wait for server to finish
and then look at the resulst
the client main thread should send a keepalive every x seconds..
Franklin Mathieu
@Snaipe
Apr 14 2016 13:34
right
This would work well but needs a bit of refactoring
first, the protocol describes a much more open test hierarchy model
(in which tests can have tests themselves)
the client is considered as a test by default, with many subtests
however, the runner code doesn't account for that change (yet)
I plan to do so once the transition from the pre-2.2.x I/O layer to the nanomsg I/O layer is completed and stable
(and call it 2.3.0)
2.3.0 would also deprecate ReportHooks, and I will start working on the refactor of the hierachy model inside the runner
since those are breaking changes to the API, the ABI, and the core mechanics, it will have to be done for 3.0.0
Franklin Mathieu
@Snaipe
Apr 14 2016 13:40
Hopefully the transition to nanmsg will be stable soon, but I have limited time and issues on windows that are delaying it
Dominik
@kaidowei
Apr 14 2016 13:41
meh windows...
Franklin Mathieu
@Snaipe
Apr 14 2016 13:41
plus my patch to make nanomsg fork()-resilient has to get merged in the nanomsg repository before that
Dominik
@kaidowei
Apr 14 2016 13:42
I see. Is there anything (not mac/windows related) that I can do to help?
Franklin Mathieu
@Snaipe
Apr 14 2016 13:44
Implementation wise, there's still work on the side of implementing the options that would make client/server convenient
You could try to implement --client and --server to optionally take URLs that gets passed to the runner, if you want (for instance)
Dominik
@kaidowei
Apr 14 2016 13:45
sounds good. You develop in bleeding?
Franklin Mathieu
@Snaipe
Apr 14 2016 13:45
(Actually, I don't remember, is there actually a --client ? iirc there's a --single for running a single test but no actual --client)
and yes, all development is on bleeding
Ah, yes, there isn't a --client yet.
and --server is actually called --wait
oh right, I remember why
--wait runs the tests on the current runner and waits for external clients
Dominik
@kaidowei
Apr 14 2016 13:49
where is the current option parsing done?
Franklin Mathieu
@Snaipe
Apr 14 2016 13:49
src/entry/params.c
you'll need to add an url field to criterion_options too
Dominik
@kaidowei
Apr 14 2016 13:51
okay. Anything else I should know?
Franklin Mathieu
@Snaipe
Apr 14 2016 13:51
I think a good starting point would be in src/core/runner.c, where the bind_server happens
you'll need to change bind_server and connect_client (defined in src/protocol/connect.c) to take in a (possibly NULL) nanomsg url
(if the parameter is NULL, the url should be the one currently generated by sprintf)
and I guess that's all
Dominik
@kaidowei
Apr 14 2016 13:54
:+1: is there a deadline?
(e.g. do you need that very soon?)
Franklin Mathieu
@Snaipe
Apr 14 2016 13:54
Nope
Dominik
@kaidowei
Apr 14 2016 13:54
okay, so no release date scheduled for 2.3?
Franklin Mathieu
@Snaipe
Apr 14 2016 13:55
I think this should be post 2.3
Dominik
@kaidowei
Apr 14 2016 13:55
ah okay.
Franklin Mathieu
@Snaipe
Apr 14 2016 13:55
2.3 has been delayed for a long time, and should be the transition to nanomsg
And while there is no scheduled date for 2.3 because I have limited time to work on criterion, the release should happen ASAP
(i.e. when all issues are cleared on the windows port & my nanomsg patch is merged)
Dominik
@kaidowei
Apr 14 2016 13:58
so have you talked to the maintainer?
Franklin Mathieu
@Snaipe
Apr 14 2016 13:58
My patch is currently being reviewed, and there are some issues with leaks sometimes
(mostly happenning if a fork happens while another thread is send/recv-ing some data)
We're trying to figure something out
if you want you can follow the progress over at nanomsg/nanomsg#587
Dominik
@kaidowei
Apr 14 2016 14:35
thanks