These are chat archives for Snaipe/Criterion

29th
Feb 2016
Dominik
@kaidowei
Feb 29 2016 11:07
hmm well, I never developed anything on OS X and I don't have a mac, so this is maybe not the best idea.
I could help with the mocking part. At the moment you (only) have an interface for stubs. (which is nice already)
Franklin Mathieu
@Snaipe
Feb 29 2016 12:21
I don't have a mac either, I just use a VM. There's indeed some work to do over on the API and everything related to usability
setting up a mock currently is simply unfortunate, and we need some macros/functions to set up what gets returned and enforce things like passed parameters or call counts
Dominik
@kaidowei
Feb 29 2016 13:19
lets say, we design the API together, how would you proceed? I haven't worked on a (distributed) open-source project yet...
Franklin Mathieu
@Snaipe
Feb 29 2016 13:24
I would probably draft a few different APIs (without implementing them at first), then pick the one that stands out to be better (in terms of simplicity, usability, ...). In this particular case, I would probably go for:
  1. Setup the mock object & stub behaviour fully from the test function, or
  2. Define the stub (& its behaviour) out of the test function, and setup the mock object from the test function
first would have a procedural style, second would have a declarative style.
All in all, the main point is that whatever the API choices are, we are going to inherit them almost forever, so every design choice must be reviewed & tested.
Dominik
@kaidowei
Feb 29 2016 14:36
Yeah, that's true. The API design is the key.
I recently looked a various existing mocking frameworks for C. Perhaps I should write a summary so we can see, what others have done, find their weaknesses and improve mimicks design based the results.
Franklin Mathieu
@Snaipe
Feb 29 2016 14:38
While we're at it, cmocka has a nice & straightforward mock API, but this still feels a lot manual to my taste (from what I gather, you still have to implement the mock stub manually)
Dominik
@kaidowei
Feb 29 2016 14:40
yeah, that's an advantage of cmock - you don't have to do that there
    boxenv_getvar_ExpectAndReturn("var", "foo", "bar");
    print_boxenv_getvar("var", "foo");
    Mockboxenv_Verify();
thats everything
Franklin Mathieu
@Snaipe
Feb 29 2016 14:41
I was actually talking about cmocka, not cmock, but cmock has the advantage of using simple functions
Dominik
@kaidowei
Feb 29 2016 14:41
I know you were
Franklin Mathieu
@Snaipe
Feb 29 2016 14:41
although cmock requires code generation & ruby on your machine
Dominik
@kaidowei
Feb 29 2016 14:42
right, I don't like that part...
Dominik
@kaidowei
Feb 29 2016 14:49
mmk_expect("malloc", sizeof(int), NULL); test_some_function_that_calls_malloc(); mmk_check();
that would be awesome... where mmk_expect is some macro-magic, that somehow generates the malloc_stub outside the function and adds the parameters to a queue for malloc.
Franklin Mathieu
@Snaipe
Feb 29 2016 14:51
what each mmk_ function should probably be explicit
also there might be a lot more options than just expecting parameters
Dominik
@kaidowei
Feb 29 2016 14:51
so you think more about the Test() syntax of criterion?
Franklin Mathieu
@Snaipe
Feb 29 2016 14:52
idk, maybe. You could describe what a mock does in a structure with a designated initializer syntax, but it's not a requirement
Dominik
@kaidowei
Feb 29 2016 14:53
then you would need some mmk_install(malloc_stub); function inside the tests.
Because I guess you don't want the mocks to be installed for every test.
I'm not that familiar with the preprocessor. Can you generate a function with the same signature as another function?
Franklin Mathieu
@Snaipe
Feb 29 2016 14:54
I'm afraid not, you have to pass the full prototype
Dominik
@kaidowei
Feb 29 2016 14:54
hmm, that's not nice
Franklin Mathieu
@Snaipe
Feb 29 2016 14:56
To be fair, i'd like something that can be easily read. in C++/C#/Java and all, you could chain your mock declaration with .returns(1).returns(2), ... etc, which is nice. You could maybe do something like:
mmk_mock(name, <prototype>) {
    mmk_expect(param0, val);
    mmk_return(1); // 1rst return
    mmk_return(2); // 2nd return
}
internally, mmk_return and all could somewhat setup a duff machine that does all the work
I have no idea if this is feasible though
another possibility is scanning the symbols at runtime, so you simply declare for instance:
static void *__wrap_malloc(size_t sz) {
    mmk_expect(sz, 10);
    mmk_return(NULL);
    mmk_return((void*) -1);
}
and it gets picked up
(edited sample)
Dominik
@kaidowei
Feb 29 2016 15:01
arg
well, the expect could change, so it should be
mmk_expect_and_return(10, NULL); 
mmk_ignore_and_return((void*) -1);
and expect should allow some special MMK_IGNORE_PARAM syntax, so we don't care about a specific parameter
Franklin Mathieu
@Snaipe
Feb 29 2016 15:02
that could work, actually
maybe
mmk_expect_and_return(mmk_any, NULL);
for instance.
Dominik
@kaidowei
Feb 29 2016 15:04
but still, the mmkexpect... syntax should be callable outside the definition, because the expected parameter could change from test to test
and a developer should not have to declare 10 mocks for the same function.
Franklin Mathieu
@Snaipe
Feb 29 2016 15:06
you'd still have to do something like
mmk_mock m = mmk_create_mock("malloc", NULL, my_func);
call_malloc();
mmk_destroy_mock(m);
which also means that the __wrap_* syntax wouldn't really work
or at least not if we don't provide 1rst class support for it in the test framework
which won't really happen, Mimick should work everywhere
Dominik
@kaidowei
Feb 29 2016 15:08
with everywhere you mean with every test framework, right?
Franklin Mathieu
@Snaipe
Feb 29 2016 15:08
yes
This also means btw to add some callbacks to define what happens when mmk_expect fails
Dominik
@kaidowei
Feb 29 2016 15:09
right. For cmock, this was just some macro (re)definition
that could work quite well
Franklin Mathieu
@Snaipe
Feb 29 2016 15:10
now that I also think about it, you could also provide a convenience macro that defines your mock & registers it from within your test if your compiler is from the GNU family
(this works because you can declare nested functions in GCC/Clang, so you can group the declaration & mmk_create_mock at once)
Dominik
@kaidowei
Feb 29 2016 15:12
do you want to make that sacrifice?
Franklin Mathieu
@Snaipe
Feb 29 2016 15:12
This would only be a convenience
the portable way should be the first choice
(i.e. declare your mock function outside the test and call mmk_create_mock from within the test)
Franklin Mathieu
@Snaipe
Feb 29 2016 15:21
On a side note, the windows port is almost ready
I'm still having enormous issues with MASM which somehows messes up with the declaration order of my assembly code
but otherwise it works well on MinGW
Dominik
@kaidowei
Feb 29 2016 15:22
so we have some mmk_declare_mock(funcname, (signature) ... maybe configuration_parameter); and later in the test you have
mmk_funcname_expect(all_parameters_from_signature_or_mmk_all, return_value);
function_that_calls_funcname();
and ideally, we should not have a verify, but directly abort inside the mocked function
Franklin Mathieu
@Snaipe
Feb 29 2016 15:24
the problem here is that it's almost guaranteed to break with the calling conventions
Dominik
@kaidowei
Feb 29 2016 15:24
(the mmk_funcname_expect can be called multiple times, each time adding the parameters to a calling queue)
Franklin Mathieu
@Snaipe
Feb 29 2016 15:25
return_value isn't always eax or rax, if you decide to return a structure with a sizeof > 8
so you can't really expect return_value to be compared to what the assembly trampoline returns
Dominik
@kaidowei
Feb 29 2016 15:25
I don't understand...
the return_value is passed back to function_that_calls_funcname
the automatically generated stub must return that value from the queue, if it was called...
Franklin Mathieu
@Snaipe
Feb 29 2016 15:27
well the main problem here is the low level details, and how functions behave at assembly level
if we do that here, we need some support on the assembly side
and on the assembly side, returns are done either by setting the value of the eax/rax register
(if the size of the return value is <= the size of a pointer)
or it gets passed as a parameter if the size is >
and the parameter in question isn't really guaranteed to be the same
on 32-bits it's simple, you look at the top of your parameter stack, but on 64 bits, it's somewhere in a register or on your stack
and we have no way to predict where it is without the user passing it where it should be explicitely
(Same deal with parameters)
I agree however that these are border cases, since almost no one passes or return full structures to/from a function
but the problem is that we have to document it, and it's a valid use case that is supported by other libraries
Franklin Mathieu
@Snaipe
Feb 29 2016 15:32
(not to mention that if a user unknowingly breaks this rule, the code exposes undefine behaviour)
Dominik
@kaidowei
Feb 29 2016 15:33
not knowing the signature at preprocessor level sucks...
Franklin Mathieu
@Snaipe
Feb 29 2016 15:33
Mmh
I think the best way here would probably be something like mentionned earlier with __wrap_malloc, but instead we register the stub manually
Dominik
@kaidowei
Feb 29 2016 15:34
I'll do some research in the evening, maybe I find something
have to go now...
Franklin Mathieu
@Snaipe
Feb 29 2016 15:34
Okay
Dominik
@kaidowei
Feb 29 2016 19:30
@Snaipe so I did some research. There is no way to get the signature of a function in C.
Franklin Mathieu
@Snaipe
Feb 29 2016 19:31
Yeah, no surprise there
you can still get away with a decent workaround by declaring your function and specifying that the prototypes must match
(warning that if they do not, it is undefined behaviour)
Dominik
@kaidowei
Feb 29 2016 19:32
I also looked at cmocka. they use the --wrap, which you suggested. But that is also unfeasible.
As a user, I don't want to add every function to the build
yeah, that would work. I think it is a good way to go
Dominik
@kaidowei
Feb 29 2016 23:00
okay, I found some macro-magic. Will show you something tomorrow...