Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Andrew McCluskey
    @arm61

    From the minutes of the meeting in Abingdon last month, the following task are classed as within our remit:

    • Data reduction and analysis reproducibility
    • Guidelines

    I am looking for ideas for how to convert these "tasks" into a realistic goal to be achieved in the next six months.

    Andrew Nelson
    @andyfaff
    Some ideas in no particular order:
    • Paper outlining guidelines to show scientists publishing with NR/XRR could achieve reproducibility in reduction/analysis. Perhaps something along the lines of the small-angle bio scattering paper (but reproducible).
    Andrew Nelson
    @andyfaff
    • website listing papers that are considered reproducible (to act as an example for others)
    • website explaining what reproducibility is, and why it's important, and how it can be achieved for NR/XRR.
    • guidelines from software maintainers on how reproducibility is achieved for their reduction/analysis packages.
    • Outreach --> talks at conferences/workshops, with talks being placed in a repository for posterity.
    Andrew McCluskey
    @arm61
    I like these ideas. Would you be interested in putting together a couple of sentances on the second bullet point (@andyfaff)?
    I can start a list for the first bullet point (although currently I am only aware of those with refnx where the python scripts are available [basically the refnx paper and a couple of mine]). @bmaranville is there any examples for Refl1D that could be included in the list
    On the fourth point, how do we feel about the working group putting in an abstract for SXNS16 to try and get a talk/poster highlighting the reproducibilty woes?
    Finally a point that I would like to add:
    • Guidelines on how instrument scientists and facilities should ensure that users are leaving with data that can be reproducibly reduced?
    Brian Benjamin Maranville
    @bmaranville
    We can certainly provide examples for Refl1d, which should be reproducible. Paul K. has worked out a scheme where the seeds for the random population initializers can be specified, I think - so that running the same fits again should give exactly the same answer (for some of the fitting engines supported by BUMPS, not necessarily all of them)
    I wonder if we should focus on some of the other tasks first though, before reaching out with a publication - if we haven't established standard practices yet (which is something else this organization is aiming to achieve) then wouldn't that be the first task? I would imagine the paper would be a way to advertise/explain these standard practices.
    Brian Benjamin Maranville
    @bmaranville
    For reproducibility in reduction, we already have a publication that explains how the re-runnable reduction recipe is embedded in the exported files for our reduction utility - see http://journals.iucr.org/j/issues/2018/05/00/po5131/index.html#SEC4.1.3
    Andrew McCluskey
    @arm61
    @bmaranville while the reductus example works for you, my point was more about general guidelines that we can all agree to follow. Things like:
    • provide users with details of all of the reduction steps
    • have documentation that outlines the math(s) in each step
    • etc.
    Andrew McCluskey
    @arm61
    By publication do you mean a poster/talk at SXNS16? if so I agree that we need somethign to advertise first!
    @bmaranville could you possibly put together a list with a view examples of papers published using Refl1D that we can include as examples of “best practice”?
    Andrew Nelson
    @andyfaff
    A full on journal article probably is a bit further down the track, I was just putting the idea out there. I think submitting an abstract for talk at SXNS16 is a good idea because it's a good form of outreach. There's nothing stopping a poster submission either.
    W.r.t. reproducible papers, it's my view that for a paper to be reproducible it needs to have enough information in the paper + supp_info to fully replicate the analysis. This would normally involve depositing the reduced data, the analysis scripts, and enough information to install the same software stack.
    So what we're looking at is a blurb on why reproducibility is important, followed by a fleshing out of the standard practices.
    Andrew Nelson
    @andyfaff
    @arm61 how deeply have you thought about the git repo architecture? If we're going to start generating content for this workgroup then the setup will need to scale with the output of 4 workgroups.
    Andrew Nelson
    @andyfaff
    Andrew Nelson
    @andyfaff
    My initial attempts on writing text can be found at https://gist.github.com/andyfaff/643bef3bf27053c1bc3b6c0d024a7cfb
    Wojciech Potrzebowski
    @wpotrzebowski
    I like the idea of paper and agree that it something that is bit further down the track. I also think the website listing reproducible papers my be a good idea but maybe reference to data/fit and link to the paper will be more valuable? It seems to me that PANOSC project (https://www.panosc.eu/) with FAIR data activities can be something to look at. I can explore possibilities as ESS is involved in the project
    Andrew McCluskey
    @arm61

    I like the idea of paper and agree that it something that is bit further down the track. I also think the website listing reproducible papers my be a good idea but maybe reference to data/fit and link to the paper will be more valuable? It seems to me that PANOSC project (https://www.panosc.eu/) with FAIR data activities can be something to look at. I can explore possibilities as ESS is involved in the project

    Perhaps a future goal should be something like this for data reduction/analysis of reflectometry data (the actual data is the scope of the File Formats WG).
    https://doi.org/10.1107/S1600576719005922

    adrianrennie
    @adrianrennie
    A comment about scope: perhaps reproducibility has also to include measurement aspects and understanding the differences between measurements on the same sample with different instruments (i.e. resolution etc.) as well as the data reduction process.
    Andrew Nelson
    @andyfaff
    Indeed, it's important to get something going quickly as well.
    Andrew McCluskey
    @arm61
    I think that a clear (longterm) goal at somepoint for reproducibility to consider is a round robin type idea (this essentially can ensure all aspects of the measurement/reduction/analysis is reproducible. The idea being that hopefully the result is robust between facilities and the reduction/analysis is reproducible within a facility in the round robin. Limme know if that doesn’t make sense.
    IMG_1206.JPG
    See the attached image for my definitions for robust and reproducible https://ibb.co/DzdPtMD (replace data with sample though)
    Andrew Nelson
    @andyfaff
    There is already a round robin sample out there. I've been very slow at making it move around facilities though. It was created with this idea in mind
    Andrew McCluskey
    @arm61
    Do we know who is leading the round robin process (might be useful to speak to them if they are not already involved)
    adrianrennie
    @adrianrennie
    Yes - Andrew Nelson.
    Andrew McCluskey
    @arm61
    @adrianrennie :laughing:
    Andrew McCluskey
    @arm61
    Happy New Year all! I have started to put together a short list of “achievable” goals to be carried out by SXNS16 in June this year. There is a meeting of the ORSO chairs in a couple of weeks time, and I would like to have these goals agreed upon with specific (named) action points before then. Any and all thoughts welcome https://github.com/reflectivity/reproducibility/blob/master/goals.md
    Andrew McCluskey
    @arm61
    A quick follow up on this!
    Brian Benjamin Maranville
    @bmaranville
    @arm61 I have examples from Refl1D, just trying to figure out the best place to stash them. Do you want me to put them in a folder in the reflectivity/reproducibility repo?
    Andrew McCluskey
    @arm61
    That would be perfect!
    Brian Benjamin Maranville
    @bmaranville
    Done
    alexandros-koutsioumpas
    @alexandros-koutsioumpas
    Sorry for joining the discussion a bit late.. Reading through the posts, I would like to emphasise the importance of discussing "measurement aspects" as pointed out by @adrianrennie . From past experiments mainly with neutrons, people at different facilities use different hardware (finger or 2D detectors for example) and their practice for estimating background, specular reflection etc.. might be quite different.. In this respect including good practices and definition of some key observables might be help.
    Andrew Nelson
    @andyfaff
    Measurement aspects... hmm... these are good to discuss. I think sometimes that scientists think that there is one 'True' way of doing things. However, I think there is actually a lot more variation than most people (users?) realise. Even if some measurement aspects seem obvious to some people, they may not be obvious to the entire NR community. (I found this out the hard way recently)
    @arm61, could you make me a member of the reflectivity community on github?
    Andrew McCluskey
    @arm61
    @andyfaff done!