Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 07 2019 15:05
    colindaven starred ewels/MegaQC
  • Dec 10 2018 16:46
    vzg100 starred ewels/MegaQC
  • Dec 10 2018 16:39
    sridhar0605 starred ewels/MegaQC
  • Nov 21 2018 11:15
    ewels commented #39
  • Nov 20 2018 23:23
    lutsik commented #39
  • Nov 20 2018 23:13
    ewels commented #39
  • Nov 20 2018 23:10
    lutsik commented #39
  • Nov 20 2018 22:59
    ewels commented #39
  • Nov 20 2018 22:27
    lutsik commented #39
  • Nov 08 2018 13:08
    ebete starred ewels/MegaQC
  • Oct 17 2018 13:26
    tsnowlan commented #70
  • Oct 11 2018 14:09
    ewels commented #67
  • Oct 11 2018 12:06
    shafqatb commented #67
  • Oct 10 2018 15:14
    ewels labeled #70
  • Oct 10 2018 15:14
    ewels labeled #70
  • Oct 10 2018 15:14
    ewels opened #70
  • Oct 10 2018 11:23
    ewels commented #69
  • Oct 10 2018 11:09
    tsnowlan commented #69
  • Oct 10 2018 10:06
    ewels commented #68
  • Oct 10 2018 10:05

    ewels on FLASK_DEBUG

    (compare)

Albert Vilella
@avilella
And where is the .json found by default? On the dir where it's called?
Phil Ewels
@ewels
It's my intention to back-port the many thousands of these files that we have generated since v1.3 MultiQC was released..
in multiqc_data/multiqc_data.json
Albert Vilella
@avilella
Gotcha
Phil Ewels
@ewels
wherever MultiQC was run
Albert Vilella
@avilella
Yes so wrt the previous question, I can see how the json bamtools_stats section does not have a "name" kv:
"multiqc_bamtools_stats": { "CEG99-707-64IC_S64_L00.bml.GRCh38.karyo": { "mapped_reads": 50568874.0, "mapped_reads_pct": 99.6749,
rather, the sample name is the key of the list of key-values for bamtools_stats
That's probably what's confusing MegaQC
Phil Ewels
@ewels
yeah, it's the key that I expected
I was hoping that I'd mapped that to something that could be filtered for in MegaQC
but it's entirely possible that I never did
Albert Vilella
@avilella
Gotcha
Another question: if I create a .csv or .tsv file with a bunch of metrics and name it with the suffix that MultiQC picks up, will they be in the .json and therefore in MegaQC?
I have done this in the past by adding _mqc.tsv to the file name
Phil Ewels
@ewels
yes, they should be
one feature that I would like to add and haven't is the option of having content in such files that is "silent" - that is, that never makes it in to MultiQC reports, but that would then be fed in to MegaQC.
Our use case for this would be to pass metadata from our LIMS through MultiQC and on to MegaQC, without ever printing it in the reports which are sent to users
In this way, MultiQC becomes a kind of normalisation filter for multiple infrastructure back-ends
(by accident)
Albert Vilella
@avilella
The custom _mqc.tsv seem to be loaded into MegaQC
It does give me though this message: Finished writing plot data (0 cfg, 164 data points) for report 22
which makes me think I need to add something to instruct how I want each field plotted. Any ideas how @ewels ?
Phil Ewels
@ewels
I'm not sure where that's being printed sorry
Albert Vilella
@avilella
Let's say I have a custom _mqc.tsv like the one below: foo name 1 sample1 2 sample2 3 sample3 4 sample4 5 sample5 6 sample6 7 sample7 8 sample8 9 sample9 10 sample10
foo     name
1       sample1
2       sample2
3       sample3
4       sample4
5       sample5
6       sample6
7       sample7
8       sample8
9       sample9
10      sample10
Do I need to add a config section somewhere so that foo becomes available to the list of metrics in MegaQC?
Ok, forgot what I said, it turns out the _mqc.tsv files are loaded into the .html report but are not found in the .json
So I will have to find a way to craft the .json file from the .tsv directly
Phil Ewels
@ewels
I'm a bit lost - the table renders in the MultiQC report correctly, but then isn't in the JSON?
I'm quite surprised if that's the case
I just doublechecked with some custom content files at my end and it worked ifne
{
    "report_general_stats_headers": [], 
    "config_version": "1.8.dev0 (ef579dc)", 
    "report_general_stats_data": [], 
    "config_creation_date": "2019-03-07, 17:42", 
    "config_title": null, 
    "config_report_comment": null, 
    "config_short_version": "1.8.dev0", 
    "config_intro_text": null, 
    "report_saved_raw_data": {
        "multiqc_my_section_with_a_table": {
            "samp_1": {
                "col1": 12.523, 
                "col2": 34.0
            }, 
            "samp_2": {
                "col1": 56.947, 
                "col2": 78.0
            }
        }
    }, 
    "report_data_sources": {}, 
    "report_multiqc_command": "/Users/ewels/miniconda2/envs/work/bin/multiqc -f table_headers_mqc.txt", 
    "config_analysis_dir": [
        "table_headers_mqc.txt"
    ], 
    "config_report_header_info": null, 
    "config_analysis_dir_abs": [
        "/Users/ewels/GitHub/MultiQC_TestData/data/custom_content/embedded_config/table_headers_mqc.txt"
    ], 
    "config_subtitle": null, 
    "report_plot_data": {}, 
    "config_script_path": "/Users/ewels/GitHub/MultiQC/multiqc/utils", 
    "config_git_hash": "ef579dcdcdb12a56ea516b2354472cb5f640353c"
}
Albert Vilella
@avilella
So in my case it produces a json with no data
avilella@avilellaM710t:~/multiqc/test$ find $PWD/ -name "*.json" | xargs cat
{
    "report_general_stats_headers": [], 
    "config_version": "1.5dev", 
    "report_general_stats_data": [], 
    "config_creation_date": "2019-03-07, 16:48", 
    "config_title": null, 
    "config_report_comment": null, 
    "config_short_version": "1.5dev", 
    "config_intro_text": null, 
    "report_saved_raw_data": {}, 
    "report_data_sources": {}, 
    "report_multiqc_command": "/usr/local/bin/multiqc --force --filename /home/avilella/multiqc/test/report__multiqc /home/avilella/multiqc/test", 
    "config_analysis_dir": [
        "/home/avilella/multiqc/test"
    ], 
    "config_report_header_info": null, 
    "config_analysis_dir_abs": [
        "/home/avilella/multiqc/test"
    ], 
    "config_subtitle": null, 
    "report_plot_data": {}, 
    "config_script_path": "/usr/local/lib/python2.7/dist-packages/multiqc/utils", 
    "config_git_hash": null
}
avilella@avilellaM710t:~/multiqc/test$ cat *.txt
# plot_type: 'table'
# section_name: 'My section with a table'
# description: 'a custom text introduction (a few sentences) for this section'
# pconfig:
#     namespace: 'Cust Data'
# headers:
#     col1:
#         title: 'My Column'
#         description: 'This is a longer hover text for my column'
#         format: '{:,.0f}'
#     col2:
#         title: 'Second Column'
#         description: 'Hover description text'
Sample            col1            col2
samp_1            12.523          34
samp_2            56.947          78
avilella@avilellaM710t:~/multiqc/test$ #/usr/local/bin/multiqc --force --filename /home/avilella/multiqc/test/report__multiqc /home/avilella/multiqc/test
Phil Ewels
@ewels
Hmm, MultiQC v1.5dev
that's pretty old
we're on 1.8dev now
My guess is that you need to update MultiQC for this to work with custom content then
Albert Vilella
@avilella
I updated to 1.7 and it worked, thanks.
Phil Ewels
@ewels
Ok great!
Sehrish Kanwal
@skanwal

Hi Phil,
I am testing the megaqc upload function (locally on my laptop) to get data into megaQC server.
I am getting Error - This function requires MultiQC to be installed..
Following this issue on github https://github.com/ewels/MegaQC/issues/66, I have multiqc installed on my laptop and I am running the upload command from the same environment where I used multiqc to generate the report/json file.
For context, I installed multiqc via conda. The multiqc conda environment is also active in my session. Trying from multiqc.utils import config as multiqc_config on python interpreter also works fine..

Any idea what could be the underlying issue?

Phil Ewels
@ewels
Hi Sehrish - that sounds very odd!
Are you super-sure that everything is definitely in the same environment? By checking which megaqc and which multiqc..?
Ah, and it’s definitely a recent version of MultiQC?
The code that triggers that message isn’t doing anything particularly special: https://github.com/ewels/MegaQC/blob/master/megaqc/commands.py#L173
Sehrish Kanwal
@skanwal

Ah, that’s going to be interesting.

which multiqc > /Users/kanwals/miniconda3/envs/multiqc/bin/multiqc
which megaqc > /Users/Kanwals/miniconda3/bin//megaqc

As you can tell, I installed both tools in separate conda environment. Aslo, used pip for megaqc installation.
Should I try installing megaqc in multiqc environment.?
Multiqc version is 1.7, so should definitely be not an issue.

Phil Ewels
@ewels
Yes they need to be installed in the same environment
Otherwise they are using different python installations, so they can’t see each other