ewels/MegaQC

A web application to collect and visualise data from multiple MultiQC reports. https://github.com/ewels/MegaQC

Albert Vilella

@avilella

Gotcha

Another question: if I create a .csv or .tsv file with a bunch of metrics and name it with the suffix that MultiQC picks up, will they be in the .json and therefore in MegaQC?

I have done this in the past by adding _mqc.tsv to the file name

Phil Ewels

@ewels

yes, they should be

one feature that I would like to add and haven't is the option of having content in such files that is "silent" - that is, that never makes it in to MultiQC reports, but that would then be fed in to MegaQC.

Our use case for this would be to pass metadata from our LIMS through MultiQC and on to MegaQC, without ever printing it in the reports which are sent to users

In this way, MultiQC becomes a kind of normalisation filter for multiple infrastructure back-ends

(by accident)

Albert Vilella

@avilella

The custom _mqc.tsv seem to be loaded into MegaQC

It does give me though this message: Finished writing plot data (0 cfg, 164 data points) for report 22

which makes me think I need to add something to instruct how I want each field plotted. Any ideas how @ewels ?

Phil Ewels

@ewels

I'm not sure where that's being printed sorry

Albert Vilella

@avilella

Let's say I have a custom _mqc.tsv like the one below:

foo     name
1       sample1
2       sample2
3       sample3
4       sample4
5       sample5
6       sample6
7       sample7
8       sample8
9       sample9
10      sample10

foo     name
1       sample1
2       sample2
3       sample3
4       sample4
5       sample5
6       sample6
7       sample7
8       sample8
9       sample9
10      sample10

Do I need to add a config section somewhere so that foo becomes available to the list of metrics in MegaQC?

Ok, forgot what I said, it turns out the _mqc.tsv files are loaded into the .html report but are not found in the .json

So I will have to find a way to craft the .json file from the .tsv directly

Phil Ewels

@ewels

I'm a bit lost - the table renders in the MultiQC report correctly, but then isn't in the JSON?

I'm quite surprised if that's the case

I just doublechecked with some custom content files at my end and it worked ifne

This file: https://github.com/ewels/MultiQC_TestData/blob/master/data/custom_content/embedded_config/table_headers_mqc.txt

{
    "report_general_stats_headers": [], 
    "config_version": "1.8.dev0 (ef579dc)", 
    "report_general_stats_data": [], 
    "config_creation_date": "2019-03-07, 17:42", 
    "config_title": null, 
    "config_report_comment": null, 
    "config_short_version": "1.8.dev0", 
    "config_intro_text": null, 
    "report_saved_raw_data": {
        "multiqc_my_section_with_a_table": {
            "samp_1": {
                "col1": 12.523, 
                "col2": 34.0
            }, 
            "samp_2": {
                "col1": 56.947, 
                "col2": 78.0
            }
        }
    }, 
    "report_data_sources": {}, 
    "report_multiqc_command": "/Users/ewels/miniconda2/envs/work/bin/multiqc -f table_headers_mqc.txt", 
    "config_analysis_dir": [
        "table_headers_mqc.txt"
    ], 
    "config_report_header_info": null, 
    "config_analysis_dir_abs": [
        "/Users/ewels/GitHub/MultiQC_TestData/data/custom_content/embedded_config/table_headers_mqc.txt"
    ], 
    "config_subtitle": null, 
    "report_plot_data": {}, 
    "config_script_path": "/Users/ewels/GitHub/MultiQC/multiqc/utils", 
    "config_git_hash": "ef579dcdcdb12a56ea516b2354472cb5f640353c"
}

Albert Vilella

@avilella

So in my case it produces a json with no data

avilella@avilellaM710t:~/multiqc/test$ find $PWD/ -name "*.json" | xargs cat
{
    "report_general_stats_headers": [], 
    "config_version": "1.5dev", 
    "report_general_stats_data": [], 
    "config_creation_date": "2019-03-07, 16:48", 
    "config_title": null, 
    "config_report_comment": null, 
    "config_short_version": "1.5dev", 
    "config_intro_text": null, 
    "report_saved_raw_data": {}, 
    "report_data_sources": {}, 
    "report_multiqc_command": "/usr/local/bin/multiqc --force --filename /home/avilella/multiqc/test/report__multiqc /home/avilella/multiqc/test", 
    "config_analysis_dir": [
        "/home/avilella/multiqc/test"
    ], 
    "config_report_header_info": null, 
    "config_analysis_dir_abs": [
        "/home/avilella/multiqc/test"
    ], 
    "config_subtitle": null, 
    "report_plot_data": {}, 
    "config_script_path": "/usr/local/lib/python2.7/dist-packages/multiqc/utils", 
    "config_git_hash": null
}
avilella@avilellaM710t:~/multiqc/test$ cat *.txt
# plot_type: 'table'
# section_name: 'My section with a table'
# description: 'a custom text introduction (a few sentences) for this section'
# pconfig:
#     namespace: 'Cust Data'
# headers:
#     col1:
#         title: 'My Column'
#         description: 'This is a longer hover text for my column'
#         format: '{:,.0f}'
#     col2:
#         title: 'Second Column'
#         description: 'Hover description text'
Sample            col1            col2
samp_1            12.523          34
samp_2            56.947          78
avilella@avilellaM710t:~/multiqc/test$ #/usr/local/bin/multiqc --force --filename /home/avilella/multiqc/test/report__multiqc /home/avilella/multiqc/test

Phil Ewels

@ewels

Hmm, MultiQC v1.5dev

that's pretty old

we're on 1.8dev now

My guess is that you need to update MultiQC for this to work with custom content then

Albert Vilella

@avilella

I updated to 1.7 and it worked, thanks.

Phil Ewels

@ewels

Ok great!

Sehrish Kanwal

@skanwal

Hi Phil,
I am testing the megaqc upload function (locally on my laptop) to get data into megaQC server.
I am getting Error - This function requires MultiQC to be installed..
Following this issue on github https://github.com/ewels/MegaQC/issues/66, I have multiqc installed on my laptop and I am running the upload command from the same environment where I used multiqc to generate the report/json file.
For context, I installed multiqc via conda. The multiqc conda environment is also active in my session. Trying from multiqc.utils import config as multiqc_config on python interpreter also works fine..

Any idea what could be the underlying issue?

Phil Ewels

@ewels

Hi Sehrish - that sounds very odd!

Are you super-sure that everything is definitely in the same environment? By checking which megaqc and which multiqc..?

Ah, and it’s definitely a recent version of MultiQC?

The code that triggers that message isn’t doing anything particularly special: https://github.com/ewels/MegaQC/blob/master/megaqc/commands.py#L173

Sehrish Kanwal

@skanwal

Ah, that’s going to be interesting.

which multiqc > /Users/kanwals/miniconda3/envs/multiqc/bin/multiqc
which megaqc > /Users/Kanwals/miniconda3/bin//megaqc

As you can tell, I installed both tools in separate conda environment. Aslo, used pip for megaqc installation.
Should I try installing megaqc in multiqc environment.?
Multiqc version is 1.7, so should definitely be not an issue.

Phil Ewels

@ewels

Yes they need to be installed in the same environment

Otherwise they are using different python installations, so they can’t see each other

Pip installs within a conda environment should be fine

which pip should show it in the conda env.

Sehrish Kanwal

@skanwal

Phil, tried reinstalling megaQC in the multiQC conda environment using pip. The installation went okay.
Both the tools are now installed in the same environment.
which megaqc > /Users/kanwals/miniconda3/envs/multiqc/bin/megaqc
which multiqc > /Users/kanwals/miniconda3/envs/multiqc/bin/multiqc

For testing, ran multiqc on a sample, and the report was successfully posted to megaqc server.
But upload is again throwing the same issue:

megaqc upload multiqc_data/multiqc_data.json
This is MegaQC v0.1.dev1

Env: test
 * Database type: sqlite
 * Database path: sqlite:////Users/kanwals/miniconda3/lib/python3.6/site-packages/test.db
Error - This function requires MultiQC to be installed

Phil Ewels

@ewels

Strange. Could you make an issue about this on the GitHub repo please?

I have no ideas as to how or why this could happen now though

It’s nothing special about MegaQC or MultiQC here - it’s just a simple python import.

Sehrish Kanwal

@skanwal

Thanks, Phil.
Sure, I'll open an issue on GitHub.

Michael Milton

@TMiguelT

I'd appreciate feedback from anyone on my Dash integration PR: ewels/MegaQC#81

Michael Milton

@TMiguelT

@ewels if you can get back to me about the dash PR and also the control chart thresholds, that would be great

I have an upcoming meeting at my work where I'll be talking about MegaQC, and if I can implement and then show off some of these features, there's a much better chance I'll be given time to work on MegaQC

Michael Milton

@TMiguelT

In particular I want to implement something for the alert thresholds that I can show, but I want to know if Dash is a reasonable direction to be taking

Phil Ewels

@ewels

I'll do my best :+1:

Where communities thrive

People

Repo info

Activity