Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Aaron Spring
    @aaronspring

    @aaronspring You think these are ok?

     hind_2000_2019 = hind_2000_2019.sel(latitude=slice(90, -60))
     hind_2000_2019 = hind_2000_2019.notnull() 
     obs_2000_2019_p2 = obs_2000_2019_p2.sel(latitude=slice(90, -60))
     obs_2000_2019_p2 = obs_2000_2019_p2.notnull() 
     obs_2020_p2 = obs_2020_p2.sel(latitude=slice(90, -60))
     obs_2020_p2  = obs_2020_p2.notnull()

    The first about selecting the apporpriate latitude and the second about selecting notnull values ( I am not sure if it does what is supposed)

    again. i cannot answer “is this OK questions?” I cannot give methodological instructions

    George Gousios
    @ggousios_gitlab
    Ok, thanks! I didn't know that If you drop the NaN grid points, you have to recreate them for the final submission file. So, using fillna(0.) is safer.
    Regarding the skill_by_year ? The new version gives an error as I wrote above (and I am not getting the RPSS column results)
    Aaron Spring
    @aaronspring

    Regarding the skill_by_year ? The new version gives an error as I wrote above (and I am not getting the RPSS column results)

    please check whether your inputs comply with the assert function

    11 replies
    Aaron Spring
    @aaronspring
    new conservatively regridded observations uploaded to EWC https://github.com/ecmwf-lab/climetlab-s2s-ai-challenge/releases/tag/0.7.6, so please download new date, but currently having a climetlab caching issue: ecmwf/climetlab#21 which @floriankrb is fixing
    soon afterwards I will exchange the biweekly ground truth observations with this new data: https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/merge_requests/18, and update website, template and renku datasets
    George Gousios
    @ggousios_gitlab
    @aaronspring , when you save the submission preds_test.to_netcdf('../submissions/ML_prediction_2020.nc'), shouldn't be preds_test.to_netcdf('../submissions/ML_prediction_2020.nc', mode='a')? Append, in order to overwrite the two variables, tp and t2m? So, I run my model first for tp, I save the file and then run the model for t2m and save again.
    Aaron Spring
    @aaronspring

    @aaronspring , when you save the submission preds_test.to_netcdf('../submissions/ML_prediction_2020.nc'), shouldn't be preds_test.to_netcdf('../submissions/ML_prediction_2020.nc', mode='a')? Append, in order to overwrite the two variables, tp and t2m? So, I run my model first for tp, I save the file and then run the model for t2m and save again.

    If you first do one var, save, and then do the second var and save again, mode=‘a’ sounds useful: http://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_netcdf.html

    George Gousios
    @ggousios_gitlab
    This file obs_2000_2019_p = xr.open_dataset(f'../data/hindcast-like-observations_2000-2019_biweekly_terciled.zarr', engine='zarr') has many null values!
    print(obs_2000_2019_p.isnull().sum())
    Data variables:
        t2m         int64 143640600
        tp          int64 129716364
        weekofyear  int64 0
    2 replies
    Aaron Spring
    @aaronspring

    ⚠️ Ground truth data changed⚠️

    Please update all observations related files (forecasts untouched). Now regridded conservatively and grouped by week (not ISOweek). see https://s2s-ai-challenge.github.io/#announcements and https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/CHANGELOG.md

    4 replies
    Björn Mayer
    @BjoernMayer92
    image.png
    I have a question to the raw daily data: I thought they are total precipiation so accumulated and should be monotonically increasing with increasing lead time. However for some points in the ecmwf forecast this does not seem to be true : This is ECMWF forecast in the training set :
    Aaron Spring
    @aaronspring
    this indeed looks odd. could you please raise an issue in https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge/-/issues? Do you have an idea @floriankrb ?
    Björn Mayer
    @BjoernMayer92
    thanks I did raise an issue
    Aaron Spring
    @aaronspring
    Aaron Spring
    @aaronspring

    Hi all,
    I see that no submissions entered https://renkulab.io/gitlab/tasko.olevski/s2s-ai-competition-scoring-image/-/tree/master so far.

    No disadvantages:

    • you wont give away any details by submitting results in the mean time
    • You can overwrite your results anytime by using the same tag again

    Advantage:

    • with an early submission you can check whether your results are accepted by the scorer

    Note you cannot submit anything Nov 1st. But it is still exactly two months until then.

    Looking forward to seeing your scores and later the implementation,
    Aaron

    George Gousios
    @ggousios_gitlab
    Hi, any ideas about using more than one features?Besides t2m or tp. So, have a multivariate series? With a small modification to the Data Generator , we can return only the X data and not the y , but how to combine them?
    8 replies
    Aaron Spring
    @aaronspring
    I think I won’t change the template as I don’t want to push towards a certain implementation
    1 reply
    Aaron Spring
    @aaronspring
    It is mentioned in the template that multiple feature variables are recommended. Also in the town halls. The template is just a very simple example and therefore with just one feature
    1 reply
    Aaron Spring
    @aaronspring

    The organizers decided to host a town hall in two weeks for technical questions and implementation challenges to foster collaboratio, identification of pain points and starting a discussion.

    Please indicate your interest in https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge/-/issues/13#note_28438

    David Landry
    @landryda_gitlab
    @aaronspring Hi Aaron, I just noticed that the test-input files for ECMWF have 51 members instead of 11. Did I do something wrong or is that expected?
    If so, do you know if there's a way to recover the first 11 members among those? So that we can apply the training model trivially on the test data
    Aaron Spring
    @aaronspring
    You are correct. There are indeed 51 2020 members from the real-time ecmwf forecast but only 11 hindcast on-the-fly forecasts. The decision to produce these members was taken by ecmwf.
    There is no connecting between the members of the hindcast or forecasts. If you just want 11 forecast members, just take any 11 of them. But I personally think it’s probably better to adapt your scripts to use all 51 members to get a more granular forecast distribution
    David Landry
    @landryda_gitlab
    Thanks
    Aaron Spring
    @aaronspring
    the technical town hall will happen next thursday at 15-16 CEST, see https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge/-/issues/13#note_28590
    jordangierschendorf
    @jordangierschendorf
    Hi Aaron,
    I’m trying to score a netcdf file by using the git lfs methods, but I have a job error when my file is pushed. Can you help me to understand why it failed?
    Here is the error:
    Fatal error from pip prevented installation. Full pip output in file:
    44 /home/jovyan/.local/pipx/logs/cmd_2021-09-14_14.16.18_pip_errors.log
    zulucomputer
    @zulucomputer
    Hi @aaronspring how do you process the original ECWMF raw data to become the climetlab-s2s-ai-challenge competition data saved on EWC? I'm particulary interested in how you regrid the raw ECMWF data to 1.5 x 1.5 degree resolution and daily observations? Is this processing script available?
    Aaron Spring
    @aaronspring
    tools/observations/build_dataset_observations.py
    tools/observations/build_dataset_observations.py
    1 reply
    Aaron Spring
    @aaronspring
    1 reply
    Wulf Luck
    @WulfLuck
    image.png
    image.png
    image.png
    Apparently I should generate a key in order to fork a template. ssh-keygen -t ed25519 -C "info@wulf-luck.de" Any hint about my mistake? Cheers Wulf
    Aaron Spring
    @aaronspring
    image.png
    Hi Wulf, I fork by clicking on “Fork” in the browser
    technical town hall starts in 30 minutes
    Aaron Spring
    @aaronspring
    for those you wondered about the dry mask and how it enters the scoring: https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge/-/issues/43
    BeeKay Koozie
    @GaelKBertrand
    Hi Aaron, checking the climetlab pluggin to input data (https://bit.ly/3tUp3Iy), how does exactly 'date' differs from 'coordinates' . Is it necessary to use both when importing data?
    2 replies
    1. Do we have to import all variables or just what was described as 'necessary only' (t2m, tp, pr)?
    2 replies
    Antonio Pérez
    @aperezvelasco
    Good morning to everybody! I'm trying to work with the project locally and I'm getting an authentication error when using the "renku clone" command... The username and password are the same as those that we are using to log in to renkulab.io, right?
    Aaron Spring
    @aaronspring
    Yes. For renku questions please provide question and the code you put in the command line to https://gitter.im/SwissDataScienceCenter/renku
    Antonio Pérez
    @aperezvelasco
    Done! Thanks!
    David Landry
    @landryda_gitlab
    @aaronspring i'm sorry if this was covered before but... For NCEP data, do we have hindcasts past 2010?
    image.png
    When I download NCEP data using climetlab, the files contain data up to 2010
    Aaron Spring
    @aaronspring
    nope. NCEP is only available until 2010. not our decision here. hindcast dates were produced by them, see https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge/-/issues/17. here is a bit more info about the hindcasts: https://doi.org/10.1175/BAMS-D-16-0017.1
    David Landry
    @landryda_gitlab
    Thanks