Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 31 2019 19:27
    rufuspollock commented #241
  • Dec 14 2018 15:27
    StephenAbbott opened #246
  • Dec 03 2018 09:12
    rufuspollock commented #245
  • Nov 26 2018 14:51
    StephenAbbott opened #245
  • Nov 08 2018 08:31
    zelima commented #243
  • Nov 08 2018 08:05
    zelima closed #244
  • Nov 08 2018 08:05
    zelima commented #244
  • Nov 08 2018 07:57
    zaneselvans commented #244
  • Nov 07 2018 07:22
    zelima commented #244
  • Nov 07 2018 07:16
    akariv commented #244
  • Nov 07 2018 07:10
    akariv commented #234
  • Nov 06 2018 16:56
    parrottsquawk commented #234
  • Nov 01 2018 13:25
    zelima commented #244
  • Nov 01 2018 13:25
    zelima commented #244
  • Nov 01 2018 13:23
    zelima commented #244
  • Nov 01 2018 08:29
    anuveyatsu commented #244
  • Nov 01 2018 08:29
    anuveyatsu commented #244
  • Oct 24 2018 19:03
    zaneselvans opened #244
  • Oct 23 2018 09:40
    geraldb starred datahq/datahub-qa
  • Oct 19 2018 08:22

    Branko-Dj on master

    [travis][s]: Added update comma… (compare)

Tito Jankowski
@titojankowski
@rufuspollock hope you’re as excited as I am about this! DataHub is a terrific home for daily carbon dioxide levels
Rufus Pollock
@rufuspollock
Very excited :smile:
@titojankowski what is the source for daily co2 data? if you have that source we can build a dataflow to auto import it each day.
Tito Jankowski
@titojankowski
@rufuspollock terrific!
Here’s the daily record where new data is pulled from, latest data point is 12/19/2018, 409.54 ppm: https://www.esrl.noaa.gov/gmd/webdata/ccgg/trends/co2_mlo_weekly.csv
However, that’s only the most recent 2 years of data or so
The remaining daily data from 1958 - 2017 is here: ftp://aftp.cmdl.noaa.gov/data/trace_gases/co2/in-situ/surface/mlo/co2_mlo_surface-insitu_1_ccgg_DailyData.txt
Rufus Pollock
@rufuspollock

OK, and we want this in one single dataset right?

ATM we have this dataset http://datahub.io/core/co2-ppm with code for this here https://github.com/datasets/co2-ppm

But this is only annual and monthly from 1980

I suggest we create a new dataset e.g. co2-ppm-daily and create the scraper for this. Do you know python at all? If not we can help do this as we already have the srapers for this kind of file in co2-ppm ...

Tito Jankowski
@titojankowski
@rufuspollock yes, one single dataset of daily data from 1958 - present
how can I help without knowing any python?
Svetozar Stojkovic
@svetozarstojkovic
@rufuspollock there should be no problem scraping these two sources into one single dataset
Tito Jankowski
@titojankowski
@svetozarstojkovic awesome!
@svetozarstojkovic how can I help you make this happen?
@rufuspollock @svetozarstojkovic Once the data is on datahub, how does the carbondoomsday frontend access it?
Svetozar Stojkovic
@svetozarstojkovic
@titojankowski so far, everything seems straightforward, we will inform you if we need some help.
Tito Jankowski
@titojankowski
@svetozarstojkovic terrific, thank you!
@svetozarstojkovic will the data be accessible via a public api?
Svetozar Stojkovic
@svetozarstojkovic
@titojankowski all the dataset files can be represented in .csv and .json format, which can be accessed by simple GET request.
Ross Hill
@rosslh

quick question regarding the sharable data links like https://datahub.io/core/co2-ppm/r/1.json

That link redirects to to https://pkgstore.datahub.io/core/co2-ppm/co2-annmean-mlo_json/data/5168771a128447a2d4c8a77e40844134/co2-annmean-mlo_json.json

Can I expect the second url to remain static? That is, can I request data from it in my web service and expect that it will keep working?

Also, some of the climate datasets only have data up to 2014. Will they at some point be updated?
Tito Jankowski
@titojankowski
@svetozarstojkovic how can I query the dataset files, ie if I want just datapoints between 1990 - 2000 for example?
reading about data transforms
Tito Jankowski
@titojankowski
@rosslh I think we’re both asking similar questions haha
Anuar Ustayev
@anuveyatsu

@rufuspollock @svetozarstojkovic Once the data is on datahub, how does the carbondoomsday frontend access it?

@titojankowski using r links you can as Svetozar said access via GET request. Re querying dataset files, there is some workarounds, e.g., we can organize resources in a way you can access data by year or decade. @rufuspollock any suggestions?

Also, please, remember that you can just embed graphs and tables from datahub

Tito Jankowski
@titojankowski
thanks @anuveyatsu!
Anuar Ustayev
@anuveyatsu
@rosslh Hi! The sharable data links like https://datahub.io/core/co2-ppm/r/1.json are static. We call them r links. These links redirect you to the latest available version of a resource. The second URL is static, however, it won’t get data update. Once data is updated, its hash is also changed and there will be a new URL for it. I would request the data using the first link so that you always have the latest data.

Also, some of the climate datasets only have data up to 2014. Will they at some point be updated?

Yes, it will be :smile:

Tito Jankowski
@titojankowski
ah, good to know as well!
Anuar Ustayev
@anuveyatsu
@rosslh @titojankowski I’d also recommend reading this page about that - https://datahub.io/docs/getting-started/getting-data :smile:
Rufus Pollock
@rufuspollock

Also, some of the climate datasets only have data up to 2014. Will they at some point be updated?

That may be a bug in the scraping code - which datasets are out of date?

Tito Jankowski
@titojankowski
thanks @anuveyatsu !
Oleg Lavrovsky
@loleg
Merry Christmas everyone. The Climate Dashboards and everything else going on around here is amazing!
Oleg Lavrovsky
@loleg
Just uploaded https://datahub.io/loleg/ttn-twist2018-dataset ~ hoping to see more TTN-sourced data here in the future
Rufus Pollock
@rufuspollock
@loleg :clap: :clap:
Anuar Ustayev
@anuveyatsu
Happy New Year everybody! :fireworks: 2019
@titojankowski hi, we’ve setup co2-ppm-daily here - https://datahub.io/core/co2-ppm-daily - please check it out :smile:
Tito Jankowski
@titojankowski
@anuveyatsu great work! I spot checked the datapoints and they all matched exactly. Happy New year team!
@anuveyatsu how do we pull the latest data? the most recent datapoint is 12/23/2018, MLO has data up to 01/01/2019
@rufuspollock this looks terrific, so glad you came to visit the carbon doomsday gitter!
Anuar Ustayev
@anuveyatsu
@titojankowski ok, I see something went wrong in Travis. I’ll take a look tomorrow :smile:
Tito Jankowski
@titojankowski
@anuveyatsu If helpful: with carbon doomsday we checked every hour for a new data point, MLO is inconsistent in terms of when they update.
Nicholas Pylypiw
@NickPyll
hello, there seems to have been an issue with the premier league data, specifically the current season. only ten games are there
Anuar Ustayev
@anuveyatsu
@NickPyll Hi, thank you for reporting this. We will have a look. cc/ @svetozarstojkovic
Nicholas Pylypiw
@NickPyll
Thank you. I know it was working as recent as Tuesday, but it looks as though the updates yesterday may have deleted records.
Nicholas Pylypiw
@NickPyll
@svetozarstojkovic and @anuveyatsu, there is a csv which appears to be correct, but when I read straight from the json data package it is not.
Svetozar Stojkovic
@svetozarstojkovic
@NickPyll Hello, we think we found a solution, can you please check again, to see if everything is in order? :)
Nicholas Pylypiw
@NickPyll
@svetozarstojkovic Yes it looks like the issue has been fixed. Thank you!
Svetozar Stojkovic
@svetozarstojkovic
@NickPyll awesome :+1:
Avishek2020
@Avishek2020
hi
I need RDF dataset