Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 31 2019 19:27
    rufuspollock commented #241
  • Dec 14 2018 15:27
    StephenAbbott opened #246
  • Dec 03 2018 09:12
    rufuspollock commented #245
  • Nov 26 2018 14:51
    StephenAbbott opened #245
  • Nov 08 2018 08:31
    zelima commented #243
  • Nov 08 2018 08:05
    zelima closed #244
  • Nov 08 2018 08:05
    zelima commented #244
  • Nov 08 2018 07:57
    zaneselvans commented #244
  • Nov 07 2018 07:22
    zelima commented #244
  • Nov 07 2018 07:16
    akariv commented #244
  • Nov 07 2018 07:10
    akariv commented #234
  • Nov 06 2018 16:56
    parrottsquawk commented #234
  • Nov 01 2018 13:25
    zelima commented #244
  • Nov 01 2018 13:25
    zelima commented #244
  • Nov 01 2018 13:23
    zelima commented #244
  • Nov 01 2018 08:29
    anuveyatsu commented #244
  • Nov 01 2018 08:29
    anuveyatsu commented #244
  • Oct 24 2018 19:03
    zaneselvans opened #244
  • Oct 23 2018 09:40
    geraldb starred datahq/datahub-qa
  • Oct 19 2018 08:22

    Branko-Dj on master

    [travis][s]: Added update comma… (compare)

Tito Jankowski
@titojankowski
hey @rufuspollock we would love to get Earth’s carbon dioxide data hosted on datahub! What’s the next step?
Andrew Cantino
@cantino
@titojankowski Looks like it's already here? https://datahub.io/collections/climate-change
Tito Jankowski
@titojankowski
that would be great, i’ll check it out!
looking through right now, only see monthly data points at the moment
a great foundation to add in daily data
Anuar Ustayev
@anuveyatsu
@titojankowski are you looking at https://datahub.io/core/co2-ppm ?
Tito Jankowski
@titojankowski
@anuveyatsu yes, that’s it
and the most recent datapoint is September 1st
would LOVE it if this was daily data as it became available, MLO posts a new datapoint every day
Anuar Ustayev
@anuveyatsu
Yes, there is only monthly data.. and yes, daily data would be cool :smile: @rufuspollock any suggestions?
Tito Jankowski
@titojankowski
what part of datahub do you work on @anuveyatsu?
Anuar Ustayev
@anuveyatsu
@titojankowski both frontend (mostly) and backend
Tito Jankowski
@titojankowski
what a great project it is!
Anuar Ustayev
@anuveyatsu
@titojankowski :smile: several months ago we’ve tried to implement something like dashboards but it never was published. Probably, we could use daily data in that dashboard as well.

:earth_asia: :bar_chart: Climate Change Dashboards

[feedback is needed]

Today we’re introducing Climate Change Dashboard on DataHub where you can explore State of the World and Impacts of Global Climate Change. We'd appreciate if you could give us any feedback:

https://datahub.io/awesome/dashboards/climate-change

The goal was to combine everything related to climate change on datahub. It would great to improve it and publish/promote. E.g., we could add another tab called “CO2 today” where we have a chart showing the latest data.
We’ve also received feedback that the dashboard’s design/layout can be improved.
Tito Jankowski
@titojankowski
checking it out now, @anuveyatsu
looks great, amazing that co2 emissions are still climbing
Rufus Pollock
@rufuspollock
@titojankowski so we’d love to collaborate on migrating the climate doomsday stuff over and integrating with what’s there. A suggesiton would be:
  • All data on DataHub as standard datasets
  • Migrating existing dashboard over (or integrating / reworking existing ones)
Tito Jankowski
@titojankowski
terrific!
how will you get the daily CO2 data onto DataHub?
@rufuspollock hope you’re as excited as I am about this! DataHub is a terrific home for daily carbon dioxide levels
Rufus Pollock
@rufuspollock
Very excited :smile:
@titojankowski what is the source for daily co2 data? if you have that source we can build a dataflow to auto import it each day.
Tito Jankowski
@titojankowski
@rufuspollock terrific!
Here’s the daily record where new data is pulled from, latest data point is 12/19/2018, 409.54 ppm: https://www.esrl.noaa.gov/gmd/webdata/ccgg/trends/co2_mlo_weekly.csv
However, that’s only the most recent 2 years of data or so
The remaining daily data from 1958 - 2017 is here: ftp://aftp.cmdl.noaa.gov/data/trace_gases/co2/in-situ/surface/mlo/co2_mlo_surface-insitu_1_ccgg_DailyData.txt
Rufus Pollock
@rufuspollock

OK, and we want this in one single dataset right?

ATM we have this dataset http://datahub.io/core/co2-ppm with code for this here https://github.com/datasets/co2-ppm

But this is only annual and monthly from 1980

I suggest we create a new dataset e.g. co2-ppm-daily and create the scraper for this. Do you know python at all? If not we can help do this as we already have the srapers for this kind of file in co2-ppm ...

Tito Jankowski
@titojankowski
@rufuspollock yes, one single dataset of daily data from 1958 - present
how can I help without knowing any python?
Svetozar Stojkovic
@svetozarstojkovic
@rufuspollock there should be no problem scraping these two sources into one single dataset
Tito Jankowski
@titojankowski
@svetozarstojkovic awesome!
@svetozarstojkovic how can I help you make this happen?
@rufuspollock @svetozarstojkovic Once the data is on datahub, how does the carbondoomsday frontend access it?
Svetozar Stojkovic
@svetozarstojkovic
@titojankowski so far, everything seems straightforward, we will inform you if we need some help.
Tito Jankowski
@titojankowski
@svetozarstojkovic terrific, thank you!
@svetozarstojkovic will the data be accessible via a public api?
Svetozar Stojkovic
@svetozarstojkovic
@titojankowski all the dataset files can be represented in .csv and .json format, which can be accessed by simple GET request.
Ross Hill
@rosslh

quick question regarding the sharable data links like https://datahub.io/core/co2-ppm/r/1.json

That link redirects to to https://pkgstore.datahub.io/core/co2-ppm/co2-annmean-mlo_json/data/5168771a128447a2d4c8a77e40844134/co2-annmean-mlo_json.json

Can I expect the second url to remain static? That is, can I request data from it in my web service and expect that it will keep working?

Also, some of the climate datasets only have data up to 2014. Will they at some point be updated?
Tito Jankowski
@titojankowski
@svetozarstojkovic how can I query the dataset files, ie if I want just datapoints between 1990 - 2000 for example?
reading about data transforms
Tito Jankowski
@titojankowski
@rosslh I think we’re both asking similar questions haha
Anuar Ustayev
@anuveyatsu

@rufuspollock @svetozarstojkovic Once the data is on datahub, how does the carbondoomsday frontend access it?

@titojankowski using r links you can as Svetozar said access via GET request. Re querying dataset files, there is some workarounds, e.g., we can organize resources in a way you can access data by year or decade. @rufuspollock any suggestions?

Also, please, remember that you can just embed graphs and tables from datahub

Tito Jankowski
@titojankowski
thanks @anuveyatsu!
Anuar Ustayev
@anuveyatsu
@rosslh Hi! The sharable data links like https://datahub.io/core/co2-ppm/r/1.json are static. We call them r links. These links redirect you to the latest available version of a resource. The second URL is static, however, it won’t get data update. Once data is updated, its hash is also changed and there will be a new URL for it. I would request the data using the first link so that you always have the latest data.

Also, some of the climate datasets only have data up to 2014. Will they at some point be updated?

Yes, it will be :smile:

Tito Jankowski
@titojankowski
ah, good to know as well!
Anuar Ustayev
@anuveyatsu
@rosslh @titojankowski I’d also recommend reading this page about that - https://datahub.io/docs/getting-started/getting-data :smile: