We got some feedback by email, I think by eurostat data people but Im not sure: "Why you not used directly the shapefiles and relied now on RDATA on your github account ? Was ESTAT infrastructure not efficient enough? Do you need another format ? We also provide GeoJSON/TopoJSON and PBF format at http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/. "
Starting to think about this.. I do not remember why we ended up hosting our own copy in RData format - do you remember - @muuankarski
tai @jlehtoma
it even seems that they might be ready to add other output formats if we ask though not sure
Muuten tämmöstäkin ilmottelivat (korjaan saman tien mutta fyi): Small typo on page 388 for the journal article,
Instead of Merge Eurostat data with geodata from Cisco
should be Merge Eurostat data with geodata from Gisco
Plus in source code get_eurostat_geospatial.R
Instead of @title Download Geospatial Data from GISCO
It should be @title Download Geospatial Data from GISCO
Instead of Merge Eurostat data with geodata from Cisco
should be Merge Eurostat data with geodata from Gisco
Plus in source code get_eurostat_geospatial.R
Instead of @title Download Geospatial Data from GISCO
It should be @title Download Geospatial Data from GISCO
(unless these are now considered equivalent)
will check later today
As for shapefile vs GeoJSON, again the only reason for using the previous is probably size (GeoJSON is basically uncompressed text file). More generally, shapefile should be avoided, but unfortunately it is still very hard.
So to reply to GISCO people, I would say that we are moving away from shapefiles and GeoJSON (which they already provide) is a good option otherwise but the larger size and unmatured (R) tools form a sort of bottleneck which we (and others) are working on. And therefore we have preprocessed and compressed the data in RData files. We could provide the code and propose that they share RData files as well in which case we can switch to that. Would you agree with such reply ?
In case we do some pre-processing, then using RData seems reasonable. If not, I wouldn't bother. Also I think it makes fairly little sense for them to provide RData files, unless they want to facilitate R access without relying on specific packages (such as
rgdal/sf) for reading.
Personally, I think it makes sense to provide the data in commonly accessible spatial data formats (e.g. GeoJSON) since reading the data is a fairly small overhead for us (unless we want to get rid of the spatial deps)
just a quick look here:
http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/ref-nuts-2013.html makes it clear that we would need to rethink few things with the package.
The main thing being that the geofile we have used contains all the different NUTS-levels and with an inner_join you have been able to subset the geodata to the same levels as your Eurostat attribute data is. Here each level is separated into its own file which will require user to spesify the NUTS level explicitly. Certainly is would be more clear to have them separate, but I kind of like the current behaviour when a single geodata always matches your primary Eurostat data.
With
get_eurostat you dont need to specify the NUTS level, and the levels available varies between datasets. However, experienced user should be aware of this and able to download the right geodata, I think.
I will try reading the files properly tomorrow
Right, we can do that by downloading all the levels, row_binding and merging I suppose.
So, this works fine at NUTS2-level:
library(eurostat)
# 1. Lataa data
sp_data <- get_eurostat("tgs00026", time_format = "raw", stringsAsFactors = FALSE) %>%
# filtteroi vuoteen 2014 ja tasolle NUTS-2 (merkkien määrä == 4) eli vaikka FI02
dplyr::filter(time == 2014, nchar(as.character(geo)) == 4)
# 2. Lataa geodata NUTS3-tasolla (RAAKAA KOODIA)
library(sf)
library(dplyr)
jsontemp <- tempfile()
download.file("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_2.geojson",
jsontemp)
nuts2 <- sf::st_read(jsontemp, stringsAsFactors = FALSE)
# 3. yhdistä
map <- left_join(nuts2,sp_data, by = c("NUTS_ID" = "geo"))
# 4. piirrä kartta
library(tmap)
tm_shape(map) +
tm_polygons("values",
title = "Disposable household\nincomes in 2010",
palette = "Oranges")
I am reading in a
geojson file, not the topojson. File size in topojson is marginally smaller, but it contains either epsg (SRID) nor proj4string field when read with st_read()
are you thinking we should switch from our own RData files into this ?
I ran into an error with eurostat_geodata so did not check yet how long the processing will take and so how necessary the ready-made RData files are. Regarding file size, we could ask GISCO to share compressed geojson files if that would help with transfer speed
To keep the current behaviour we can just download and
rbindall the levels as in this examplelibrary(eurostat)
# 1. Lataa data
sp_data <- get_eurostat("ilc_li01", time_format = "raw", stringsAsFactors = FALSE) %>%
# filtteroi vuoteen 2014 ja tasolle NUTS-2 (merkkien määrä == 4) eli vaikka FI02
dplyr::filter(time == 2016, hhtyp == "A1", currency == "EUR", indic_il == "LI_C_M40")
# 2. Lataa geodata KAIKILLA NUTS-tasolla
library(sf)
library(dplyr)
# NUTS0
jsontemp <- tempfile()
download.file("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_0.geojson",
jsontemp)
nuts0 <- sf::st_read(jsontemp, stringsAsFactors = FALSE)
# NUTS1
jsontemp <- tempfile()
download.file("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_1.geojson",
jsontemp)
nuts1 <- sf::st_read(jsontemp, stringsAsFactors = FALSE)
# NUTS2
jsontemp <- tempfile()
download.file("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_2.geojson",
jsontemp)
nuts2 <- sf::st_read(jsontemp, stringsAsFactors = FALSE)
# NUTS0
jsontemp <- tempfile()
download.file("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_3.geojson",
jsontemp)
nuts3 <- sf::st_read(jsontemp, stringsAsFactors = FALSE)
nuts <- rbind(nuts0,nuts1,nuts2,nuts3)
# 3. yhdistä
map <- inner_join(nuts,sp_data, by = c("NUTS_ID" = "geo"))
# 4. piirrä kartta
library(tmap)
tm_shape(map) +
tm_polygons("values",
title = "Poverty thresholds",
palette = "Oranges")
Yes, I think we should. Good thing about this is also that data comes from the same domain
http://ec.europa.euas with Eurostat-package and requires no new domain to be whitelisted by IT..
Ok, I will reply to the GISCO guys, I think this is clear. Once they share compressed files, we can (and perhaps should if we ask..) switch to use those.
@muuankarski
In fact downloading these geojson files is already fast now. Do we really need compressed versions ?
It also comes to mind that we could have readily processed versions in the R package data/ folder to avoid downloads entirely..
Ok so I can tell them that this was due to historical reasons & we are just planning switch when the time allows. I may also mention that we still consider at some point having copies of the most common files in the R package in order to avoid download.
No personal experience on the CRSs / TopoJSON tho