just a quick look here:
http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/ref-nuts-2013.html makes it clear that we would need to rethink few things with the package.
The main thing being that the geofile we have used contains all the different NUTS-levels and with an inner_join you have been able to subset the geodata to the same levels as your Eurostat attribute data is. Here each level is separated into its own file which will require user to spesify the NUTS level explicitly. Certainly is would be more clear to have them separate, but I kind of like the current behaviour when a single geodata always matches your primary Eurostat data.
With
get_eurostat you dont need to specify the NUTS level, and the levels available varies between datasets. However, experienced user should be aware of this and able to download the right geodata, I think.
I will try reading the files properly tomorrow
Right, we can do that by downloading all the levels, row_binding and merging I suppose.
So, this works fine at NUTS2-level:
library(eurostat)
# 1. Lataa data
sp_data <- get_eurostat("tgs00026", time_format = "raw", stringsAsFactors = FALSE) %>%
# filtteroi vuoteen 2014 ja tasolle NUTS-2 (merkkien määrä == 4) eli vaikka FI02
dplyr::filter(time == 2014, nchar(as.character(geo)) == 4)
# 2. Lataa geodata NUTS3-tasolla (RAAKAA KOODIA)
library(sf)
library(dplyr)
jsontemp <- tempfile()
download.file("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_2.geojson",
jsontemp)
nuts2 <- sf::st_read(jsontemp, stringsAsFactors = FALSE)
# 3. yhdistä
map <- left_join(nuts2,sp_data, by = c("NUTS_ID" = "geo"))
# 4. piirrä kartta
library(tmap)
tm_shape(map) +
tm_polygons("values",
title = "Disposable household\nincomes in 2010",
palette = "Oranges")
I am reading in a
geojson file, not the topojson. File size in topojson is marginally smaller, but it contains either epsg (SRID) nor proj4string field when read with st_read()
are you thinking we should switch from our own RData files into this ?
I ran into an error with eurostat_geodata so did not check yet how long the processing will take and so how necessary the ready-made RData files are. Regarding file size, we could ask GISCO to share compressed geojson files if that would help with transfer speed
To keep the current behaviour we can just download and
rbindall the levels as in this examplelibrary(eurostat)
# 1. Lataa data
sp_data <- get_eurostat("ilc_li01", time_format = "raw", stringsAsFactors = FALSE) %>%
# filtteroi vuoteen 2014 ja tasolle NUTS-2 (merkkien määrä == 4) eli vaikka FI02
dplyr::filter(time == 2016, hhtyp == "A1", currency == "EUR", indic_il == "LI_C_M40")
# 2. Lataa geodata KAIKILLA NUTS-tasolla
library(sf)
library(dplyr)
# NUTS0
jsontemp <- tempfile()
download.file("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_0.geojson",
jsontemp)
nuts0 <- sf::st_read(jsontemp, stringsAsFactors = FALSE)
# NUTS1
jsontemp <- tempfile()
download.file("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_1.geojson",
jsontemp)
nuts1 <- sf::st_read(jsontemp, stringsAsFactors = FALSE)
# NUTS2
jsontemp <- tempfile()
download.file("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_2.geojson",
jsontemp)
nuts2 <- sf::st_read(jsontemp, stringsAsFactors = FALSE)
# NUTS0
jsontemp <- tempfile()
download.file("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_3.geojson",
jsontemp)
nuts3 <- sf::st_read(jsontemp, stringsAsFactors = FALSE)
nuts <- rbind(nuts0,nuts1,nuts2,nuts3)
# 3. yhdistä
map <- inner_join(nuts,sp_data, by = c("NUTS_ID" = "geo"))
# 4. piirrä kartta
library(tmap)
tm_shape(map) +
tm_polygons("values",
title = "Poverty thresholds",
palette = "Oranges")
Yes, I think we should. Good thing about this is also that data comes from the same domain
http://ec.europa.euas with Eurostat-package and requires no new domain to be whitelisted by IT..
Ok, I will reply to the GISCO guys, I think this is clear. Once they share compressed files, we can (and perhaps should if we ask..) switch to use those.
@muuankarski
In fact downloading these geojson files is already fast now. Do we really need compressed versions ?
It also comes to mind that we could have readily processed versions in the R package data/ folder to avoid downloads entirely..
Ok so I can tell them that this was due to historical reasons & we are just planning switch when the time allows. I may also mention that we still consider at some point having copies of the most common files in the R package in order to avoid download.
No personal experience on the CRSs / TopoJSON tho
But: everything seems to be in order, so carry on :)
One issue still prevails, as in current implementation of get_eurostat_geospatial user can opt for SpatialPolygonDataFrame , fortified data.frame or sf output. We could provide those conversions "on-the-fly" if we will rely on the json-files from eurostat (now they come preprocessed using download.file()). If we would provide all three it would require following steps on-the-fly.
# =======================================================
# If user passes output_class = "sf" OR does not spesify it (default behaviour)
## Download and return a sf-object
# =======================================================
library(sf)
library(dplyr)
jsontemp <- tempfile()
download.file("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_0.geojson",
jsontemp)
shape <- sf::st_read(jsontemp, stringsAsFactors = FALSE)
return(shape)
# =======================================================
# If user passes output_class = "sp" this is done in addition to default behaviour
## Convert sf-object into sp-object SpatialPolygonDataFrame
# =======================================================
shape_sp <- as(shape, "Spatial")
return(shape_sp)
# =======================================================
# If user passes output_class = "data.frame" this is done in addition to steps above
## Convert SpatialPolygonDataFrame into "fortified" regular data.frame to be plotted with ggplot2::geom_polygon
# =======================================================
shape_sp$id <- row.names(shape_sp)
fortified <- broom::tidy(shape_sp)
fortified <- left_join(fortified,shape_sp@data)
return(fortified)@jlehtoma what do you think, is that feasible to do on-the-fly OR should we provide just a sf-output and nothing else..?
Yep, that is the current behavior (in sf-branch), but providing the other options would require adding broom-dependency at least.
I could try with preserving the exact same behavior as currently, but change the source and processing. A new attribute would be nuts_level where user could pass either 0,1,2,3 or all. all would be default allowing the current behavior of subsetting with inner_join only.
Sound very good to me. I think we can import one more package but it is true that we are starting to have quite many imported packages. Could investigate at some point if these can be reduced.. or splitting data retrieval and geovisualization components in separate packages is also an option (though perhaps a bit complicated one)
now the basic idea is implemented. At least, cache needs to be revised. Can test with:
library(eurostat)
library(dplyr)
library(sf)
# sf
shape_sf <- get_eurostat_geospatial(nuts_level = "0", output_class = "sf")
shape_sf %>% select(NUTS_ID) %>% plot()
# data.frame
shape_df <- get_eurostat_geospatial(nuts_level = "0", output_class = "df")
shape_df %>% ggplot2::ggplot(.) + ggplot2::geom_polygon(aes(x=long,y=lat,group=group,fill=NUTS_ID))
# spdf
shape_spdf <- get_eurostat_geospatial(nuts_level = "0", output_class = "spdf")
sp::spplot(obj = shape_spdf, "NUTS_ID")
i saw wrong
Perhaps there is a better method for fetching this json data http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_60M_2013_4258_LEVL_3.geojson than
download.file()...
instead of
download.file() I implemented it using httr::GETnow, the two options are listed below:resolution <- "60"
# option 1
resp <- httr::GET(paste0("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_",resolution,"M_2013_4258_LEVL_1.geojson"))
nuts1 <- sf::st_read(httr::content(resp, as="text"), stringsAsFactors = FALSE)
# option 2
jsontemp <- tempfile()
download.file(paste0("http://ec.europa.eu/eurostat/cache/GISCO/distribution/v1/geojson/nuts-2013/NUTS_RG_",resolution,"M_2013_4258_LEVL_1.geojson"), jsontemp)
nuts1 <- sf::st_read(jsontemp, stringsAsFactors = FALSE)