pelias/pelias

Pelias is a modular open-source geocoder using ElasticSearch for fast geocoding.

@louispersohn

Hey Everyone,

I'm trying to install geonames. I've done this dozens of times in the past but I can't get it to work both on an AWS instance and my a local vagrant image.

During the postinstall steps (npm run download_metadata) I can't get the country data to download / import. The error is below. I can download the AU.zip from geonames without issue. Any ideas ?

internal/streams/legacy.js:61
      throw er; // Unhandled stream error in pipe.
      ^

CsvError: Invalid Record Length: columns length is 19, got 1 on line 1
    at Parser.__onRecord (/home/vagrant/geonames/node_modules/csv-parse/lib/index.js:792:9)
    at Parser.__parse (/home/vagrant/geonames/node_modules/csv-parse/lib/index.js:668:38)
    at Parser._transform (/home/vagrant/geonames/node_modules/csv-parse/lib/index.js:474:22)
    at Parser.Transform._read (_stream_transform.js:191:10)
    at Parser.Transform._write (_stream_transform.js:179:12)
    at doWrite (_stream_writable.js:403:12)
    at writeOrBuffer (_stream_writable.js:387:5)
    at Parser.Writable.write (_stream_writable.js:318:11)
    at Request.ondata (internal/streams/legacy.js:19:31)
    at Request.emit (events.js:314:20) {
  code: 'CSV_RECORD_DONT_MATCH_COLUMNS_LENGTH',
  bytes: 36,
  comment_lines: 0,
  empty_lines: 0,
  invalid_field_length: 0,
  lines: 1,
  records: 0,
  columns: [
    { name: 'ISO' },
    { name: 'ISO3' },
    { name: 'ISO_Numeric' },
    { name: 'fips' },
    { name: 'Country' },
    { name: 'Capital' },
    { name: 'Area' },
    { name: 'Population' },
    { name: 'Continent' },
    { name: 'tld' },
    { name: 'CurrencyCode' },
    { name: 'CurrencyName' },
    { name: 'Phone' },
    { name: 'Postal_Code_Format' },
    { name: 'Postal_Code_Regex' },
    { name: 'Languages' },
    { name: 'geonameid' },
    { name: 'neighbours' },
    { name: 'EquivalentFipsCode' }
  ],
  error: undefined,
  header: false,
  index: 1,
  column: 'ISO3',
  quoting: false,
  record: [ '# ================================' ]
}
npm ERR! code 1
npm ERR! path /home/vagrant/geonames
npm ERR! command failed
npm ERR! command sh -c npm run download_metadata

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-02-25T05_29_49_336Z-debug-0.log

Andre-Pars

@Andre-Pars

hey guys, does anyone know how do we enable fuzzy search for autocomplete ?

Andre-Pars

@Andre-Pars

Hey guys, one more question, when looking at the openaddress csv file ca_on_amhertburg, i notice same address, different hash and slightly different lat, lon values. 6 months difference between the two downloads, none of the hash's match up between the download, does this make sense ? How does pelias account for lat lon values changing ? Data:
-83.0804109,42.0769702,3561,CREEK RD,,,,,,,9365b4842379489b
-83.0804060,42.0770327,3561,CREEK RD,,,,,,,a3e27072eb740fc8

Stefano Cudini

@stefanocudini

hi in reference to this 2016 thread I was wondering in which part of the github repository the mentioned update scripts are located: https://github.com/pelias/pelias/issues/412#issuecomment-244391254

Quốc Nhật

@tranquocnhat

I just test reverse with link https://pelias.github.io/compare/#/v1/reverse?point.lat=10.84856934922595&point.lon=106.79846048355103, response this point is in county: Quan 8, i return check this point is location in other county, particular is county: Quan 9; I hope you will help me find solution with expect response

Brad Hards

@bradh

I think this is because the data in whosonfirst says Quan 8.

See https://spelunker.whosonfirst.org/id/1108782041/

Quốc Nhật

@tranquocnhat

Brad Hards

@bradh

Is this the expected location: https://www.openstreetmap.org/way/315899289#map=18/10.84926/106.79927

Quốc Nhật

@tranquocnhat

you see this location in Quan 9 not Quan 8

adarshvarma15

@adarshvarma15

Whenever I run the whosonfirst service, I get this error when starts:

Brad Hards

@bradh

Its fairly self-explanatory.

You're asking for a latitude that is more than 90 degrees.

Brad Hards

@bradh

The convention is that bounding boxes are [min_longitude, min_latitude, max_longitude, max_latitude]. If you're familiar with projected maps, think easting, then northing.

1 reply

If you are familiar with Cartesian coordinates, X-then-Y.

stepan-chatalyan

@stepan-chatalyan

Hi everyone, I'm new to all this geo data :| I just start few days ago using, pelias and I'm not sure if I'm doing something wrong or maybe this data is not available. Well, I'm trying to explore if pelias have shape data about localities and neighborhoods (I'm not sure, but can be that are called "admin boundaries"). I found that WOF have this data hosted in: https://spelunker.whosonfirst.org/id/101748323/ but how can I import it in the pelias elastic?

3 replies

Brad Hards

@bradh

I'm not sure I really understand what you are asking for, but https://github.com/pelias/wof-admin-lookup might help.

Quốc Nhật

@tranquocnhat

Hi everyone, I'm just start pelias but add sources or layers param in search endpoint, then respose get error, i don't understand about it, can u help me?

1 reply

Narek

@narzero

Hi guys,

I got everything up and running very smoothly, thanks for the great work and documentation.

Question: I run Pelias for a small build on a single machine. I want to update the data once a week on the same machine. Am I correct to assume that I only need to run the following in order to update the data?

pelias download all && pelias prepare all && pelias import all

Łukasz Milunas

@Milunas

Hi all,
I have kubernetes cluster where I want to add pelias-api for autocomplete feature.
I found it the easiest to fetch elastic data with: https://github.com/pelias/docker and cp /data/nodes to elastic on cluster.
Unfortunately when I'm doing GET request https://elastic_host/_cat/indices I'm only receiving .geoip_databases
When I'm running api with docker-compose data is fetched properly
Maybe you have a similarly simple solution for my case that actually work? :)

Joe M

@jgmarce

Recently imported openaddresses from collection-global.zip. This produces .geojson files. All imported except Australia:
$ find au -name "*.geojson"
au/act/statewide-addresses-state.geojson
au/nsw/statewide-addresses-state.geojson
au/qld/brisbane_city_council-addresses-city.geojson
au/qld/city_of_gold_coast-addresses-city.geojson
au/qld/logan_city-addresses-county.geojson
au/qld/statewide-addresses-state.geojson
au/qld/sunshine_coast_council-addresses-city.geojson
au/qld/townsville_city_council-addresses-city.geojson
au/countrywide-addresses-country.geojson
au/tas/launceston_city_council-addresses-city.geojson
au/tas/statewide-addresses-state.geojson
au/vic/city_of_greater_geelong-addresses-city.geojson
au/vic/city_of_melbourne-addresses-city.geojson
au/vic/statewide-addresses-state.geojson

I've tested several (but not all) individually and they produce an error that looks like this:

error: [openaddresses] gnaf_mapper error
error: [openaddresses] TypeError: Cannot read property 'length' of undefined
    at DestroyableTransform._transform (/code/pelias/openaddresses/lib/streams/gnafMapperStream.js:20:18)
    at DestroyableTransform.Transform._read (/code/pelias/openaddresses/node_modules/through2/node_modules/readable-stream/lib/_stream_transform.js:177:10)
    at DestroyableTransform.Readable.read (/code/pelias/openaddresses/node_modules/through2/node_modules/readable-stream/lib/_stream_readable.js:456:10)
    at flow (/code/pelias/openaddresses/node_modules/through2/node_modules/readable-stream/lib/_stream_readable.js:939:34)
    at DestroyableTransform.pipeOnDrainFunctionResult (/code/pelias/openaddresses/node_modules/through2/node_modules/readable-stream/lib/_stream_readable.js:749:7)
    at DestroyableTransform.emit (events.js:314:20)
    at onwriteDrain (/code/pelias/openaddresses/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:479:12)
    at afterWrite (/code/pelias/openaddresses/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:467:18)
    at onwrite (/code/pelias/openaddresses/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:461:7)
    at WritableState.onwrite (/code/pelias/openaddresses/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:160:5)
error: [openaddresses] {
  "name": {
    "default": "118 Berrigan Road"
  },
  "phrase": {
    "default": "118 Berrigan Road"
  },
  "parent": {},
  "address_parts": {
    "number": "118",
    "street": "Berrigan Road",
    "zip": "7310"
  },
  "center_point": {
    "lon": 146.346872,
    "lat": -41.193704
  },
  "category": [],
  "addendum": {},
  "source": "openaddresses",
  "layer": "address",
  "source_id": "au/tas/statewide-addresses-state.geojson:9da0317fe91511d8"
}

Peter Johnson

@missinglink

thanks for the report, fixed in pelias/openaddresses#505
despite being noisy, they are only warnings which you can ignore, the import process will proceed uninhibited

collinjc

@collinjc:matrix.org

[m]

Does anyone know the process by which one can switch to their own token for OpenAddresses? This PR suggests it is as simple as setting imports.openaddresses.token
pelias/config#137
but doing so results in the following error:
Error: "imports.openaddresses.token" is not allowed
at getValidatedSchema (/code/pelias/openaddresses/node_modules/pelias-config/index.js:37:11)
at Object.generate (/code/pelias/openaddresses/node_modules/pelias-config/index.js:27:12)
at Object.<anonymous> (/code/pelias/openaddresses/utils/download_data.js:2:43)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
at internal/main/run_main_module.js:17:47

collinjc

@collinjc:matrix.org

[m]

Possibly related, it seems that the out-of-the-box configuration still points to the now outdated results.openaddresses.io

Juras Norkus

@jnorkus

Hi guys,
I'm solving an issue where WOF data is very limited for an area. Cities, neighbourhoods missing or having weird names that nobody knows about. The interesting thing is that OSM data has all of this information but is not used at all in Pelias importer.

This config just skips the lookups and leaves the data without admin properties.

"adminLookup": {
        "enabled": false
},

I also found this code that I assume was once used to extract admin data. Uncommenting it does not work, as I understand because model changed since then.

Any directions on solving this?

Keshav Nandan

@keshavnandan

Hi Team, I am trying to run the openstreetmap importer for full planet data but the importer gets stuck at pbf2json converter step for a really long time and ultimately the importer node gets OOMKilled terminated. The importer works fine for smaller datasets like portland. I kept increasing the compute and mem resources but looks like it needs much more than 8 GB RAM as mentioned in the documentation.

2022-04-04T21:48:28.232Z - info: [openstreetmap] Creating read stream for: /dl/openstreetmap/planet-220307.osm.pbf

Process finished with exit code 0

Container status

Containers:
  openstreetmap-import-container:
    Image:         pelias/openstreetmap:latest
    Port:          <none>
    Host Port:     <none>
    Command:
      ./bin/start
    State:          Terminated
      Reason:       OOMKilled
      Exit Code:    0
      Started:      Mon, 04 Apr 2022 17:48:27 -0400
      Finished:     Mon, 04 Apr 2022 19:48:10 -0400
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     3
      memory:  16Gi
    Requests:
      cpu:     1500m
      memory:  8Gi
    Environment:
      PELIAS_CONFIG:  /conf/pelias.json
    Mounts:
      /conf from pelias-config (rw)
      /dl from planet-volume (rw)

How much cpu and mem is needed to run the openstreetmap importer for full planet build?

Keshav Nandan

@keshavnandan

The importer is working after I bumped up the mem req to 16 Gi

Adrien Lagamelle

@adrienlagamelle

Anyone here knows where is elasticsearch installed using the docker installation steps. Im getting this:/usr/bin/which: no elasticsearch in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)

gelsas

@gelsas

I am getting this error shown: Error: EACCES: permission denied, mkdir '/data/geonames'

Brad Hards

@bradh

Probably missing a prefix somewhere. Hard to say without more context.

gelsas

@gelsas

Started over completly new now getting this:

pelias compose pull
Pulling libpostal ... done
Pulling schema ... done
Pulling api ... done
Pulling placeholder ... done
Pulling whosonfirst ... done
Pulling openstreetmap ... done
Pulling openaddresses ... done
Pulling geonames ... done
Pulling csv-importer ... done
Pulling transit ... done
Pulling polylines ... done
Pulling interpolation ... done
Pulling pip ... done
Pulling elasticsearch ... done
Pulling fuzzy-tester ... done
[ttap.io@app planet]$ pelias elastic start
Removing pelias_elasticsearch
Recreating abf1253d5782_pelias_elasticsearch ... error

ERROR: for abf1253d5782_pelias_elasticsearch Cannot start service elasticsearch: driver failed programming external connectivity on endpoint pelias_elasticsearch (6b4217b570655a656a53251fb77701a3b00d6e9714f8aad29de47fdd6d56daea): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 127.0.0.1 --dport 9300 -j DNAT --to-destination 172.18.0.7:9300 ! -i br-92febc9bc408: iptables: No chain/target/match by that name.
(exit status 1))

ERROR: for elasticsearch Cannot start service elasticsearch: driver failed programming external connectivity on endpoint pelias_elasticsearch (6b4217b570655a656a53251fb77701a3b00d6e9714f8aad29de47fdd6d56daea): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 127.0.0.1 --dport 9300 -j DNAT --to-destination 172.18.0.7:9300 ! -i br-92febc9bc408: iptables: No chain/target/match by that name.
(exit status 1))
ERROR: Encountered errors while bringing up the project.

and when running pelias download all this:

pelias download all
info: [openaddresses-download] Attempting to download all data
error: [openaddresses-download] error making directory /data/openaddresses message=EACCES: permission denied, mkdir '/data/openaddresses', stack=Error: EACCES: permission denied, mkdir '/data/openaddresses', errno=-13, code=EACCES, syscall=mkdir, path=/data/openaddresses
error: [openaddresses-download] Failed to download data message=EACCES: permission denied, mkdir '/data/openaddresses', stack=Error: EACCES: permission denied, mkdir '/data/openaddresses', errno=-13, code=EACCES, syscall=mkdir, path=/data/openaddresses
internal/fs/utils.js:269
throw err;
^

Error: EACCES: permission denied, mkdir '/data/geonames'
at Object.mkdirSync (fs.js:921:3)
at module.exports (/code/pelias/geonames/lib/tasks/download.js:13:6)
at Object.<anonymous> (/code/pelias/geonames/bin/downloadData.js:11:1)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
at internal/main/run_main_module.js:17:47 {
errno: -13,
syscall: 'mkdir',
code: 'EACCES',
path: '/data/geonames'
}
internal/fs/utils.js:269
throw err;
^

Error: EACCES: permission denied, mkdir '/data/whosonfirst/sqlite'
at Object.mkdirSync (fs.js:921:3)
at download (/code/pelias/whosonfirst/utils/download_sqlite_all.js:34:6)
at Object.<anonymous> (/code/pelias/whosonfirst/utils/download_sqlite_all.js:126:1)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
at internal/main/run_main_module.js:17:47 {
errno: -13,
syscall: 'mkdir',
code: 'EACCES',
path: '/data/whosonfirst/sqlite'
}
warn: [csv-download] No files to download, quitting
info: [openstreetmap-download] Downloading sources: https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf
error: [openstreetmap-download] error making directory /data/openstreetmap message=EACCES: permission denied, mkdir '/data/openstreetmap', stack=Error: EACCES: permission denied, mkdir '/data/openstreetmap', errno=-13, code=EACCES, syscall=mkdir, path=/data/openstreetmap
error: [openstreetmap-download] Failed to download data message=EACCES: permission denied, mkdir '/data/openstreetmap', stack=Error: EACCES: permission denied, mkdir '/data/openstreetmap', errno=-13, code=EACCES, syscall=mkdir, path=/data/openstreetmap
info: [interpolation(TIGER)] downloading all TIGER data
warn: [transit] 'pelias.json' config lacks a transit object entry. Transit importer quitting after taking no action
error: [interpolation(TIGER)] message=getaddrinfo EAI_AGAIN data.geocode.earth, stack=Error: getaddrinfo EAI_AGAIN data.geocode.earth
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26), errno=EAI_AGAIN, code=EAI_AGAIN, syscall=getaddrinfo, hostname=data.geocode.earth, response=undefined
error: [interpolation(TIGER)] message=getaddrinfo EAI_AGAIN data.geocode.earth, stack=Error: getaddrinfo EAI_AGAIN data.geocode.earth
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26), errno=EAI_AGAIN, code=EAI_AGAIN, syscall=getaddrinfo, hostname=data.geocode.earth, response=undefined

gelsas

@gelsas

I do not really understand why I am seeing this error

getting the same errors when using: sudo pelias download all

I would be willing to pay for help to set it up correctly

gelsas

@gelsas

I am trying to get stuff working for hours now

gelsas

@gelsas

Got it

Lucas Hänke de Cansino

@l4b4r4b4b4

Hey everyone,
For some reason I can't seem to make reverse geoconding work. I setup the docker-compose stack behind traefik 2, moved the Germany Dataset germany-valhalla.polylines.0sv to ./data/polylines, edited the files .env and pelias.json accordingly and ran the installation commands, excluding pelias prepare polylines.Everything went through and I can send queries to search and autocomplete endpoints succesfully getting a proper answer with GeoFeatures in the FeatureCollection Array. When querrying the reverse endpoint I still get an answer but the FeatureCollection Array is empty.
My research brought me to the point, that I might have to adjust ./stream/pipeline.js but honstly I couldn't find any further information, where to find it and how to edit it, to process the provided Dataset.
Suggestions, hints and help on how to get Reverse API working would be highly appreciated :)
Cheers Luke

With pelias.json as follows, when importing it still takes the generated extract.0sv instead of the provided germany-valhalla.polylines.0sv:

"imports": {
    "polyline": {
      "datapath": "/data/",
      "files": [ "germany-valhalla.polylines.0sv" ]
    },

The following on the other hand does seem to take the provided dataset, but reverse endpoint still doesnt give back contentfull answers:

"imports": {
    "polyline": {
      "datapath": "/data/polylines",
      "files": [ "germany-valhalla.polylines.0sv" ]
    },

Sorry other way around.

Lucas Hänke de Cansino

@l4b4r4b4b4

hmmm taking everything back, i guess. It seems to work now xD

Keshav Nandan

@keshavnandan

Whats the download path for getting the interpolation data for full planet? I am able to download the data for portland using path "https://s3.amazonaws.com/pelias-data.nextzen.org/portland-metro/interpolation" but am not sure whats the path for the planet, I tried planet and planet-latest but its not working

Where communities thrive

People

Repo info

Activity