Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Tom Erik Støwer
    @testower
    :slight_smile:
    Joe M
    @jgmarce
    Ah, I had not... I'm doing development using new-york-city and that docker-compose.yml makes no attempt to set ES_JAVA_OPTS but now I understand where Xmx8g came from in my "planet" environment. It's all becoming clear now, Thanks Julian.
    Julian Simioni
    @orangejulius
    we're considering a bit of a reorganization of the docker repository that would combine all the docker-compose.yml files into one. that probably would have made it more clear that different settings are recommended for the planet. although that said, what we've discovered over the years is that heap size is not nearly as important these days with ES6+ than it was back in the early days
    I don't think 0.5GB is enough, but anything beyond that should be fine in general
    Joe M
    @jgmarce

    Elasticdump is useful if you need to change the documents... Back in 7/2019 I had to change my addendum style to conform to pelias's addendum feature.

    I hope "jewels of wisdom" from this elasticsearch transformation will make it to the "considerations for planet builds" document.

    Some additional questions I'm left with after reviewing that page... It isn't clear what can be accomplished given RAM restrictions. It is fine to know 32 hours given 64G but can the planet import succeed on a 16G RAM system given more time? If working with 16G will someone eventually succeed simply by taking a more granular steps to build?

    Is the PIP service always a single hit to RAM, because in the past I thought I had observed a PIP server per OA import thread, but I may be wrong on that...

    It all seems to work and that is amazing, but it would be good to know if any part of a planet import will fail on a 16G RAM host and have that in the "considerations" document.

    Julian Simioni
    @orangejulius
    each importer running will need about 8GB of RAM for the PIP service, yes. so if you want to run a fast planet build with several importers in parallel, you do want a lot of RAM. The OA importer can parallelize the work it does itself. Same concept but handled for you. Elasticsearch itself doesn't seem to need a ton of RAM, but it's pretty CPU heavy. so all in all you wind up with needing a lot of CPU and RAM :)
    Joe M
    @jgmarce
    Great... Can someone do the full planet with 16Gig of RAM or should I try and report back?
    Julian Simioni
    @orangejulius
    lets see. with 16GB of ram I'm guessing you're going to need to do it with no parallelization which means it will probably take at least 3 days. if you want to try that, go for it
    a good way to get a ballpark estimate is to see what import rate you're getting from an importer. We normally see 8000 records per second in our logs, even when running 3-4 importers in parallel. it might be worth making sure you're getting close to that number before you let a build run to completion
    Joe M
    @jgmarce
    This is even a question on AWS as there might be a spot instance with a very low price at 16G RAM but no deal to be found on a spot instance at 64-128G RAM. I'll see what I can do and give back (captured times) if I can.
    And it's great that I see where to reduce the elasticsearch "-Xmx" value as I'll have to steal back RAM where I can... 8g->4g I imagine.
    Julian Simioni
    @orangejulius
    another thing that takes some RAM is the pbf2json component of the OpenStreetMap importer
    Joe M
    @jgmarce
    I won't start just now, I'd be happy if someone in this group would say, I've already done it, don't waste the effort....
    jeff
    @jeff36476865_twitter
    How do I perform a crossstreet search in pelias docker?
    something like /v1/search?text=broadway+and+alvernon+Tucson"
    I have all the north american data loaded from the project
    cbayerlein
    @cbayerlein
    Hi! I'm trying to run my own instance of Pelias using Docker, covering only Baden-Württemberg (a federal state of Germany). Therefor, I cloned the directory germany in projects to a new one called germany-bawue and edited the pelias.json in here like this: https://gist.github.com/cbayerlein/9b7af1f8d7b26d93e1f1785c8d9484ea . Obviously everything went fine and Pelias is running. However, I have problems with streets. For example, when I query "korber str waibling" I get a bunch of results including addresseswith house numbers in this street but the actual "Korber Straße, Waiblingen, BW, Germany" is missing, although it exists in OSM.
    toton6868
    @toton6868
    Hello, I am trying to search for a keyword near to me. I am using something like this url/search?text= pizza hut&source=my_source&focus.point.lat=23.768510&focus.point.lon=90.354267. So I am finding for the pizza hut near to me but in the response, I am getting a huge list of those containing other pizza stores with confidence 1 along with pizza hut something like Pizza and Pizza, Pizza Lovers, Pizza Inn etc. Is there any way to search the exact keyword Pizza Hut.? I am finding the exact text match. Thanks in advance.
    bdmapman
    @bdmapman
    @jgmarce I have an idle VPS of 64 GB Ram though that is not AWS. If you want I can start the test. I saw my elasticsearch current heap size is 223.4 MB and Max heap is 1GB. I can increase it using ES_JAVA_OPTS="-Xms2g -Xmx2g" ./bin/elasticsearch but in that document, they strongly recommended not to change the heap size. Do I start the planet OSM import using the same heap size?
    Joe M
    @jgmarce
    With 64GB I'd increase elasticsearch to 4g others could share their opinion. The 20 days to build "interpolation" zapped me of some motivation to build and time my builds of everything, but if you contact me directly I could be convinced to continue. Together we could just post the summary on this channel rather than the play by play.
    bdmapman
    @bdmapman
    @jgmarce You can contact me directly also. What is the best way to contact you?
    bdmapman
    @bdmapman

    Hello, I am updating a single entry of my pelias index by gid which was inserted through csv-importer. The gid is something like custompoi:restaurant:45632. I am updating the layer type of this entry by

    curl -X POST "localhost:9200/pelias/_update/custompoi:restaurant:45632?refresh=true&pretty=true" -H 'Content-Type: application/json' -d'
    {
      "script" : "ctx._source.layer = \u0027shop\u0027"
    }
    '

    Unfortunately after updating successfully (curl is providing updated data in the terminal) this specific entry was not found in search endpoint. But can be found in reverse. Is there any specific reason behind this? I need to update data dynamically.

    Julian Simioni
    @orangejulius
    @bdmapman I think that behavior won't work out of the box unless you handle the phrase.* fields (making changes will erase them and break queries for that document in the search endpoint). You can read more about it here. It's been something we've wanted to fix for a while pelias/schema#285
    bdmapman
    @bdmapman
    @orangejulius ... Seems like I am going to fall into big trouble. I have already made a pipeline for live data updates. Is there any other way or shortcut of doing it? Besides I didn't understand by "unless you handle the phrase.*". If there is any way it will save me.
    kbeetz
    @kbeetz
    Hello, I am quite new to pelias and local installation of the server. My question ist: Where do I put the extra file with the optional variables? And if I don't have a API key, I remove it from the queryParams.json?
    Also I am not quite sure where to put the script-batch-search folder. For now I put in the docker -> project -> Germany where I downloaded the OSM data before. Is that correct? Thank you!
    Bimbol
    @Bimbol_gitlab
    Hello did Pelias support incremental updates? Or I have to import all the data at once, and replace index in elasticsearch?
    Sachin Sagoo
    @gohansagoo_twitter
    Hi Pelias Team!
    Looking for some help to understand the result of text=Newyork, USA with search API
    Result is confusing to me, here is the link to run the same search https://pelias.github.io/compare/#/v1/search?text=Newyork%2C+USA&debug=1
    image.png
    Sachin Sagoo
    @gohansagoo_twitter
    Attaching the result snap as well. The expected result should be Newyork from USA. Instead got different set of results.
    Patrick Wilson
    @patdevinwilson
    Hi, folks - I'm looking for some clarity on whether or not component address input is supported or not? This link shows it is a milestone, however I could not determine that in the roadmap on github. Thanks in advance! [https://pelias.io/milestones/component_geocoding/]
    Patrick Wilson
    @patdevinwilson
    bdmapman
    @bdmapman
    Hi, I don't know what I am missing but in the autocomplete endpoint if there are two keywords it is not returning expected results. I am trying to get the exact pizza hut keyword but it is not returning the expected result. I saw this (pelias/placeholder#88) issue but it is related to space at the end. I can't use search API for a few restrictions. Another interesting finding, if I use a hyphen in between keywords then it is returning near to expected result but for more than two keywords hyphen is not working
    Stefán Baxter
    @acmeguy
    hello all, I'm a newbie.
    As such I have two questions.
    1. is there a docker image available that has been preloaded with data (using Elsasti/Solr I assume)
    2. What is done in Pelios to merge/link/consolidate data from multiple sources?
    14 replies
    havlas.me
    @havlasme
    Hello, I'd like to use pelias autocomplete endpoint to separately autocomplete municipality (locality) name, street name, and housenumber. When I search for municipality name, everything works as expected as I start writing, I'm receiving list of possible options -> say I send text "Some" to autocomplete endpoint I receive all localities that start (or contain) string "Some" (eg. "Some City", "Some Other City", "Somecity", etc). However when I try to do the same with streets (set layers to street), when I send request with text "Some" I get empty response, only time I get some hit with streets is when I send request with full name, eg. I send text "Some Street" and I receive one result containing "Some Street", I would expect that I'll receive the "Some Street", even when I just send a parial string in request - say "Some". Is this a known issue, or am I maybe doing something wrong? I can provide access to our public instance of pelias, together with sample queries and expected results.
    phyroslam
    @phyroslam
    I am having an issue with my implementation of Pelias, if I search a street that has an accented character it works fine: /v1/search/structured?address=jesús gaona 33&postalcode=15500 but if the same query is done without the accent I don't get any results: /v1/search/structured?address=jesus gaona 33&postalcode=15500. Shouldn't it match the not accented street with the most equivalent one
    Sergey
    @romankovsv
    Hi guys, Could you please help me, I downloaded addresses dataset from openaddress.org but addresses don`t have zip codes. Do you know where I can downloaded addresses for US with zip codes.
    Would be really grateful for any suggestion