Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Julian Simioni
    @orangejulius
    hi @jpdantur the only case I've heard of where indices are suddenly disappearing is if your Elasticsearch instance is open to the internet and you were hit by the "Meow" attack: https://arstechnica.com/information-technology/2020/07/more-than-1000-databases-have-been-nuked-by-mystery-meow-attack/
    Juan Dantur
    @jpdantur

    Hi @orangejulius thanks for the help. I will add security to the server so that not everyone can have access. For the time being I deleted the index and re-created it, and ran import again. After that I got the following error

    {
      "error" : {
        "root_cause" : [
          {
            "type" : "illegal_argument_exception",
            "reason" : "Fielddata is disabled on text fields by default. Set fielddata=true on [source] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
          }
        ],
        "type" : "search_phase_execution_exception",
        "reason" : "all shards failed",
        "phase" : "query",
        "grouped" : true,
        "failed_shards" : [
          {
            "shard" : 0,
            "index" : "pelias",
            "node" : "gFoLazB0SaSeC5KTRqOCpA",
            "reason" : {
              "type" : "illegal_argument_exception",
              "reason" : "Fielddata is disabled on text fields by default. Set fielddata=true on [source] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
            }
          }
        ],
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "Fielddata is disabled on text fields by default. Set fielddata=true on [source] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.",
          "caused_by" : {
            "type" : "illegal_argument_exception",
            "reason" : "Fielddata is disabled on text fields by default. Set fielddata=true on [source] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
          }
        }
      },
      "status" : 400
    }

    Any ideas what might have caused it? I imported my own csvs alongside all OpenAdresses, OpenStreetMap, etc. data

    Peter Huffer
    @peterhuffer

    Hello. I am trying to install the Pelias portland-metro example on CentOS. I am seeing the following error when running pelias elastic start:

    Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data"

    I have seen multiple issues about running as root may cause this. So I created a separate user but am seeing the same issues. I made sure to chown the DATA_DIR to the pelias user. Am I missing something? It may be worth noting that I have successfully ran this demo locally in Windows WSL shell.

    1 reply
    Juan Dantur
    @jpdantur
    Hi. Is there a way of adding authentication to ElasticSearch (port 9200) in a Pelias Docker instance? Thanks :)
    petertoner
    @petertoner
    This is a good place to start with security and user authentication: https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-security.html
    Pravin kalbhor
    @pravink

    Hi, I have a working setup of pelias and can query the pelias for addresses. which responds correctly. for all. layers except the postalcode..
    working request example -

    https://mydomain.com/pelias/api/v1/search?text=2429&layers=address&boundary.country=AUS

    but when i request with layer postalcode like below -

    https://mydomain.com/pelias/api/v1/search?text=2429&layers=postalcode&boundary.country=AUS

    then it responds with below error -

     "errors": [
                "'postalcode' is an invalid layers parameter. Valid options: coarse,address,venue,street,locality,neighbourhood,county,localadmin,region,macrocounty,country,macroregion,borough,macrohood,marinearea,disputed,dependency,empire,continent,ocean"
            ]

    I did imported the postalcodes as well when i have downloaded the wof data for postalcodes..
    does anyone have any idea on why i am facing this issue?

    Ian Axelrod
    @ian-axelrod
    Hi folks. I want to cache results from geocode.earth, since Google's Geocoding API does not permit caching (beyond lat/lon) of any sort. Is there anything in your terms that would disallow this? I cannot find anything, other than potential requirements for attribution when data is taken from one of the many data sources. I work for a company that uses geocoding and autocomplete in support of our product, but it is not a significant feature of the product.
    Julian Simioni
    @orangejulius
    hi @ian-axelrod. Best place for geocode.earth questions is hello@geocode.earth, but our terms do not prohibit caching. benefits of open data :)
    Ian Axelrod
    @ian-axelrod
    Thanks! I figured that was the case. I realize geocode.earth is more appropriate for this next question as well, but I figure I'd ask here anyway just in case anyone else happens to be searching for a similar answer. I'll shoot the question to hello@geocode.earth as well. You would not happen to know if attribution is necessary if I use pelias/geocode.earth as part a system that powers our autocomplete for job search (my employer is themuse.com -- we are looking to move away from Google's geocoder)? Most TOS for major providers imply that this is a requirement, but from what I can tell no website for any of the major providers (mapbox, tomtom, here) actually does this. Rather, they only display attribution for maps.
    So either every site I've looked at is violating TOS... which could very well be ... or I am misinterpreting the TOS.
    It does seem quite awkward to display attribution for autocomplete functionality. I am not really sure where you would display this in a way that is both prominent but aesthetically acceptable. With maps, you can attribute inside the map itself in one of the corners.
    Ian Axelrod
    @ian-axelrod
    Sent. Thanks again!
    Sen Han
    @senyan
    Hi Guys, we are a bit new to the pelias service. We deployed the pelias services to kubernetes cluster. But seems the index are not being automatically created and data are not being ingested.
    Is there any job or command I should run in order to get it working properly in kubernetes?
    Johan
    @joacub
    Hi guys, thanks for the work
    there is any script what you have or way for keep updating the docker with all databases (openstreetmap,etc)
    Juan Dantur
    @jpdantur

    Hi. I'm deploying a pelias Docker build on a server. The thing is that ElasticSearch is running on another server. I modified pelias.json to include the correct URL in esclient.hosts and created the pelias index on that server successfully using pelias elastic create. Yet when I run pelias import all I get the following error. Note that the index still exists when I hit {elasticsearch_url}/_cat/indices

    ERROR: Elasticsearch index pelias does not exist
    You must use the pelias-schema tool (https://github.com/pelias/schema/) to create the index first
    For full instructions on setting up Pelias, see http://pelias.io/install.html
    /code/pelias/whosonfirst/node_modules/pelias-dbclient/src/configValidation.js:39
            throw new Error(`elasticsearch index ${config.schema.indexName} does not exist`);
            ^
    
    Error: elasticsearch index pelias does not exist
        at existsCallback (/code/pelias/whosonfirst/node_modules/pelias-dbclient/src/configValidation.js:39:15)
        at respond (/code/pelias/whosonfirst/node_modules/elasticsearch/src/lib/transport.js:368:9)
        at /code/pelias/whosonfirst/node_modules/elasticsearch/src/lib/transport.js:396:7
        at Timeout.<anonymous> (/code/pelias/whosonfirst/node_modules/elasticsearch/src/lib/transport.js:429:7)
        at listOnTimeout (internal/timers.js:531:17)
        at processTimers (internal/timers.js:475:7)

    Any reason why this could be happening? Maybe I'm missing an important step? Thanks a lot :)

    cyfugr
    @cyfugr
    Hi, i was wondering why this query "coffee shops in london" won't return coffee shops in london.
    Brad Hards
    @bradh
    @cyfugr Where are you querying, what data do you have loaded, what are you expecting (number of records), and what do you get instead?
    toton6868
    @toton6868
    repost: Is there any option/feature/API in Pelias which will search bulk places? Such as I am finding for 10-20 places in JSON format. I want to get every first search result of each place keywords. Such as I have a json like this "places": ["New York", "San Francisco", "Dallas", "Detroit", "Chikago", "Washington", "White House"]. Which will return all the first search result without calling API multiple times.
    markpiercey
    @markpiercey
    Anyone know if it's possible to disable libpostal logging? I tried adding the following to my entrypoint:
    -gracehttp.log false but i'm getting flag provided but not defined: -gracehttp.log
    image.png
    wof-libpostal-server github says that ☝️
    Oliver S
    @oli1605
    Hi folks, I am currently setting up Pelias for the whole planet and noticed some poor coverage at some countries in the world like China, India or Africa. Source: https://pelias.github.io/scripts-geocoding-coverage
    Do you have any suggestions on how I can get results or at least some normalized data to get some data for those cases.
    I am trying to map existing address data (street, city, country, coordinates more or less existing) and came across that.
    I thought about something like a parsed libpostal + normalized data response.
    My goal is to check equality of existing address data.
    Julian Simioni
    @orangejulius

    @oli1605 unfortunately we haven't updated that page in a couple years. its on our todo list since some details about data coverage could be great to share.

    the good news is open data is growing rapidly, and a lot of people are working hard to help it grow, seeking out new data all the time. but you are definitely right there are some areas like China where there is a real lack. if you find anything, we all want to know :)

    Ringo Leese
    @ringostarr80
    Hi, I'm setting up pelias on my own server and I want to access the API via nginx.
    I want to access it by typing https://someserver.com/pelias/v1/search?text=New York. is this possible?
    If I got to https://someserver.com/pelias, the server always redirects me to https://someserver.com/v1
    Martin Minnoni
    @mminnoni_twitter
    hey folks! I wonder what should I change to get more than 10 results from a query in pelias? I have 23 records imported into elastic for a particular address but I am only getting 10 results....
    Martin Minnoni
    @mminnoni_twitter
    hey I solved it adding size at the end of the query...
    one more question..when doing a query on pelias and I use 2 hyphens instead of 1 hyphen I get the exact match...I think this might be a matter of the tokenizer in pelias and ES....how can I modified that in the docker for pelias?
    rwrx
    @rwrx
    Hello, I have run all planet build of Pelias in Docker container. I have run this script: https://github.com/pelias/docker#quickstart-build-script only replacing "cd projects/portland-metro" by "cd projects/planet". After build completed everything works properly and elastic index is there. But after approximately 3 days, elastic search index is removed, I dont know what is happening I have only remained Pelias running in background. I mean it just suddenly replace content of docker/projects/planet/data/elasticsearch directory with a new indices and indices created by Pelias Docker script are not there. Is this standard behavior and I must do something else to retain indices created by Pelias Docker script? Or am I doing something wrong?
    rwrx
    @rwrx
    Thank you, but it is not a "Meow" attack
    Sunitha Selvan
    @isunitha98selvan
    Hi. I am trying to do the docker build for pelias with elasticsearch v7.10. However it throws an error "Invalid apiVersion "7.10", expected a function or one of _default, 7.6, 7.5, 7.4, 7.3, 7.2, 7.1, 7.0, 6.8, 5.6, 7.7, 7.x". Is there no support for es version>7.7 ?
    Julian Simioni
    @orangejulius
    hi @isunitha98selvan you should be fine if you keep the Elasticsearch version in pelias.json unchanged from whatever we currently have it set to. The changes between minor versions of Elasticsearch are, well...minor :)
    Sunitha Selvan
    @isunitha98selvan
    @orangejulius I was looking to integrate it with an existing Elasticsearch instance(v7.7.1) that we have in production, will that be possible?
    albertors90
    @albertors90
    I am experiencing the same issue as @rwrx with smaller regions (not the full planet). I followed the steps described in https://medium.com/@mohammedayub44/yet-another-awesome-geo-coder-open-sourced-spatial-search-c295970a8226 and I get it up and running. However, after a few days the elasticsearch index is removed and I have to import everything again. Any clues of what I may be missing?
    Anis Oudjehih
    @aoudjehih
    Hello, I would like to know what is the tags-filter used to generate the osmium.extract.geojsonl.gz file used in openstreetmap import (spatial). Can someone provide us the command line plz? @missinglink
    JamieGruener
    @JamieGruener
    We're setting up our environment to use api.geocode.earth. Are the IPs returned for that FQDN static?
    rwrx
    @rwrx
    @albertors90 I still did not solved this problem with elasticsearch index being removed. Do you solved it?
    Julian Simioni
    @orangejulius
    @rwrx @albertors90 indices disappearing suddenly is very concerning. I can't think of anything that would cause it, but Elasticsearch can definitely do weird things. The Elasticsearch logs are confusing and verbose, but there might be something in there that gives hints as to what happened
    Rowan Winsemius
    @rowanwins
    Hey Pelias Team, I was wondering if there was any interest in kicking off the work on getting Pelias to support better units
    pelias/pelias#861 ?
    There appears to be a bunch of PRs that are reasonably close but have never been merged for various reasons. If there is anything I can contribute let me know :)
    akshay-skills19
    @akshay-skills19
    hello I have one doubt, our company already configured OSM with overpass API, but now I want to do reverse geocoding on that OSM database so can I use peliase with existing OSM DB?
    Brad Hards
    @bradh
    @akshay-skills19 You need the pelias ES approach. You won't (without a significant amount of code) be able use overpass.
    albertors90
    @albertors90
    @rwrx Still stuck with it, and can't find any useful information on ES logs
    Ondřej Machulda
    @OndraM
    Hi there! We have set up pelias.json imports to use data from OSM, OA, WOF for our country (Czech republic). We run prepare & import and start the containers, and everything seems OK. But when we run the same search request for an eg. street name using Pelias Compare Tool against official api.geocode.earth vs. our service, we are getting slightly different results (our service return less and also worse results - on the same request). It seems it returns less polylines points than official API. Any idea what could be causing this?
    robertoamoreno
    @robertoamoreno
    Anybody have an idea why when using the reverse api its only returning 1 result instead of multiple
    5 replies
    robertoamoreno
    @robertoamoreno
    overlap
    Enrico Hofmann
    @gigo1980
    hi together, does pelias also give me the "district" name inside a city ?
    Enrico Hofmann
    @gigo1980
    And how can I get the ZIP code