hey folks, wondering where I can find https://www.geomesa.org/img/taxi.gif (map manager?) once i run geo mesa server so that one can see a UI
3 replies
Hi, I was able to successfully install geomesa PySpark, I want to perform a spatial join now, any reference links for the same(I see one in Scala in the official documentation), also does the pyspark variant support all other functionality, which is supported by Scala?
6 replies
sorry, ones again this is how the page with new feature of 3.0 look like for me
Also, can you post the URL you are trying to visit?
Dear community, hopefully again I can receive some suggestions here...
I'm working for on connecting my local spark to geomesa accumulo.
I have results that are in spark dataframes (e.g. coordinates and a count -like in a spatial density profile), and what I want is write the results to geomesa/accumulo.
At this point I am interested in writing any dataframe (like the ones on which you can do sparksql queries) to geomesa, just to see if the connection works.
Any suggestion of code anywhere to do a very simple test to see if this is working? Most documentation talks more about rdds.
I would use the example here https://www.geomesa.org/documentation/2.2.2/user/spark/core.html#example but could not load the ECQL class (I believe when I run the geomesa-accumulo-spark-runtime jar there is a conflict with one of the other jars but can't figure out with which one.) and also this one looks like it is an rdd not a dataframe.
I'm working for on connecting my local spark to geomesa accumulo.
I have results that are in spark dataframes (e.g. coordinates and a count -like in a spatial density profile), and what I want is write the results to geomesa/accumulo.
At this point I am interested in writing any dataframe (like the ones on which you can do sparksql queries) to geomesa, just to see if the connection works.
Any suggestion of code anywhere to do a very simple test to see if this is working? Most documentation talks more about rdds.
I would use the example here https://www.geomesa.org/documentation/2.2.2/user/spark/core.html#example but could not load the ECQL class (I believe when I run the geomesa-accumulo-spark-runtime jar there is a conflict with one of the other jars but can't figure out with which one.) and also this one looks like it is an rdd not a dataframe.
5 replies
sorry for questions around upsert, but that is really important for us.
So far we've done always appending writes, that is why we came to the point where the main ID index is ok - we always have unique FID values enforced by database storage. But for "secondary" attribute indexes we have cases when old values coexist with new ones (referencing the same
So far we've done always appending writes, that is why we came to the point where the main ID index is ok - we always have unique FID values enforced by database storage. But for "secondary" attribute indexes we have cases when old values coexist with new ones (referencing the same
fid). Is it possible to identify somehow the old/stale values vs the latest ones in those indexes and filter out them during reads or maybe run kind of compaction process during which to clean them out?
Generally, it maybe preferable to switch to deleting and inserting when writing to GeoMesa. (Given the context of Cassandra/other key-value stores, an upsert would effectively have to delete the old entries and write the new ones)
geomesa-3.0.0+hbase-2.0.5;after I appending a new attr in an existing FeatureType,when I use geomesa query that table with the FeatureType, but nothing out ,there is logs
com.ruqi.appserver.ruqi.geomesa.db.GeoDbHandler - Running query dtg DURING 2019-12-31T00:00:00+00:00/2021-01-02T00:00:00+00:00；typeName:recommendPoint
hsqldb.db.HSQLDB6BD103EBD8.ENGINE - dataFileCache open start
org.locationtech.geomesa.index.planning.StrategyDecider$ - Using strategy provider 'org.locationtech.geomesa.index.planning.StrategyDecider$CostBasedStrategyDecider'
org.locationtech.geomesa.utils.audit.AuditLogger$ - {"storeType":"hbase","typeName":"recommendPoint","date":1596620766679,"user":"unknown","filter":"dtg DURING 2019-12-31T00:00:00+00:00/2021-01-02T00:00:00+00:00","hints":"RETURN_SFT\u003drpId:String,addressName:String,title:String,dtg:Date,updateTime:Integer,channel:Integer,*rGeom:Point:srid\u003d4326:precision\u003d6,ext:String,shotCount:Integer","planTime":141,"scanTime":0,"hits":0,"deleted":false}
com.ruqi.appserver.ruqi.geomesa.db.GeoDbHandler - Running query INCLUDE；typeName:recommendPoint
com.ruqi.appserver.ruqi.geomesa.db.GeoDbHandler - Returned 0 total features but when I check it in Hbase shell ,this the data below,so can someone help me out?
First, when you appended the new attribute, you added it to the end of the SFT definition, correct?
(Doing otherwise, would likely cause issues)
Second, how are you querying for the data?
If you aren't using the command line tools, I'd recommend them as a quick way to debug things (sometimes using GeoServer in the mix can add complexity, etc)
As a third suggestion, have you looked at the logs on the region servers? (I'm wondering if there are any errors or exceptions there about failing to read data, etc)
In fact, it uses zookeeper to get a distributed lock on the necessary metadata to avoid Really, Quite Bad Things
if the process is hung, you could jstack it, and you might see something about that distributed lock
if so, then there may be an issue with Zookeeper, etc
(or writes to the HBase metadata tables)
@yuqiyich hmm.... interesting.
first ,thanks for replays，After receiving your reply, I checked carefully ,before add,the SFT is the "rpId:String,addressName:String,title:String,dtg:Date,updateTime:Integer,channel:Integer,rGeom:Point:srid=4326:precision=6;geomesa.stats.enable='true',geomesa.index.dtg='dtg',geomesa.indices='z3:6:3:rGeom:dtg,z2:5:3:rGeom,id:4:3:,attr:8:3:rpId:rGeom:dtg,attr:8:3:updateTime:rGeom:dtg'",after add two 'ext:String' And 'shotCount:Integer' attribute end of SFT , the SFT change to
the"rpId:String,addressName:String,title:String,dtg:Date,updateTime:Integer,channel:Integer,rGeom:Point:srid=4326:precision=6,ext:String,shotCount:Integer;geomesa.index.dtg='dtg',geomesa.indices='attr:8:3:rpId:rGeom:dtg,attr:8:3:updateTime:rGeom:dtg'",and querying cql is "dtg DURING 2019-12-31T00:00:00.000Z/2021-01-02T00:00:00.000Z",i run it in geomesa commandtools shell "geomesa-hbase explain -c t_rpt_dev_7777 -f recommendPoint -q "dtg DURING 2019-12-31T00:00:00.000Z/2021-01-02T00:00:00.000Z"" and get result that 'No filter plans found' ,but run hbase shell that data really exists,and next i run geomesa-hbase command 'geomesa-hbase explain -c t_rpt_dev_7777 -f recommendPoint -q "rpId = '122.984662_23.986662'"',I can get result ,so what's the problem ?
@yuqiyich hmm.... interesting.
I finally find out what's the problem,because I use java Api to update schema ,Use Java class DataStore#updateSchema(String typeName, SimpleFeatureType newfeatureType), and param 'newfeatureType' that contain old attr( keep old,not change) and append the new attr end of SFT ,there is schema changes shotcut that contains change before and change after above,from shotcut you will found that new attr is added ,but old attr index has disappeared, so I had one questions now, first ,how could I useDataStore#updateSchema(String typeName, SimpleFeatureType newfeatureType)Java Api update schema,what should the input param newfeatureType should like?,is there any update schema by java Api demos? ,And I slove this problem by using geomesa-hbase command tools to update schema（add a new attr into SFT，and doc is there link） 。if there have some update schema by java Api demo ，u can send to yichitgo@gmail.com mailbox, thank you very much,best wish for u.
your choices are 1) may sure to query on the things you have indexed (e.g. rpId and/or updateTime) or add indexes which will support date range queries on 'dtg'. The previous example which had the z3 index on rGeom and dtg is an example. Alternatively, you could add an index on just dtg (if you expect to support queries like the one you indicated)
All paths work, the params are loaded right, how should i compile the library to make it works ?
4 replies
Hi, I'm having trouble running geomesa with google bigtable. I downloaded version 3.0.0 on github and when I run the following error: "Caused by: java.io.IOException: java.lang.NoSuchMethodException: com.google.cloud.bigtable.hbase1_x.BigtableConnection. <init> (org .apache.hadoop.conf.Configuration, java.util.concurrent.ExecutorService, org.apache.hadoop.hbase.security.User) ". I believe it is something related to the dependecies, but I couldn't make it work, can someone help me?
That said, Google Bigtable changes some details about their API here and there and this causes issues
what version of Google Cloud Bigtable are you using?
and what exactly were you trying to do when you hit that error?
it may be that there's a Cloud Bigtable jar which you need to download. Or it may be the case that there's a version mismatch somewhere
@jnh5y thanks for your reply. You mean the bigtable jars that i'm using? To fix this error I was trying to update the jar files in geomesa/lib...tried to get the latest version of 'bigtable-hbase.....jar', 'geomesa-bigtable.....jar' and the hadoop and hbase jars
@elahrvivaz
You proposed this repo: https://github.com/ccri/cloud-local
-> Thank you! Couple questions:
Some qs:
1)it seems like the repo is capable of working with geoserver, but that it is not included. Is this correct?
2)my final goal is to use hbase (as the system we want to run it has hbase). So far I was spending some time on accumulo, as most examples are for this. What is the best link for some examples with basic hbase stuff (doing some geocalculation, and writing results to hbase on geomesa and then rendering on geoserver)
You proposed this repo: https://github.com/ccri/cloud-local
-> Thank you! Couple questions:
Some qs:
1)it seems like the repo is capable of working with geoserver, but that it is not included. Is this correct?
2)my final goal is to use hbase (as the system we want to run it has hbase). So far I was spending some time on accumulo, as most examples are for this. What is the best link for some examples with basic hbase stuff (doing some geocalculation, and writing results to hbase on geomesa and then rendering on geoserver)
ALso related to https://github.com/ccri/cloud-local, when I run the bin/cloud-local.sh init script, it looks promising, until I am asked to login to my own machine (which serves as the hbase master) and provide the password when he complains about JAVA_HOME not set:
Starting hbase...
starting master, logging to /home/user/Documents/cloud-local/hbase-1.3.1/logs/hbase-user-master-ds-gpu11.out
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
jb@ds-gpu11's password:
ds-gpu11: +======================================================================+
ds-gpu11: | Error: JAVA_HOME is not set |
ds-gpu11: +----------------------------------------------------------------------+
ds-gpu11: | Please download the latest Sun JDK from the Sun Java web site |
ds-gpu11: | > http://www.oracle.com/technetwork/java/javase/downloads |
ds-gpu11: | |
ds-gpu11: | HBase requires Java 1.7 or later. |
ds-gpu11: +======================================================================+The erros comes from /cloud-local/hbase-1.3.1/bin/hbase-config.sh where there is a check if JAVA_HOME is defined.
I believe that in cloud-local/hbase-1.3.1/conf/hbase-env.sh folder JAVA_HOME can be set , but this file is overwritten every time you run the cloud-local.sh init command.
However on my machine, proper version of java is installed and JAVA_HOME is not empty:
(base) jb@ds-gpu11:~/Documents/cloud-local$ java --version
openjdk 11.0.8 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1, mixed mode, sharing)
(base) jb@ds-gpu11:~/Documents/cloud-local$ echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64I'm having an error when I try to ingest json file on google bigtable, tried on version 2.4.1 and 3.0.0. Both with the same problem, can anyone help me?
ERROR [org.locationtech.geomesa.tools.ingest.LocalConverterIngest] Failed to write '=1970-01-18T11:00:26.000Z|POINT (-77.7724 5.99455)|test'
java.lang.IllegalArgumentException: Error indexing feature ':Sun Jan 18 11:00:26 UTC 1970|POINT (-77.7724 5.99455)|test'Caused by: java.lang.IllegalArgumentException: Row length is 0STFS:
"light" = {
attributes = [
{ name = "datetime", type = "Date", index = true }
{ name = "geom", type = "Point", index = true, srid = 4326 }
{ name = "source", type = "String", index = false }
]
}CONVERTER:
"light" = {
type = "json"
options {
error-mode = "raise-errors"
}
fields = [
{ name = "datetime", json-type = "string", path = "$.datetime.$date", transform = "date('yyyy-MM-dd\\'T\\'HH:mm:ss.SSS\\'Z\\'', $0)" }
{ name = "lat", json-type = "double", path = "$.lat" }
{ name = "lon", json-type = "double", path = "$.lon" }
{ name = "source", json-type = "string", path = "$.source"}
{ name = "geom", transform = "point($lon, $lat)" }
]
}FILE:
{"datetime":{"$date":"1970-01-18T11:00:26.000Z"},"lon":-77.7724,"lat":5.99455,"source":"test"}