by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
andrijz
@andrijz
And if you don't mind to answer me a quick question if something has been changed in 3.0 regarding the data upserts on accumullo and cassandra. Has anything changed there?
1 reply
Karthik Ram
@listaction
hey folks, wondering where I can find https://www.geomesa.org/img/taxi.gif (map manager?) once i run geo mesa server so that one can see a UI
3 replies
Wechattt
@Wechattt
@jnh5y Solved, the problem of the data itself, thanks bro
Rahul Bhatia
@rbhatia46
Hi, I was able to successfully install geomesa PySpark, I want to perform a spatial join now, any reference links for the same(I see one in Scala in the official documentation), also does the pyspark variant support all other functionality, which is supported by Scala?
6 replies
andrijz
@andrijz
image.png
sorry, ones again this is how the page with new feature of 3.0 look like for me
James Hughes
@jnh5y
@andrijz sorry for the trouble. Are you logged into the GeoMesa Atlassian instance? (I wouldn't want that folks have to make an account to access the JIRA/Confluence, but that maybe the way things are configured at the minute.
Also, can you post the URL you are trying to visit?
JB-data
@JB-data
Dear community, hopefully again I can receive some suggestions here...
I'm working for on connecting my local spark to geomesa accumulo.
I have results that are in spark dataframes (e.g. coordinates and a count -like in a spatial density profile), and what I want is write the results to geomesa/accumulo.
At this point I am interested in writing any dataframe (like the ones on which you can do sparksql queries) to geomesa, just to see if the connection works.
Any suggestion of code anywhere to do a very simple test to see if this is working? Most documentation talks more about rdds.
I would use the example here https://www.geomesa.org/documentation/2.2.2/user/spark/core.html#example but could not load the ECQL class (I believe when I run the geomesa-accumulo-spark-runtime jar there is a conflict with one of the other jars but can't figure out with which one.) and also this one looks like it is an rdd not a dataframe.
5 replies
andrijz
@andrijz
sorry for questions around upsert, but that is really important for us.
So far we've done always appending writes, that is why we came to the point where the main ID index is ok - we always have unique FID values enforced by database storage. But for "secondary" attribute indexes we have cases when old values coexist with new ones (referencing the same fid). Is it possible to identify somehow the old/stale values vs the latest ones in those indexes and filter out them during reads or maybe run kind of compaction process during which to clean them out?
James Hughes
@jnh5y

Good question. Off the top of my head, I have some 'bad' ideas for how to implement something like that....

I could imagine dumping the second index table, looking for duplicate FIDs, and then for the duplicate FIDs querying the ID table to see which should be kept.

Generally, it maybe preferable to switch to deleting and inserting when writing to GeoMesa. (Given the context of Cassandra/other key-value stores, an upsert would effectively have to delete the old entries and write the new ones)
yich
@yuqiyich
geomesa-3.0.0+hbase-2.0.5;after I appending a new attr in an existing FeatureType,when I use geomesa query that table with the FeatureType, but nothing out ,there is logs com.ruqi.appserver.ruqi.geomesa.db.GeoDbHandler - Running query dtg DURING 2019-12-31T00:00:00+00:00/2021-01-02T00:00:00+00:00;typeName:recommendPoint hsqldb.db.HSQLDB6BD103EBD8.ENGINE - dataFileCache open start org.locationtech.geomesa.index.planning.StrategyDecider$ - Using strategy provider 'org.locationtech.geomesa.index.planning.StrategyDecider$CostBasedStrategyDecider' org.locationtech.geomesa.utils.audit.AuditLogger$ - {"storeType":"hbase","typeName":"recommendPoint","date":1596620766679,"user":"unknown","filter":"dtg DURING 2019-12-31T00:00:00+00:00/2021-01-02T00:00:00+00:00","hints":"RETURN_SFT\u003drpId:String,addressName:String,title:String,dtg:Date,updateTime:Integer,channel:Integer,*rGeom:Point:srid\u003d4326:precision\u003d6,ext:String,shotCount:Integer","planTime":141,"scanTime":0,"hits":0,"deleted":false} com.ruqi.appserver.ruqi.geomesa.db.GeoDbHandler - Running query INCLUDE;typeName:recommendPoint com.ruqi.appserver.ruqi.geomesa.db.GeoDbHandler - Returned 0 total features but when I check it in Hbase shell ,this the data below,so can someone help me out?
yich
@yuqiyich
image.png
James Hughes
@jnh5y
@yuqiyich hmm.... interesting.
First, when you appended the new attribute, you added it to the end of the SFT definition, correct?
(Doing otherwise, would likely cause issues)
Second, how are you querying for the data?
If you aren't using the command line tools, I'd recommend them as a quick way to debug things (sometimes using GeoServer in the mix can add complexity, etc)
As a third suggestion, have you looked at the logs on the region servers? (I'm wondering if there are any errors or exceptions there about failing to read data, etc)
loridigia
@loridigia
Guys, i have a question.... do you know why (using java api) " ds.createSchema(schema) " where:
ds is a Datastore
schema is a SimpleFeatureType
is blocking?
The method actually creates a table on Hbase, but the method never end
James Hughes
@jnh5y
yes. You do want that method to be synchronous.
In fact, it uses zookeeper to get a distributed lock on the necessary metadata to avoid Really, Quite Bad Things
if the process is hung, you could jstack it, and you might see something about that distributed lock
if so, then there may be an issue with Zookeeper, etc
(or writes to the HBase metadata tables)
yich
@yuqiyich
image.png
image.png

@yuqiyich hmm.... interesting.

first ,thanks for replays,After receiving your reply, I checked carefully ,before add,the SFT is the "rpId:String,addressName:String,title:String,dtg:Date,updateTime:Integer,channel:Integer,rGeom:Point:srid=4326:precision=6;geomesa.stats.enable='true',geomesa.index.dtg='dtg',geomesa.indices='z3:6:3:rGeom:dtg,z2:5:3:rGeom,id:4:3:,attr:8:3:rpId:rGeom:dtg,attr:8:3:updateTime:rGeom:dtg'",after add two 'ext:String' And 'shotCount:Integer' attribute end of SFT , the SFT change to
the"rpId:String,addressName:String,title:String,dtg:Date,updateTime:Integer,channel:Integer,
rGeom:Point:srid=4326:precision=6,ext:String,shotCount:Integer;geomesa.index.dtg='dtg',geomesa.indices='attr:8:3:rpId:rGeom:dtg,attr:8:3:updateTime:rGeom:dtg'",and querying cql is "dtg DURING 2019-12-31T00:00:00.000Z/2021-01-02T00:00:00.000Z",i run it in geomesa commandtools shell "geomesa-hbase explain -c t_rpt_dev_7777 -f recommendPoint -q "dtg DURING 2019-12-31T00:00:00.000Z/2021-01-02T00:00:00.000Z"" and get result that 'No filter plans found' ,but run hbase shell that data really exists,and next i run geomesa-hbase command 'geomesa-hbase explain -c t_rpt_dev_7777 -f recommendPoint -q "rpId = '122.984662_23.986662'"',I can get result ,so what's the problem ?

rexuexiangbei
@rexuexiangbei
i upgrade from 2.2.2 to 3.0 ,compute become slowly.anyone can give me some advise
14 replies
loridigia
@loridigia

yes. You do want that method to be synchronous.

ok i'll take a look to zookeeper and Hbase meta, thx

yich
@yuqiyich
image.png
yich
@yuqiyich
@yuqiyich hmm.... interesting.
I finally find out what's the problem,because I use java Api to update schema ,Use Java class DataStore#updateSchema(String typeName, SimpleFeatureType newfeatureType), and param 'newfeatureType' that contain old attr( keep old,not change) and append the new attr end of SFT ,there is schema changes shotcut that contains change before and change after above,from shotcut you will found that new attr is added ,but old attr index has disappeared, so I had one questions now, first ,how could I use DataStore#updateSchema(String typeName, SimpleFeatureType newfeatureType) Java Api update schema,what should the input param newfeatureType should like?,is there any update schema by java Api demos? ,And I slove this problem by using geomesa-hbase command tools to update schema(add a new attr into SFT,and doc is there link) 。if there have some update schema by java Api demo ,u can send to yichitgo@gmail.com mailbox, thank you very much,best wish for u.
James Hughes
@jnh5y
@yuqiyich the "No filter plans found" note is coming back since there's no index on 'dtg'. GeoMesa will fall back to running full table scans on some tables but not all (and generally it is my position that full table scans are pretty much unacceptable)
your choices are 1) may sure to query on the things you have indexed (e.g. rpId and/or updateTime) or add indexes which will support date range queries on 'dtg'. The previous example which had the z3 index on rGeom and dtg is an example. Alternatively, you could add an index on just dtg (if you expect to support queries like the one you indicated)
James Srinivasan
@jrs53
@jnh5y can I clarify the upsert thing, what happens if I try to write data (say using the CLI or NiFi) with the same fid, assuming other data values are different? (if all the data is the same, it doesn't really matter!)
13 replies
loridigia
@loridigia
Sorry one question, when i run my code from intellij it works fine, but when i compile with maven (tried shade-plugin and maven-assembly) even a simple read of a dataset (using DataFinder) returns null, even if it exists
All paths work, the params are loaded right, how should i compile the library to make it works ?
4 replies
Rafael Ticianelli
@RVTicianelli_twitter
Hi, I'm having trouble running geomesa with google bigtable. I downloaded version 3.0.0 on github and when I run the following error: "Caused by: java.io.IOException: java.lang.NoSuchMethodException: com.google.cloud.bigtable.hbase1_x.BigtableConnection. <init> (org .apache.hadoop.conf.Configuration, java.util.concurrent.ExecutorService, org.apache.hadoop.hbase.security.User) ". I believe it is something related to the dependecies, but I couldn't make it work, can someone help me?
James Hughes
@jnh5y
@RVTicianelli_twitter sorry for the trouble! I can't remember if @elahrvivaz and I had time to test the Google Cloud Bigtable support before the GeoMesa 3.0.0 release.
That said, Google Bigtable changes some details about their API here and there and this causes issues
what version of Google Cloud Bigtable are you using?
and what exactly were you trying to do when you hit that error?
it may be that there's a Cloud Bigtable jar which you need to download. Or it may be the case that there's a version mismatch somewhere
mingnet
@mingnet
How to run geomesa process (such as point2pointprocess) in CQL of GeoServer. Are there any examples.
Rafael Ticianelli
@RVTicianelli_twitter
@jnh5y thanks for your reply. You mean the bigtable jars that i'm using? To fix this error I was trying to update the jar files in geomesa/lib...tried to get the latest version of 'bigtable-hbase.....jar', 'geomesa-bigtable.....jar' and the hadoop and hbase jars
Rafael Ticianelli
@RVTicianelli_twitter
@jnh5y I also realized that the on version that I download is missing the file geomesa/conf/dependencies.sh
JB-data
@JB-data
@elahrvivaz
You proposed this repo: https://github.com/ccri/cloud-local
-> Thank you! Couple questions:
Some qs:
1)it seems like the repo is capable of working with geoserver, but that it is not included. Is this correct?
2)my final goal is to use hbase (as the system we want to run it has hbase). So far I was spending some time on accumulo, as most examples are for this. What is the best link for some examples with basic hbase stuff (doing some geocalculation, and writing results to hbase on geomesa and then rendering on geoserver)
JB-data
@JB-data

ALso related to https://github.com/ccri/cloud-local, when I run the bin/cloud-local.sh init script, it looks promising, until I am asked to login to my own machine (which serves as the hbase master) and provide the password when he complains about JAVA_HOME not set:

Starting hbase...
starting master, logging to /home/user/Documents/cloud-local/hbase-1.3.1/logs/hbase-user-master-ds-gpu11.out
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
jb@ds-gpu11's password: 
ds-gpu11: +======================================================================+
ds-gpu11: |                    Error: JAVA_HOME is not set                       |
ds-gpu11: +----------------------------------------------------------------------+
ds-gpu11: | Please download the latest Sun JDK from the Sun Java web site        |
ds-gpu11: |     > http://www.oracle.com/technetwork/java/javase/downloads        |
ds-gpu11: |                                                                      |
ds-gpu11: | HBase requires Java 1.7 or later.                                    |
ds-gpu11: +======================================================================+

The erros comes from /cloud-local/hbase-1.3.1/bin/hbase-config.sh where there is a check if JAVA_HOME is defined.
I believe that in cloud-local/hbase-1.3.1/conf/hbase-env.sh folder JAVA_HOME can be set , but this file is overwritten every time you run the cloud-local.sh init command.

However on my machine, proper version of java is installed and JAVA_HOME is not empty:

(base) jb@ds-gpu11:~/Documents/cloud-local$ java --version
openjdk 11.0.8 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1, mixed mode, sharing)
(base) jb@ds-gpu11:~/Documents/cloud-local$ echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64
Rafael Ticianelli
@RVTicianelli_twitter

I'm having an error when I try to ingest json file on google bigtable, tried on version 2.4.1 and 3.0.0. Both with the same problem, can anyone help me?

ERROR [org.locationtech.geomesa.tools.ingest.LocalConverterIngest] Failed to write '=1970-01-18T11:00:26.000Z|POINT (-77.7724 5.99455)|test'
java.lang.IllegalArgumentException: Error indexing feature ':Sun Jan 18 11:00:26 UTC 1970|POINT (-77.7724 5.99455)|test'
Caused by: java.lang.IllegalArgumentException: Row length is 0

STFS:

"light" = {
    attributes = [
        { name = "datetime",            type = "Date",          index = true }
        { name = "geom",                type = "Point",         index = true, srid = 4326 }
        { name = "source",              type = "String",        index = false }
    ]
}

CONVERTER:

"light" = {
  type = "json"
  options {
    error-mode = "raise-errors"
  }
  fields = [
    { name = "datetime",            json-type = "string",   path = "$.datetime.$date",      transform = "date('yyyy-MM-dd\\'T\\'HH:mm:ss.SSS\\'Z\\'', $0)" }
    { name = "lat",                 json-type = "double",   path = "$.lat" }
    { name = "lon",                 json-type = "double",   path = "$.lon" }
    { name = "source",              json-type = "string",   path = "$.source"}
    { name = "geom",                transform = "point($lon, $lat)" }
  ]
}

FILE:

{"datetime":{"$date":"1970-01-18T11:00:26.000Z"},"lon":-77.7724,"lat":5.99455,"source":"test"}