Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Emilio
@elahrvivaz
i've seen that the 'initialization' part of a test body can sometimes be run more than once, or out of order, or different things like that... but making things lazy or defs should fix that
we'd have to move the add features lines inside the lazy init block for each val
Damon Stone
@nomadgis
Is it possible to connect to a GM datastore from a Java only app and cql query and read from it?
Emilio
@elahrvivaz
@nomadgis yes, you just use the GeoTools API
for example, this page shows you how to create an HBase data store: https://www.geomesa.org/documentation/user/hbase/usage.html#creating-a-data-store
the other stores have similar sections
you can also take a look at the example code in our tutorials repo: https://github.com/geomesa/geomesa-tutorials
this class has some good examples: https://github.com/geomesa/geomesa-tutorials/blob/master/geomesa-tutorials-common/src/main/java/org/geomesa/example/quickstart/GeoMesaQuickStart.java
James Srinivasan
@jrs53
Is there a way of passing accumulo.connector to the Spark df reader? Seems to expect only string->string
Emilio
@elahrvivaz
it has to be serializable... for simplicity we use string->string
passing in the connector directly is mainly for test support
James Srinivasan
@jrs53
I'm trying to use it to test a Kerberos workaround
but org.apache.spark.sql.DataFrameReader expects options to be [String,String]
not [String,Serializable]
Emilio
@elahrvivaz
Connector isn't even serializable though
James Srinivasan
@jrs53
how does that work then?
Emilio
@elahrvivaz
type erasure haha
it's kind of a hack, in other words
in spark, we actually do serialize the param map though
string is just easier to deal with since all our 'real' params are primitives anyway
James Srinivasan
@jrs53
so is it possible to pass in a pre-built Connector when using Spark?
Emilio
@elahrvivaz
no, b/c we can't serialize it
James Srinivasan
@jrs53
oh well, will need to make some other changes :-(
Emilio
@elahrvivaz
is the issue that accumulo isn't supporting the proxy auth right?
i saw your email to the accumulo list
you could add some new params to specify the proxy through string keys, we could use that to construct the connector appropriately
replicate however you're creating the connector now
James Srinivasan
@jrs53
The problem is that in Accumulo 1.7, I can't create a KerberosToken for a proxy user because the guard for that fn is too strict. It was corrected to match the docs for 1.9
I'm not using the 1.9.3 client in GeoMesa and the world's worst Accumulo client (written by yours truly) works having manually built myself a connector
I was hoping to use that connector to test geomesa directly, but can't due to the serialisation issue
Emilio
@elahrvivaz
ooh, you wrote your own client? hardcore
James Srinivasan
@jrs53
I refer you to "world's worst..."
Emilio
@elahrvivaz
haha
you could write a new data store factory that wraps the geomesa accumulo one, and creates your connector appropriately
James Srinivasan
@jrs53
nah, because all I am doing is creating a KerberosToken() rather than KerberosToken(keytab,...)
there are use cases not to have a keytab, and this is one of them 
// Don't try this at home
val tableScanner = conn.createScanner("geomesa.gdelt", new Authorizations());

while(tableScanner.iterator.hasNext) {
    println(tableScanner.iterator.next.getKey.toString)
}
tableScanner.close
I'm thinking the PR will allow neither keytab nor password to be set in AccumuloDataStoreFactory (currently this is an error), in which case it uses the no-args KerberosToken() ctor. I'd need to add something to the cli to handle this - currently omitting both prompts for a password (guess new cli option)
Emilio
@elahrvivaz
ah, makes sense
James Srinivasan
@jrs53
not a breaking change?
Emilio
@elahrvivaz
doesn't seem like it would be
jhkcool
@jhkcool
Now I have a problem integrating the SSI algorithm into GEOMESA, CQLFilter seems to have lost its function. Through debugging, I found that the data at scan conforms to the range of CQLFilter, but the result is filtered, and the result size is always 0. The default filter was also tried in Google S2 before, and this problem was not found.
Is there something I haven't noticed that's been bothering me all day.
@elahrvivaz
Emilio
@elahrvivaz
@jhkcool make sure that your geomesa coprocessor version matches the geomesa client version
if you're using hbase, you can try setting hbase.remote.filtering to false in the data store params
that should let you debug locally any filtering
jhkcool
@jhkcool
Is this configuration in Query hints？
Emilio
@elahrvivaz
no, in the data store parameter map
jhkcool
@jhkcool
OK，Let me try.
jhkcool
@jhkcool
@elahrvivaz yeah, I setting hbase.remote.filtering to false, then the query result is right. but while remote filtering the xz2's query result is ok and ssi2's query result is not ok.