Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
James Srinivasan
@jrs53
using build/mvn
had hoped -am -pl :geomesa-accumulo-dist_2.11 would not build that bit
Emilio
@elahrvivaz
i wonder if it's some kind of timing issue, maybe the vals in that class need to be lazy
James Srinivasan
@jrs53
am using multithreaded build
does CI pass?
Emilio
@elahrvivaz
yeah
and i haven't seen it fail building locally either
James Hughes
@jnh5y
@elahrvivaz threading! That's a nice catch
maybe we should slam a 'sequential' at the top of GeoCQEngineIndexChoiseTest?
James Srinivasan
@jrs53
build/mvn clean install -T8 -pl :geomesa-accumulo-dist_2.11 -am is my command line
@jnh5y that's my usual solution to nasty test execution order things
Emilio
@elahrvivaz
@jnh5y it should be thread-safe for reads right?
@jrs53 I usually split the build and test phases, since tests fail sometimes it can save some time
mvn clean install -T8 -DskipTests followed by mvn surefire:test -T3
i've seen that the 'initialization' part of a test body can sometimes be run more than once, or out of order, or different things like that... but making things lazy or defs should fix that
we'd have to move the add features lines inside the lazy init block for each val
Damon Stone
@nomadgis
Is it possible to connect to a GM datastore from a Java only app and cql query and read from it?
Emilio
@elahrvivaz
@nomadgis yes, you just use the GeoTools API
for example, this page shows you how to create an HBase data store: https://www.geomesa.org/documentation/user/hbase/usage.html#creating-a-data-store
the other stores have similar sections
you can also take a look at the example code in our tutorials repo: https://github.com/geomesa/geomesa-tutorials
James Srinivasan
@jrs53
Is there a way of passing accumulo.connector to the Spark df reader? Seems to expect only string->string
Emilio
@elahrvivaz
it has to be serializable... for simplicity we use string->string
passing in the connector directly is mainly for test support
James Srinivasan
@jrs53
I'm trying to use it to test a Kerberos workaround
but org.apache.spark.sql.DataFrameReader expects options to be [String,String]
not [String,Serializable]
Emilio
@elahrvivaz
Connector isn't even serializable though
James Srinivasan
@jrs53
how does that work then?
Emilio
@elahrvivaz
type erasure haha
it's kind of a hack, in other words
in spark, we actually do serialize the param map though
string is just easier to deal with since all our 'real' params are primitives anyway
James Srinivasan
@jrs53
so is it possible to pass in a pre-built Connector when using Spark?
Emilio
@elahrvivaz
no, b/c we can't serialize it
James Srinivasan
@jrs53
oh well, will need to make some other changes :-(
Emilio
@elahrvivaz
is the issue that accumulo isn't supporting the proxy auth right?
i saw your email to the accumulo list
you could add some new params to specify the proxy through string keys, we could use that to construct the connector appropriately
replicate however you're creating the connector now
James Srinivasan
@jrs53
The problem is that in Accumulo 1.7, I can't create a KerberosToken for a proxy user because the guard for that fn is too strict. It was corrected to match the docs for 1.9
I'm not using the 1.9.3 client in GeoMesa and the world's worst Accumulo client (written by yours truly) works having manually built myself a connector
I was hoping to use that connector to test geomesa directly, but can't due to the serialisation issue
Emilio
@elahrvivaz
ooh, you wrote your own client? hardcore
James Srinivasan
@jrs53
I refer you to "world's worst..."
Emilio
@elahrvivaz
haha
you could write a new data store factory that wraps the geomesa accumulo one, and creates your connector appropriately
James Srinivasan
@jrs53
nah, because all I am doing is creating a KerberosToken() rather than KerberosToken(keytab,...)
there are use cases not to have a keytab, and this is one of them