Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 17:36
    schnerd starred locationtech/geowave
  • Jan 30 2019 11:01
    hsg77 commented #1474
  • Jan 30 2019 10:58
    hsg77 commented #1474
  • Jan 30 2019 10:57
    hsg77 commented #1474
  • Jan 30 2019 10:53
    hsg77 commented #1474
  • Jan 30 2019 10:53
    hsg77 commented #1474
  • Jan 30 2019 10:51
    hsg77 commented #1474
  • Jan 29 2019 16:30
    JWileczek commented #1474
  • Jan 29 2019 16:30
    JWileczek commented #1474
  • Jan 29 2019 16:12
    rfecher commented #1474
  • Jan 29 2019 10:44
    hsg77 commented #1474
  • Jan 28 2019 22:47
    sunapi386 starred locationtech/geowave
  • Jan 28 2019 21:12

    rfecher on gh-pages

    Lastest javadoc on successful t… (compare)

  • Jan 28 2019 20:47

    rfecher on master

    fixing coveralls (#1488) (compare)

  • Jan 28 2019 20:47
    rfecher closed #1488
  • Jan 28 2019 20:47
    rfecher opened #1488
  • Jan 28 2019 17:02

    rfecher on master

    Update README.md (compare)

  • Jan 28 2019 16:53

    rfecher on master

    updated readme.md (#1486) (compare)

  • Jan 28 2019 16:53
    rfecher closed #1486
Davis Silverman
@sinistersnare
so it looks like the underlying DataStore isnt saving the IndexStrategy and IndexModel for secIdx when I do .addType(). So when the writer does index.getIndexStrategy().getPredefinedSplits(), .getPredefinedSplits NPEs.
Davis Silverman
@sinistersnare
Okay, so if I disable the underlying HBase tables before running my ingest, spark creates >=2 partitions. The first partition correctly creates the indices and the writer has the correct indices in its instance vars. But the second partition will not create the writer correctly, making the indexStrategy and IndexModel null for the writers' secondary index.
Davis Silverman
@sinistersnare
Scratch that, when I swap secIdx and primaryIdx its still returning a null indexStrategy and indexModel for secIdx, even with it being in the first position. I guess it has something to do with my CustomNameIndex, because primaryIdx is an Index provided via geowave APIs. Any idea why a CustomNameIndex would not be retrieved correctly from Geowave?
Davis Silverman
@sinistersnare

this is how Im creating the Index:

default Index makeFuseIdIndex(AttributeDescriptor descriptor) {
    NumericIndexStrategy indexStrategy = new MyIndexStrategy();
    FeatureAttributeDimensionField[] dimms = {new FeatureAttributeDimensionField(descriptor)};
    CommonIndexModel indexModel = new BasicIndexModel(dimms) {
        @Override
        public boolean useInSecondaryIndex() {
            // we want this to be a secondary index. TODO: Is this necessary?
            return true;
        }
    };
    return new CustomNameIndex(indexStrategy, indexModel, this.indexName() + "_ID");
}

Seems fairly benign

Davis Silverman
@sinistersnare
nvm i didnt have the IndexStrategy in the PersistableRegistrySPI, which doesnt explain why the BasicIndexModel didnt get persisted, but that error doesnt occur anymore.
rfecher
@rfecher
sorry, didn't see this until now...hmm, I'd guess it really bombed out reading the CustomNameIndex because the index strategy wasn't persisted properly which could have lead to that basicindexmodel NPE you experienced (and by the way you don't need to override that method for useInSecondaryIndex and if you do you'd want it to be part of a index model class that is registered as persistable) - the meaning of that is if true the dimensions/fields defined in your basicindexmodel will be serialized within each secondary index along with the data ID so that queries can be fully resolved without referring to the data ID index to bring back all the data (but then if downstream processing needs all the data, it will eventually need to hit the data ID index). For example, a spatial index would put the geometry in the basicindexmodel by default and a spatial query would be able to fully run geometry intersection without referring back to the data ID index, but of course if you wanted to get simplefeatures with all the data the geometries that pass the filter would eventually need to get data from the data ID index (if on the other hand you were just doing some form of aggregation that didn't need all the fields, like a count or a heatmap you wouldn't need to go to the data ID index) ... but its at the cost of course of storing the basicindexmodel data in each secondary index, so I think generally speaking unless you know what you're doing with that useInSecondaryIndex method and you have a real use case that supports it pretty well, I'd suggest just leaving that as the default
hipotato
@hipotato
Hello everyone, I just want to ingest a geotif file into gw by the rest api of v0/ingest/localToGW, the type of param "indexList" is string, and the Code annotation is " Array of
Strings". When I pass the param as '[\"abc\"]' or '{\"abc\"}' or other any string , the service aways throw an exception like "java.lang.String cannot be cast to org.json.JSONArray", who can tell me what style string should i pass.
Johnathan Garrett
@jdgarrett
I believe the parameter should be a comma-delimited list such as 'index1,index2'
So in your case just using 'abc' should be sufficient
Johnathan Garrett
@jdgarrett
Hello everyone, I have cut and tagged the GeoWave 1.1.0 release on GitHub. GeoWave 1.1.0 includes a complete overhaul of the user and developer documentation in order to make GeoWave easier to understand and use. In addition it includes the following improvements:
  • New GeoWave GitHub pages site for improved user experience
  • New GeoWave vector query language to simplify queries and aggregations through the CLI
  • Improved custom index capabilities through the programmatic API
  • Additional standalone data stores that can be used for testing via the CLI
  • New quality of life commands to list and describe data stores and types
  • Various bug fixes and improvements
    https://locationtech.github.io/geowave/
Denis Rykov
@drnextgis
Hello. I installed GeoWave on Linux according to Installation Guide. But when I run geowave help command I get these warnings:
$ geowave help
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by javassist.ClassPool (file:/home/denis/geowave/lib/core/javassist-3.20.0-GA.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
WARNING: Please consider reporting this to the maintainers of javassist.ClassPool
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Denis Rykov
@drnextgis
seems like it is something related to Java 11, with Java 8 I don't see it anymore
Johnathan Garrett
@jdgarrett
@drnextgis Thank you for bringing this to our attention. Currently GeoWave is built and tested on Java 8.
Grigory
@pomadchin
@jdgarrett @drnextgis I actually tried to build a library with geowave as a dep using Java 11 (tbh by an accident) - it fails indeed (almost all the reflective stuff throws exceptions)
Nick Levandoski
@metric-chicken
Good morning, I'm using GeoWave-1.1.0 and trying to ingest data with a Mapreduce job. When I run the job I get an error: org.opengis.referencing.NoSuchAuthorityCodeException: No code "EPSG:4326" from authority "European Petroleum Survey Group" found for object of type "IdentifiedOb ject". I'm stumped. Has anyone received this error before?
rfecher
@rfecher
@metric-chicken are you using a jar file that geowave supplies or are you building your own? My understanding is that a dependency org.geotools:gt-epsg-wkt is critical to have on the classpath for discovering these EPSG coordinate reference systems (using SPI), and should be in included in any of the geowave shaded jars that are distributed. The only other thing I can think of if it is on your classpath is that SPI is being hosed somehow in the context of your hadoop workers, not exactly sure how.
Nick Levandoski
@metric-chicken
@rfecher I've been using org.geotools:gt-epsg-hsql. I'll try the gt-epsg-wkt
rfecher
@rfecher
ahh, hsql writes out a DB to a tmp directory and I believe that in fact does cause some trouble in the context of a mapreduce job while gt-epsg-wkt just has it as a resource on the classpath and is fine in the MR worker node context - I think and hope that this will do the trick for you
Nick Levandoski
@metric-chicken
20/04/20 17:07:53 ERROR vector.FeatureDataAdapter: Unable to decode 'EPSG:4326' CRS org.opengis.referencing.NoSuchAuthorityCodeException: No code "EPSG:4326" from authority "European Petroleum Survey Group" found for object of type "IdentifiedObject". I wished it worked
rfecher
@rfecher
damn
Nick Levandoski
@metric-chicken
Ive worked with a co-worker who's really GeoWave savvy but were stumped
its like the Index never gets set in the GeoWaveOutputFormat
I've even tried making my own CUSTOM:4326 CRS using WKT
rfecher
@rfecher
did you take org.geotools:gt-epsg-hsql out of the classpath when you added org.geotools:gt-epsg-wkt?
Nick Levandoski
@metric-chicken
yes. I commented it out
rfecher
@rfecher
I'm not entirely sure what you mean by "its like the Index never gets set in the GeoWaveOutputFormat"
well, in other words, I'm not sure how that relates to the problem
Nick Levandoski
@metric-chicken
It might not. Im new to GeoWave. I assumed that I had to specify the CRS in the SpatialTemporalIndex.
rfecher
@rfecher
you can optionally specify it, although the default CRS is EPSG:4326
Nick Levandoski
@metric-chicken
I just noticed that in the source, sorry
Maybe adding the gt-epsg-wkt to the distributed class will work. let me try that
rfecher
@rfecher
thats ok, but then to do any coordinate transforms geotools needs to look up the crs in the classpath (geotools is a popular underlying library that geowave uses for many GIS operations such as this)
yeah, it has to be on the worker nodes classpath because they are the ones really doing the ingest
Nick Levandoski
@metric-chicken
getting the same error but I'm seeing this as well WARNING: Extension lookup 'ExtensionFilter', but ApplicationContext is unset. 20/04/20 17:30:42 ERROR util.GeometryUtils: Unable to decode EPSG:4326 CRS
rfecher
@rfecher
i take it this is not using the geowave CLI, right? If so, I wonder if you can try using the CLI: geowave-$VERSION-$VENDOR-tools as documented here: https://locationtech.github.io/geowave/downloads.html
I believe it assumes HADOOP_HOME is set so that it can launch the MR ingest job
Nick Levandoski
@metric-chicken
no its using yarn to deploy the job
rfecher
@rfecher
at least that way if that part works we could back off and see what is different or it doesn't work with the same problem then we'd know it isn't necessarily related to how you launch it
Nick Levandoski
@metric-chicken
ok ill give it a shot and let you know
Michael Fazio
@michaeljfazio
Any advise for building fat jar with geowave-datastore-hbase and geowave-adapter-vector for Spark? Seems like a never ending assembly merge conflict :(
Michael Fazio
@michaeljfazio
at least with geowave-adapter-vector, it would seem there are many conflicting transitive dependencies.
Michael Fazio
@michaeljfazio
Nevermind! Gave up on sbt. Went for maven instead ;)
Michael Fazio
@michaeljfazio
Hi @rfecher
I've developed a plugin you might be interested in for addition to geowave core.
Let me know if you have some time to discuss
rfecher
@rfecher
sorry, I just caught this message now @michaeljfazio ... is this the Glue MetaStore ingest format discussed in issue #1705 ?
Zachary Whitley
@zacharywhitley
I know this is a stupid question but can someone let me know what artifact provides SpatialIndexBuilder?
rfecher
@rfecher
@zacharywhitley it's in org.locationtech.geowave:geowave-core-geotime
Kirin Patel
@ajchili
Hi all, I am just getting started with Geowave and DynamoDB for spatial-temporal queries and I not sure if the latency I am experiencing with queries should be expected. Currently I have some GDELT data (roughly 4k rows) and I am querying on actor name but my query is taking roughly 6 seconds. I am using the Java SDK. Does anyone have any insight on why this may be the case or if this is the expected behavior?
James Hughes
@jnh5y
Did you create an index on actor name? (If not, your query may end up scanning everything in DynamoDB)