Bringing the scalability of distributed computing to modern geospatial software.
@hipotato you need to be more descriptive. Please remember we can't see your screen, so you need to provide background that allows us to understand what you have done before this, what your environment looks like (e.g. the IDE - does the basic geowave command work, have you adjusted
PATH or similar) and what you expected to happen instead.
I have ingested raster (GeoTiff) successfully on the standalone installation on linux. Now I want debug the source code of gw in IntelliJ IDEA on Windows7, to go through add store ,add index ,ingest raster function. The main class is
org.locationtech.geowave.core.cli.GeoWaveMain,the program arguments is store add raster --gwNamespace geowave.raster -t hbase --zookeeper node24,node25,node26:2181, when run ,the console output is Usage: geowave [options]
Options:
-cf, --config-file
Override configuration file (default is
<home>/.geowave/config.properties)
--debug
Verbose output
--version
Output Geowave build version information
Commands:
config
Commands that affect local configuration only
explain
See what arguments are missing and what values will be used for GeoWave commands
help
Get descriptions of arguments for any GeoWave command
util, utility
GeoWave utility commands
Error: Expected a command, got store
@hipotato It looks like only the core CLI is on your classpath. GeoWave builds up the CLI with each project adding commands here and there. In order to get all of the commands, execute the main class from a classpath that has the artifacts for all of the GeoWave projects. One easy way to do it is to debug from the
geowave-test project, which includes most of the other GeoWave projects as dependencies.
Is secondary indexing supposed to be as easy as
options.setSecondaryIndexing(true);
...
dataStore.addType(adapter, primaryIdx, secIdx);or is there some other way to say that secIdx is the secondary index of the first? Currently I am doing the above and hitting a strange NPE in the GW codebase
the data store option for secondary indexing writes out a data ID index where all the actual values are stored with the key being the data ID, then the other indices (in your case
primaryIdx and secIdx store just the data IDs as the values, and if necessary will refer back to the data ID index for the full data). This all should happen transparently to the API (ie. the data ID index does not need to be referenced anywhere). Does that make more sense given what you're experiencing?
Okay, so if I disable the underlying HBase tables before running my ingest, spark creates >=2 partitions. The first partition correctly creates the indices and the writer has the correct indices in its instance vars. But the second partition will not create the writer correctly, making the indexStrategy and IndexModel null for the writers' secondary index.
Scratch that, when I swap
secIdx and primaryIdx its still returning a null indexStrategy and indexModel for secIdx, even with it being in the first position. I guess it has something to do with my CustomNameIndex, because primaryIdx is an Index provided via geowave APIs. Any idea why a CustomNameIndex would not be retrieved correctly from Geowave?
this is how Im creating the Index:
default Index makeFuseIdIndex(AttributeDescriptor descriptor) {
NumericIndexStrategy indexStrategy = new MyIndexStrategy();
FeatureAttributeDimensionField[] dimms = {new FeatureAttributeDimensionField(descriptor)};
CommonIndexModel indexModel = new BasicIndexModel(dimms) {
@Override
public boolean useInSecondaryIndex() {
// we want this to be a secondary index. TODO: Is this necessary?
return true;
}
};
return new CustomNameIndex(indexStrategy, indexModel, this.indexName() + "_ID");
}Seems fairly benign
sorry, didn't see this until now...hmm, I'd guess it really bombed out reading the CustomNameIndex because the index strategy wasn't persisted properly which could have lead to that basicindexmodel NPE you experienced (and by the way you don't need to override that method for
useInSecondaryIndex and if you do you'd want it to be part of a index model class that is registered as persistable) - the meaning of that is if true the dimensions/fields defined in your basicindexmodel will be serialized within each secondary index along with the data ID so that queries can be fully resolved without referring to the data ID index to bring back all the data (but then if downstream processing needs all the data, it will eventually need to hit the data ID index). For example, a spatial index would put the geometry in the basicindexmodel by default and a spatial query would be able to fully run geometry intersection without referring back to the data ID index, but of course if you wanted to get simplefeatures with all the data the geometries that pass the filter would eventually need to get data from the data ID index (if on the other hand you were just doing some form of aggregation that didn't need all the fields, like a count or a heatmap you wouldn't need to go to the data ID index) ... but its at the cost of course of storing the basicindexmodel data in each secondary index, so I think generally speaking unless you know what you're doing with that useInSecondaryIndex method and you have a real use case that supports it pretty well, I'd suggest just leaving that as the default
Hello everyone, I just want to ingest a geotif file into gw by the rest api of v0/ingest/localToGW, the type of param "indexList" is string, and the Code annotation is " Array of
Strings". When I pass the param as '[\"abc\"]' or '{\"abc\"}' or other any string , the service aways throw an exception like "java.lang.String cannot be cast to org.json.JSONArray", who can tell me what style string should i pass.
Strings". When I pass the param as '[\"abc\"]' or '{\"abc\"}' or other any string , the service aways throw an exception like "java.lang.String cannot be cast to org.json.JSONArray", who can tell me what style string should i pass.
So in your case just using 'abc' should be sufficient
Hello everyone, I have cut and tagged the GeoWave 1.1.0 release on GitHub. GeoWave 1.1.0 includes a complete overhaul of the user and developer documentation in order to make GeoWave easier to understand and use. In addition it includes the following improvements:
- New GeoWave GitHub pages site for improved user experience
- New GeoWave vector query language to simplify queries and aggregations through the CLI
- Improved custom index capabilities through the programmatic API
- Additional standalone data stores that can be used for testing via the CLI
- New quality of life commands to list and describe data stores and types
- Various bug fixes and improvements
https://locationtech.github.io/geowave/
Hello. I installed GeoWave on Linux according to Installation Guide. But when I run
geowave help command I get these warnings:$ geowave help
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by javassist.ClassPool (file:/home/denis/geowave/lib/core/javassist-3.20.0-GA.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
WARNING: Please consider reporting this to the maintainers of javassist.ClassPool
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Good morning, I'm using GeoWave-1.1.0 and trying to ingest data with a Mapreduce job. When I run the job I get an error:
org.opengis.referencing.NoSuchAuthorityCodeException: No code "EPSG:4326" from authority "European Petroleum Survey Group" found for object of type "IdentifiedOb
ject". I'm stumped. Has anyone received this error before?
@metric-chicken are you using a jar file that geowave supplies or are you building your own? My understanding is that a dependency
org.geotools:gt-epsg-wkt is critical to have on the classpath for discovering these EPSG coordinate reference systems (using SPI), and should be in included in any of the geowave shaded jars that are distributed. The only other thing I can think of if it is on your classpath is that SPI is being hosed somehow in the context of your hadoop workers, not exactly sure how.
ahh, hsql writes out a DB to a tmp directory and I believe that in fact does cause some trouble in the context of a mapreduce job while
gt-epsg-wkt just has it as a resource on the classpath and is fine in the MR worker node context - I think and hope that this will do the trick for you
20/04/20 17:07:53 ERROR vector.FeatureDataAdapter: Unable to decode 'EPSG:4326' CRS
org.opengis.referencing.NoSuchAuthorityCodeException: No code "EPSG:4326" from authority "European Petroleum Survey Group" found for object of type "IdentifiedObject". I wished it worked
its like the Index never gets set in the GeoWaveOutputFormat
I've even tried making my own CUSTOM:4326 CRS using WKT
well, in other words, I'm not sure how that relates to the problem
Maybe adding the
gt-epsg-wkt to the distributed class will work. let me try that
yeah, it has to be on the worker nodes classpath because they are the ones really doing the ingest
i take it this is not using the geowave CLI, right? If so, I wonder if you can try using the CLI:
geowave-$VERSION-$VENDOR-tools as documented here: https://locationtech.github.io/geowave/downloads.html
I believe it assumes HADOOP_HOME is set so that it can launch the MR ingest job
