by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 17:36
    schnerd starred locationtech/geowave
  • Jan 30 2019 11:01
    hsg77 commented #1474
  • Jan 30 2019 10:58
    hsg77 commented #1474
  • Jan 30 2019 10:57
    hsg77 commented #1474
  • Jan 30 2019 10:53
    hsg77 commented #1474
  • Jan 30 2019 10:53
    hsg77 commented #1474
  • Jan 30 2019 10:51
    hsg77 commented #1474
  • Jan 29 2019 16:30
    JWileczek commented #1474
  • Jan 29 2019 16:30
    JWileczek commented #1474
  • Jan 29 2019 16:12
    rfecher commented #1474
  • Jan 29 2019 10:44
    hsg77 commented #1474
  • Jan 28 2019 22:47
    sunapi386 starred locationtech/geowave
  • Jan 28 2019 21:12

    rfecher on gh-pages

    Lastest javadoc on successful t… (compare)

  • Jan 28 2019 20:47

    rfecher on master

    fixing coveralls (#1488) (compare)

  • Jan 28 2019 20:47
    rfecher closed #1488
  • Jan 28 2019 20:47
    rfecher opened #1488
  • Jan 28 2019 17:02

    rfecher on master

    Update README.md (compare)

  • Jan 28 2019 16:53

    rfecher on master

    updated readme.md (#1486) (compare)

  • Jan 28 2019 16:53
    rfecher closed #1486
rfecher
@rfecher
yep, it seems like we should add a couple convenience methods here to wrap that logic up, it should be easy
shortwavedave
@shortwavedave

I wrote a convenience method that allows me to pass in the adapter and datastore directly. seems to work ok, I'll fork/pull after I get some work done.

On another note, any idea what the purpose of the coverage name is? It seems like we use it to label an adapter, but I think I came across a case where I couldn't ingest two coverages with the same name, so I couldn't reuse the adapter to search over both coverages. Does that seem right?

rfecher
@rfecher
Yes, please PR when you have a chance. Eclipse requires us to have contributors sign a contributor agreement electronically here.
rfecher
@rfecher
the coverage name is a label for the adapter serving as the layer name, but it allows you to either ingest images as individual layers (different coverage names) or mosaic them into a single layer (same name). For the latter to work, the images have to have the same SampleModel which is basically data layout. Did you get any meaningful error when you tried to ingest multiple with the same coverage name?
perhaps playing around with landsat8 commandline utilities would give you a good idea of how that works? by playing with --coverage you end up choosing whether to mosaic landsat scenes or keep them separate (using ${entityId})
John Meehan
@n0rb3rt
Hi I’m wondering if Geowave can support heterogeneous SFTs in a single table?
mawhitby
@mawhitby
@n0rb3rt Yes it does. We use a common index model for all data types so you can ingest them all into a single table if you need to
rfecher
@rfecher
yeah, and a little more on that "common index model" - if you want you can query across all SFTs within a single scan using the indexed fields such as space/time (and potentially other fields but you'd have to do something a bit custom to support normalizing other fields across heterogeneous SFTs). This graphic from our developer guide is meant to be a Venn diagram on the left, in that the primary index data is a subset of the common indexed data which is a subset of the native data, and our default is simply use the primary indexed fields (what goes in the key, usually space/time) for the common index data. Basically, this notion under the hood allows doing a single scan for anything that is part of common index data rather than requiring a scan per SFT if that's a use case of interest.
John Meehan
@n0rb3rt
Playing that out a bit more, if I ingested features with various SFTs and then try to access that table in GeoServer via WFS, would I get just the common types as the attributes of those features, or would i need to write my own feature reader to handle that sort of thing?
rfecher
@rfecher
as it works now out of the box each SFT by unique feature type name would be a layer in GeoServer - there'd need to be some custom code to expose the layers differently
it'd be fairly straightforward to expose a single layer composed of many SFTs with basically just a geometry attribute for spatial, or geometry + time if you're using spatial-temporal, but it seems that would only make sense if you really wanted the individual layers too (otherwise you'd just model your data as one SFT in the first place)
rfecher
@rfecher
as far as exposing common subsets of attributes into hybrid layers, I'd have to understand how that is generalizable, and think of how it could be exposed reasonably simplistically (seems like there could be lots of candidate combinations for sets of attributes) - it feels like a fairly custom use case
John Meehan
@n0rb3rt
I'll have have to experiment a bit. In my case I have potentially thousands of different SFTs, though each with a few common attributes. So the common model seems very useful. But I need to figure out what to do with the rest of the sparse attributes. Maybe I just stick them in a generic bag.
For the common attributes, when is it better to make them dimensions of the SFC vs secondary indices? When they're nullable?
or not nullable rather
rfecher
@rfecher
hmm, yeah, sticking the extended fields in a generic bag could be an option - if you have SFT1 with attributes A,B,C and SFT2 with attributes C,D,E, do you really just want SFT3 with attribute C + generic bag, or do you also want SFT1 and SFT2 as individual layers within GeoServer?
how you index the data is going to be driven by the selectors you most frequently want to use on retrieval
John Meehan
@n0rb3rt
But if the common attributes are in all or most queries then they're generally better off as SFC dimensions?
rfecher
@rfecher
definitely - are the common attributes numeric?
John Meehan
@n0rb3rt
off the top of my head it would be id (Long), layerName or fileName (String), timestamp (Date as Long)
rfecher
@rfecher
are you doing a range query with id? seems to me the answer is likely no?
John Meehan
@n0rb3rt
no
rfecher
@rfecher
if you already have the id, is space-time constraints necessary?
John Meehan
@n0rb3rt
id isn't unique
it represents a group of input data tied to an external system
rfecher
@rfecher
ahh, got it
so our index is composable and to me, I'd structure it as <layerName> + <id> + <SFC>
we have a partition key and a sort key that are completely logically separated throughout the code, the SFC is meant to be well-sorted and support range scan, the other things are not necessary to be well-sorted and can form part of the partition key (we call it partition key, which maps to partition key in cassandra, hash key in dynamodb, and in hbase and accumulo just become a row ID prefix that leverages the concept of "pre-splitting")
John Meehan
@n0rb3rt
Ok sounds like I should roll up my sleeves and try it. Thanks for the tips!
rfecher
@rfecher
ha, yeah, let me know if you have questions along the way - out of the box we provide spatial and spatiotemporal indices through SPI and when I have provided custom indexing I end up also extending FeatureDataAdapter to make sure the attributes within the feature get mapped correctly, here's one way I've done it for NYC taxi data, and I've done other approaches as well not as shareable, but mapping fields like layername and ID to the index is where at the moment some custom code in our data adapter is required
so I always end up making some one-off hard-coded "adapter" that does the mapping and its never very hard, but while you're in there I'd love to get feedback if anything dawns on you for a decent more generic approach that wouldn't require that custom code to do the mappings
Brad Hards
@bradh
The user guide (https://locationtech.github.io/geowave/userguide.html) says: "A query in GeoWave currently consists of a set of ranges on the dimensions of the primary index. Up to three dimensions, plus temporal optionally, can take advantage of any complex OGC geometry for the query window. For dimensions of four or greater the query can only be a set of ranges on each dimension, e.g., hyper-rectangle, etc."
Does that mean I can do a query for an n-dimensional bounding box in any dimensionality, but can do an efficient query for (say) a buffered linestring?
Thinking about storing RF emitter information (where the dimensions might be 3D+T, plus some kind of transmit frequency (or maybe a frequency range + instantaneous bandwidth).
rfecher
@rfecher
hi @bradh - yes, I think you have it right. For the full scope of geometry types and relationships (the DE-9IM model we heavily leverage JTS, but for arbitrary dimensionality we generally support basic range constraints (essentially hyper-rectangle intersection). With some custom code an index can be composed of any dimensional definitions you define. The core of the project treats indexing as purely generic multi-dimensional and the default indexing for spatial and spatial-temporal are added on top via SPI here in the geowave-geotime module. But the idea is that special purpose dimensional definitions can be added in the same way that these are.
HuiWang
@scially

When use other CRS(CGCS2000), the table name in accumulo is inconsistent with the name of the table to be queried by the geoserver .

And Does GeoWave support Gaussian projection plane coordinate system, such as "CGCS2000 / 3-degree Gauss-Kruger CM 114E"( EPSG:4547)?

image.png
image.png
rfecher
@rfecher
that is strange that it would look for SPATIAL_IDX and my initial guess is it may be because of some mishaps on that datastore at some point (maybe a failed ingest to the default index?) ... try geowave remote listindex <datastore> and if SPATIAL_IDX prints to the console then there must have been an ingest attempt at some point to that index. On a new ingest, the indexing scheme and data type is serialized to the GEOWAVE_METADATA table and geoserver is simply referencing that info. So my guess is it finds it in the metadata table and is looking for the index. If it is listed in your datastore now, try a new gwNamespace and on a clean ingest, try geowave remote listindex <store> to see if it exists
re: EPSG:4547, because it has infinite bounds, I think you'll have to wait for 0.9.8 and it'll work. 0.9.7 works for CRS's with strict bounds, but 0.9.8 will work for infinite bound CRS's as well.
HuiWang
@scially
ok,thank you , i try it
Josée-Anne Langlois
@jalanglois1_twitter
Hello, I'm new to Geowave. I want to test raster ingestion in an accumulo data store. I have an accumulo installed on an ubuntu standalone VM. I created the package accumulo-container-singlejar from the source and put the jar file in $ACCUMULO_HOME/lib directory. But when I do this, I can't restart accumulo. I have the following error :
lanj2410@ubuntu2:/opt$ /opt/accumulo-1.7.4/bin/start-all.sh
Starting monitor on localhost
WARN : Max open files on localhost is 1024, recommend 32768
Starting tablet servers .... done
Starting tablet server on localhost
WARN : Max open files on localhost is 1024, recommend 32768
2018-08-01 16:03:58,395 [start.Main] ERROR: Uncaught exception
java.util.ServiceConfigurationError: org.apache.accumulo.start.spi.KeywordExecutable: Provider org.apache.accumulo.server.conf.ConfigSanityCheck not a subtype
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.accumulo.start.Main.checkDuplicates(Main.java:223)
at org.apache.accumulo.start.Main.getExecutables(Main.java:215)
at org.apache.accumulo.start.Main.main(Main.java:78)
Starting master on localhost
WARN : Max open files on localhost is 1024, recommend 32768
Starting garbage collector on localhost
WARN : Max open files on localhost is 1024, recommend 32768
Starting tracer on localhost
WARN : Max open files on localhost is 1024, recommend 32768
Can you help me with that?
rfecher
@rfecher
It looks like a bit of an accumulo configuration issue but I'm not certain as I've never seen that before.
Try following these instructions: https://locationtech.github.io/geowave/userguide.html#accumulo-config as that's the basics I always follow.
mawhitby
@mawhitby
@jalanglois1_twitter Sorry if I'm miss-understanding, but are you talking about building the geowave accumulo jar?
If so that jar is would need to be put in the accumulo/lib directory on HDFS not in the local accumulo lib directory.
Josée-Anne Langlois
@jalanglois1_twitter
Thanks @mawhitby ! I was putting it in the local accumulo lib directory. Now that the file is on HDFS I can start Accumulo.
mawhitby
@mawhitby
Awesome! Happy to help.
HuiWang
@scially
when i ingest shapefile, and add layer to geoserver, some filed(Chinese) in openlayer are garbled, the shapefile charset is GBK