Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
regis larrode
@regislarrode_gitlab
I am trying to ingest json data with geomesa 3.2.0 ingest cmd. output shows failure what ever json converter and json document i am trying to use . The back-end is geomesa kafka datastore. I would need some help here and can provide of course more details. anyone ? Thanks a lot
feature is very simple :

INFO Describing attributes of feature 'windvectorforecasts'
coord | Point
level | Integer
time | Date
uwind | Float
vwind | Float

User data:
geomesa.index.dtg | time
geomesa.kafka.topic | geomesa-ds-kafka-windvectorforecasts

json document is :
{
"id": 1,
"coord": [45.500,5.600],
"level": 200,
"time": 2021-05-04 18:00:00,
"uwind": 2.5623285294e+01,
"vwind": 2.0769083500e-01
}
regis larrode
@regislarrode_gitlab
jon ingest command output shows this:
NFO Schema 'windvectorforecasts' exists
INFO Running ingestion in local mode
INFO Ingesting 1 file with 1 thread
[ ] 0% complete 0 ingested 0 failed in 00:00:01ERROR Fatal error running local ingest worker on /u00/MeteoSwiss/VNZR83/windvectorforecast/windvector_level200small
[============================================================] 100% complete 0 ingested 0 failed in 00:00:01
INFO Local ingestion complete in 00:00:01
INFO Ingested 0 features with no failures for file: /u00/MeteoSwiss/VNZR83/windvectorforecast/windvector_level200small
WARN Some files caused errors, ingest counts may not be accurate
Converter used is ;
geomesa.converters.swisswindjsonconverter = {
type = "json"
id-field = "$id"
feature-path = "$.Features[*]"
fields = [
{ name = "id", json-type = "integer", path = "$.id", transform = "toString($0)" }
{ name = "coord", json-type = "geometry", path = "$.coord", transform = "point($0)" }
{ name = "level", json-type = "integer", path = "$.level" }
{ name = "time", json-type = "date", path = "$.time" }
{ name = "uwind", json-type = "float", path = "$.uwind" }
{ name = "vwind", json-type = "float", path = "$.vwind" }
]
}
regis larrode
@regislarrode_gitlab
SimpleFeature def is :
geomesa.sfts.windvectorforecasts = {
attributes = [
{ name = "coord", type = "Point" }
{ name = "level", type = "Integer" }
{ name = "time", type = "Date" }
{ name = "uwind", type = "Float" }
{ name = "vwind", type = "Float" }
]
}
regis larrode
@regislarrode_gitlab
I get same error types with xml converter and doc i guess the error is somewhere else. fyi: i just noticed 1 error at create-schema command geomesa complaining about Zk/Util path as a warning
James Srinivasan
@jrs53
Have you tried converting your data via the convert command rather than ingest?
Can't remember if float vs double matters in the sft part
Emilio
@elahrvivaz
if your json doc is flat like that, you don't want a feature-path element
James Srinivasan
@jrs53
was bout to say that too :-)
Emilio
@elahrvivaz
i think you'll need to parse out the coordinate array, as the geometry function expects WKT or geojson but not just an array of coords
James Srinivasan
@jrs53
feature-path is when you have an array of features something like {features: [{lat: 1, lon:1...}, {lat: 2, lon: 2...}]}
Emilio
@elahrvivaz
other than that, use the convert command for testing, as James says. and turn up the logging as detailed here to get more error messages: https://www.geomesa.org/documentation/stable/user/convert/parsing_and_validation.html#logging
regis larrode
@regislarrode_gitlab
ok , thanks a lot ! i will try that right now
James Srinivasan
@jrs53
Also there isn't a json-type of date, you'll have to use long or string and convert accordingly
regis larrode
@regislarrode_gitlab
yes thanks ! it got a lot easier when simply realizing the logs in geomesa folder..dumb.. so in fact yes the pb was the date which got solved (also adjust the json doc) like this:
{ name = "time", json-type = "string", path = "$.time", transform = "isoOffsetDateTime($0)" }
no more errors ! tx to all.
date json doc like this : time": "2021-05-04T18:00:00Z"
Emilio
@elahrvivaz
:+1:
regis larrode
@regislarrode_gitlab
NB: The coord were also wrong as you said I had also to change the json doc.
Now it's all ingested.
Question: i am using the SimpleFeature style data above , where could I find an SLD style definition to represent the vector Uwind and Vwind above ? I successfully displayed them as Raster but not yet as Vector data. Any pointer for a ready-to-use SLD vector style def ;-) ?
regis larrode
@regislarrode_gitlab
(i mean SFT type above)
Emilio
@elahrvivaz
hmm, not sure. the geoserver docs have an "SLD cookbook" section that might be useful
regis larrode
@regislarrode_gitlab
NB: As raster is a different layer not the one above of course
Yes i have seen, i need to work on this , in case pre-made are available is easier ;-)
James Srinivasan
@jrs53
Any idea why ingesting data into geomesa on accumulo might lead to massive files in hdfs://user/accumulo/.Trash
Emilio
@elahrvivaz
hmm, i don't even have a '/user' directory in my local hdfs...
James Srinivasan
@jrs53
Will try asking over in Accumulo land
The files do usually disappear over time, but I've filled hdfs -(
James Srinivasan
@jrs53
Do you have HDFS trash enabled? I don't (now)
Emilio
@elahrvivaz
hmm, i didn't disable it but i don't see any trash files even when i delete things
i'm just running a local pseudo-distributed install
James Srinivasan
@jrs53
This may be more a geoserver question, but is there a way of configuring geoserver to invoke a WPS service as if it were a WMS or WFS service? Sort of similar to how geomesa triggers the geomesa:Density function with certain styles
Am thinking how we can get dumb WMS/WFS clients to trigger clever geomesa processing
Emilio
@elahrvivaz
wms you can use a style to invoke a wps in the same way as the density function. i'm not sure if there's any equivalent you could use for WFS. although a lot of the geomesa functions can be triggered using the wfs viewparams parameters, i.e. https://www.geomesa.org/documentation/stable/user/datastores/analytic_queries.html
James Srinivasan
@jrs53
I've got a client that I can't easily control
James Hughes
@jnh5y
Yeah, I think I've seen that issue with Accumulo's GC deleting files in HDFS (and those going to trash!)
Separately, yeah, WMS can fire WPSes in SLDs easily enough.
For WPSes in a WFS.... that's tougher. Ideally there'd be some kind of WPS DataStore that could be set up to apply operations
I ginned up something close-ish to that here: geotools/geotools#3196
There are some somewhat subtle details to fix up along the way
James Srinivasan
@jrs53
Thanks. Also pondering why my id index is the largest