Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
JB-data
@JB-data
Q1:I wonder if geomesa-hbase has been extensively used with cloud storage, or more with hdfs. I have the feeling the IO might be quite a bit better when using hdfs and maybe during the development days, hdfs was standard, while now there has been more a shift towards things like s3/adls. I would be really interested in seeing some performance study for larger volumes of data.
Q2:A way to mitigate the time it takes to read data, is to invest in caching. Therefore, I am wondering (but this is more a hbase question), what is the best method to do a full table scan so that the table I am interested in, will be fully cached, before I start firing CQL queries on it.
34 replies
Maxime Richard
@RichaMax

Hello, I am using GeoMesa with accumulo and i would like to use ABAC capability of accumulo,
I was wondering if it was possible to directly set the visibility and security attribut in the SimpleFeatureType while ingesting files using geomesa-accumulo ingest ...

I was thinking of something like this but I did find anything regarding this in the documentation

geomesa {
  sfts {
    dvf2014 = {
      fields = [
        { name = "public_visible_attribut", type = "String", visibility = "public|user|admin" }
        { name = "user_visible_atrribut"  , type = "String", visibility = "user|admin" }
        { name = "my_secret_attribut"     , type = "String", visibility = "admin" }
      ]
    }
  }
}

I saw this tutorial: https://www.geomesa.org/documentation/stable/tutorials/geomesa-examples-featurelevelvis.html?highlight=visibility#feature-level-visibility-and-security
but this requires to write code and I would like to avoid this as much as possible for now

James Srinivasan
@jrs53
Am having a bit of a dependency nightmare - spent a long time getting a geomesa-accumulo CLI working. If I copy my entire CLI directory somewhere else, it still works. If I try to recreate it (such that diff -rq old new says only the log file differs) I get a cp error
39 replies
robertovarga14
@robertovarga14:matrix.org
[m]
I have geomesa-hbase 3.2.0 in a node and the ingestions work for me, however I have geomesa-hbase 3.1.2 in another node and configured in the same way and it does not work for me and i obtain this error java.lang.reflect.InvocationTargetException
java.util.concurrent.CompletionException: java.lang.reflect.InvocationTargetException
at com.github.benmanes.caffeine.cache.UnboundedLocalCache$UnboundedLocalLoadingCache.lambda$new$0(UnboundedLocalCache.java:929)
at com.github.benmanes.caffeine.cache.UnboundedLocalCache.lambda$computeIfAbsent$2(UnboundedLocalCache.java:235)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at com.github.benmanes.caffeine.cache.UnboundedLocalCache.computeIfAbsent(UnboundedLocalCache.java:231)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:113)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:65)
at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$.getConnection(HBaseConnectionPool.scala:104)
at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:42)
at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:33)
at org.geotools.data.DataAccessFinder.getDataStore(DataAccessFinder.java:119)
at org.geotools.data.DataStoreFinder.getDataStore(DataStoreFinder.java:69)
at org.locationtech.geomesa.tools.DataStoreCommand$class.loadDataStore(Command.scala:67)
at org.locationtech.geomesa.hbase.tools.HBaseRunner
KaTeX parse error: Can't use function '$' in math mode at position 5: anon$̲2.loadDataStore…: anon$2.loadDataStore(HBaseRunner.scala:32)
        at org.locationtech.geomesa.tools.DataStoreCommand$class.withDataStore(Command.scala:61)
        at org.locationtech.geomesa.hbase.tools.HBaseRunner
anon$2.withDataStore(HBaseRunner.scala:32)
at org.locationtech.geomesa.tools.ingest.IngestCommand
KaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲getSftAndConver…: anonfun$getSftAndConverter$1
anonfun$17
KaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲18.apply(Ingest…: anonfun$18.apply(IngestCommand.scala:221)
        at org.locationtech.geomesa.tools.ingest.IngestCommand
anonfun$getSftAndConverter$1
KaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲17: anonfun$17
anonfun$18.apply(IngestCommand.scala:221)
at scala.
1 reply
James Hughes
@jnh5y
@robertovarga14:matrix.org you cannot mix and match GeoMesa versions on HBase. Also, the error you posted is not rendering well enough in Gitter for it to be read.
Emilio
@elahrvivaz
@robertovarga14:matrix.org we use reflection to work around the HBase api to support multiple hbase versions. one of the calls to hbase is failing, but it's hard to say what. if there's a caused by that might help. but likely the issue is the hbase client jars you're using are not matching up with the hbase server, and/or you're using a version of hbase that geomesa doesn't support
Maxime Richard
@RichaMax

Hello, I tried the solution you proposed to set the visibility rights, but unfortunatly I cannot get it to work inside my python or scala code. I defined converters as in the examples you linked and ingested the data, set the rights of my differents users and in the accumulo shell it works perfectly each user can only see what he is allowed to see. The problem appears when i try to read the data from my code, for example (in scala using spark-shell):

$> spark-shell --master my_master --jars /path/to/geomesa/spark/runtimeaccumulo1.jar

...

scala> # doing necessary import

scala> val params1 = Map("accumulo.instance.id" -> "accumulo","accumulo.zookeepers" -> "zookeeper", "accumulo.user" -> "user1", "accumulo.password" -> "mdp1", "accumulo.catalog" -> "data_with_feature_level_vis") // user1 is allowed to read the table and has all of the necessary auths according to getauths -u user1 in accumulo shell


scala> val df = spark.read.format("geomesa").options(params_ABC).option("geomesa.feature", "feature_level_vis").load()

scala> df.show(5)

The show function return an empty dataframe (all lines are empty except the header one). If I load the data into layer in geoserver with different users it works correctly. I was wondering if you knew where my problem might be comming from ?

15 replies
Andrey Avtomonov
@andreybavt

Hello! I'm trying to understand how FastDwithin works with a threshold in meters. I noticed that the logic is based on calculation of a min and max degree distances from a given point in order to avoid calculating expensive getOrthodromicDistance for objects that are too close or too far. Here's the corresponding code:

https://github.com/locationtech/geomesa/blob/main/geomesa-utils/src/main/scala/org/locationtech/geomesa/utils/geotools/GeometryUtils.scala#L107

    override def evaluate(obj: AnyRef): Boolean = {
      val geom = exp1.evaluate(obj).asInstanceOf[Geometry]
      if (geom == null || envelope.distance(geom.getEnvelopeInternal) > maxDegrees) { false } else {
        val op = new DistanceOp(geometry, geom)
        op.distance <= minDegrees || {
          val Array(p1, p2) = op.nearestPoints()
          val calculator = calculators.get
          calculator.setStartingGeographicPoint(p1.x, p1.y)
          calculator.setDestinationGeographicPoint(p2.x, p2.y)
          calculator.getOrthodromicDistance <= meters
        }
      }
    }

The maximum possible angle is defined by going straight east (azimuth 90°).

However, I wonder how it works. For instance for a POINT (37.630866 55.752256) and a distance of 2400 km
An angular distance when going with azimuth 90° will be 35° whereas if we go with azimuth 49° a distance will be 41°, so greater than what FastDwithin considers as maximum.

Thanks in advance

17 replies
Michał Gołębiewski
@mjgolebiewski
hello guys, does geomesa supports points cloud data from LIDAR?
8 replies
Kolya Yanchiy
@gokareless_twitter

Hi! I'm trying to ingest features to Geomesa (backed by Accumulo) and getting

ERROR Unable to connect to Zookeeper instance within timeout period. Consider increasing the Zookeeper timeout by setting the system property 'instance.zookeeper.timeout' (default: 5000ms).

Any idea why?
my zoo.cfg is

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=DATA_DIR/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=100
# whitelist a few useful four-letter-word commands
4lw.commands.whitelist=mntr,stat,ruok,wchs
# disable the admin server listening on 8080 by default
admin.enableServer=false
instance.zookeeper.timeout=60000

and my accumulo-site.xml does contain

  <property>
    <name>instance.zookeeper.timeout</name>
    <value>90s</value>
  </property>
11 replies
James Srinivasan
@jrs53
Am getting this error trying to set visibilities: Caused by: org.apache.accumulo.core.util.BadArgumentException: bad character (44) near index 4
21 replies
geomesa.feature.visibility ="'fooo,baar,baar,baar...'"
James Srinivasan
@jrs53
Is this familiar to anyone from geoserver? net.sf.saxon.trans.XPathException: The SAX2 parser org.geotools.xml.transform.TransformerBase$XMLReaderSupport does not support setting the &apos;namespaces&apos; feature to true The SAX2 parser org.geotools.xml.transform.TransformerBase$XMLReaderSupport does not support setting the &apos;namespaces&apos; feature to true Feature: http://xml.org/sax/features/namespaces when invoking GetCapabilities?
20 replies
robertovarga14
@robertovarga14:matrix.org
[m]
good I have geomesa-hbase 3.2.0 and I have the following error in a python notebook after reading from hbase when I do a df.head () Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:372)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
2 replies
dianahd
@dianahd:matrix.org
[m]
Hi, We would need some help please. We have an HDP3.1.4 distribution with kerberos and with HBase 2.0.2 ..we try to read data from HBase with geomesa, but there is an error , we tried with Geomesa 3.2.0 and Geomesa 3.1.2 but they both do not work. Which version do you recomend us please? Any suggestion? Thanks in advance! ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:372)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
5 replies
Deepak Lewis
@tailrecur

Hey folks,

I'm brand new to the geospatial world and would appreciate some feedback/review of my design choices:
I'm building a system that needs to find all polygons that contain an arbitrary point in an input query. The dataset has around a million polygons and receives polygon updates over Kafka. It need to handles a high query throughput at low latency.

My gut feel (will validate with tests later) is that the dataset should fit in memory on a single server and was thinking of having an autoscaled pool of such servers. Each server keeps the entire index in memory in a Kafka Data Store. This store would be fed from a separate Kafka compacted topic containing the full dataset.

The concerns I have are:

  • Time to reload the dataset into the Kafka Data Store in the case of a restart.
  • Need to hack the store implementation to be able to have multiple consumers of the same Kafka partition.

Are there other spatial indices/data stores more suited to this need of keeping the data in local memory and fast loading on boot-up? eg: The FileSystem DataStore alongwith S3.

Thanks in advance!

15 replies
James Srinivasan
@jrs53
If I add index = true to timestamp and geospatial attrs in my sft, that'll create extra attribute indices? I probably don't want that
1 reply
James Srinivasan
@jrs53
Having used delete-catalog is there a magic accumulo command IIII need to run to free up space in HDFS?
4 replies
dianahd
@dianahd:matrix.org
[m]
Hi, yes, sorry I did not tell you! We are using geomesa_pyspark..The binary from maven is valid for all datastores (hbase,accumulo, ..)https://mvnrepository.com/artifact/org.locationtech.geomesa/geomesa_pyspark/3.1.2 is it ok? Thank you very much!
1 reply
James Srinivasan
@jrs53
Is there a way of writing a json converter when your input data file is a raw json array of objects?
5 replies
James Srinivasan
@jrs53
Interesting, running latest geomesa-accumulo w spark and getting this on the second invocation of any spark code touchign the df
57 replies
JB-data
@JB-data
A question about data engineering and processing to write data to geomesa:
is it possible to save a column as field array of string?
I would like to store for each geospatial datapoint, during my processing some additional info about an attribute, let's call it the "nature" of my point in a field that is an array.
This attribute can have quite a lot of values (3k different ones).
Multiple natures are allowed, but I dont expect to have more than 10 values.
So I will end up with:
POINT1, [nature1, nature50, nature289]
POINT2, [nature3, nature12,nature600,nature879]
POINT3, [nature10,nature50,nature989]
etc.
Then in my CQL filter I would like to write
natures in ('nature50') and hope to return only POINT1 and POINT3.
Does this sound feasible?
2 replies
James Srinivasan
@jrs53
Just hit an interesting error with accumulo auths. I have some features with attribute level visibilities applied, let's call them foo and bar. If I specifically either foo or bar in a cli export, everything seems ok. If I specified them both, my output feature seems to get mangled so there seems to be an odd off by one error with attribute encoding, some initial chars mission etc (sorry, can't easily share data)
102 replies
James Srinivasan
@jrs53
GitHub says 3.1.2 is the latest release; is that correct? https://github.com/locationtech/geomesa/releases/tag/geomesa_2.11-3.1.2
James Hughes
@jnh5y
GitHub is confusing... that's the most recent release.
19 replies
James Srinivasan
@jrs53
Sorry, me again - I assume join index use should be transparent to CLI users? I'm getting no features back if I query for a feature based on an attribute which is join indexed
35 replies
If I do the same with a full index, it is fine
robertovarga14
@robertovarga14:matrix.org
[m]
Hello, is there a way to include user and keytab in the creation of the params with geomesa-hbase 3.2.0?
1 reply
James Hughes
@jnh5y
Nice!
Yeah, any time I've tried to look at a graph of all GeoMesa's dependencies.... it usually has turned into spaghetti like that
there needs to be a tool which can filter graphs a bit more to help show things in a sensible way
tinvukhac
@tinvukhac
Hi all, when I run a Spark example, I always got this error: Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class org.locationtech.geomesa.index.view.MergedDataStoreViewFactory$
at org.locationtech.geomesa.index.view.MergedDataStoreViewFactory.getDisplayName(MergedDataStoreViewFactory.scala:87) . Can anyone please help me?
2 replies
tinvukhac
@tinvukhac
Does anyone have problem with spatial join performance? I ran spatial join successfully with small input (i.e. 100 entries for each input), but when I'm joining 2 large datasets (2 millions and 1 million entries) with this command: val joinDF = df1.join(df2, st_intersects($"geom1", $"geom2")). It has been run like forever: the job stage is showing "[Stage 4:> (0 + 16) / 16]" and does not change. My Spark cluster is pretty strong (12 nodes x 12 cores).
3 replies
ChangeSub
@ChangeSub
I want to ask, does geomesa2.11-3.1.2 support hadoop3.12?
8 replies
Kunwar Shaanjeet Singh Grover
@Groverkss

Hi! I was trying to use geomesa using pyspark. But i seem to be getting some weird error:

java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/expressions/PredicateHelper$class

Code is almost same as the geomesa tutorial:

import geomesa_pyspark

conf = geomesa_pyspark.configure(
     packages=['geomesa_pyspark','pytz'],
     spark_home='/usr/local/apps/spark/') \
    .setAppName('geomesa-test') \
    .set("spark.executor.memory", "1g") \
    .set("spark.executor.cores", "2") \
    .set("spark.cores.max", "4") \
    .setMaster("spark://172.16.0.10:7077")

from pyspark.sql import SparkSession

spark = ( SparkSession
    .builder
    .config(conf=conf)
    .enableHiveSupport()
    .getOrCreate()
)

feature = "test1"
params = {
    "accumulo.instance.id": "accumulo",
    "accumulo.zookeepers": "node02:2181",
    "accumulo.user": "root",
    "accumulo.password": "csg123",
    "accumule.catalog": "geomesa.test",
}

df = ( spark
    .read
    .format("geomesa")
    .options(**params)
    .option("geomesa.feature", feature)
    .load()
)

I don't know much java, but i tried checking the spark-catalyst package and it does contain this class (i think?)

It contains the class:

org/apache/spark/sql/catalyst/expressions/PredicateHelper.class

Does the $ here make a difference?

6 replies
JB-data
@JB-data

Dear experts,
I am trying to add one field to my existing processing to write results in spark to geomesa-hbase.
The field is an array containing strings.
Inspired by the example-csv-complex that ships with geomesa,
I added to a working .conf file for the sft in the fields section:

{
            "name" : "extrafield",
            "transform" :  "parseList('String', $7)"
}

for the attributes:

{
            "index": "true"
            "name" : "extrafield",
            "type" : "List[String]"
}

and then load this using the create-schema command
My df including the new extrafield (the field looks like [hello1, hello2]) shows is type array <string>.
When I try to modify my working pyspark job to write the df including the new extrafield referring to the loaded .conf file:
dftotal2.write.format("geomesa").options(**params_out).option("geomesa.feature", "conf_extrafield" ).save()
this throws:
scala.MatchError: ArrayType(StringType,false) (of class org.apache.spark.sql.types.ArrayType) at org.locationtech.geomesa.spark.SparkUtils$$anonfun$createFeatureType$2.apply(SparkUtils.scala:90) at org.locationtech.geomesa.spark.SparkUtils$$anonfun$createFeatureType$2.apply(SparkUtils.scala:89) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) ...

20 replies
JB-data
@JB-data
I notice the out-of-the box example works:
geomesa-hbase ingest -c example -C example-csv-complex -s example-csv-complex example.csv, so I guess workaround could be to store my output as csv (stringify my array), and load it using the command tools.
I notice that my cql query like
friends in ('Will') works and returns me the record where friends="Will, Mark, Suzon", as desired, so that is hopeful!
federicaesse
@federicaesse
Hi all, i am new in geomesa. I tried to ingest in geomesa-fs a parquet file that contain the coordinate (lat e long column) of 3 points using this command line : "geomesa-fs ingest -p /temp/my folder/ -e parquet --converter test_parquet --partition-scheme z2-2bit myfile.parquet.gzip -f test_parquet". It seem that it was ingested correctly beacuse when i launched the command "geomesa-fs export -p /temp/myfolder -f test_parquet" the results was my table with a new colum "geometry".
After that i create a geomesa filesystem store in geoserver and i have published the layer parquet.
Despite the layer settings give me the right coordinate system, the right extent and fields of the parquet file, the layer preview give me a box completetly blank.
Have you got any suggestion? why in geoserver i am not able to see the layer?
27 replies
Eren Cankurtaran
@ieuD
Hi. We're testing geomesa-kafka with geoserver but we have an issue about geoserver consumer side. Even we define the broker and zookeeper IP's geoserver consumer tries to connect localhost:9092
Here's the output of geoserver
4:12:55.654 [pool-26-thread-1] DEBUG o.a.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=f2f1fdcf-884e-4b73-a2be-95f64ecbbe11] Connection with localhost/127.0.0.1 disconnected
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_292]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[na:1.8.0_292]
    at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50) ~[kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:173) ~[kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:515) [kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.common.network.Selector.poll(Selector.java:467) [kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535) [kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) [kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) [kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:227) [kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161) [kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:241) [kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:316) [kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1214) [kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1179) [kafka-clients-2.1.1.jar:na]
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1164) [kafka-clients-2.1.1.jar:na]
    at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292]
    at org.locationtech.geomesa.kafka.KafkaConsumerVersions$$anonfun$fromDuration$1$1$$anonfun$applyOrElse$1.apply(KafkaConsumerVersions.scala:53) [geomesa-kafka-utils_2.11-3.1.2.jar:na]
    at org.locationtech.geomesa.kafka.KafkaConsumerVersions$$anonfun$fromDuration$1$1$$anonfun$applyOrElse$1.apply(KafkaConsumerVersions.scala:53) [geomesa-kafka-utils_2.11-3.1.2.jar:na]
    at org.locationtech.geomesa.kafka.KafkaConsumerVersions$.poll(KafkaConsumerVersions.scala:25) [geomesa-kafka-utils_2.11-3.1.2.jar:na]
    at org.locationtech.geomesa.kafka.consumer.ThreadedConsumer$ConsumerRunnable.run(ThreadedConsumer.scala:82) [geomesa-kafka-utils_2.11-3.1.2.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_292]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_292]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_292]
14:12:55.655 [pool-26-thread-1] DEBUG o.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1,

Screenshot from geoserver kafka store config
https://ibb.co/NxCj8Sg

Hi. We're testing geomesa-kafka with geoserver but we have an issue about geoserver consumer side. Even we define the broker and zookeeper IP's geoserver consumer tries to connect localhost:9092
Here's the output of geoserver

9 replies
federicaesse
@federicaesse
hi all, i have another problem. I would like to connect my geomesa-fs to hdfs store (i have hdp 3.1.4). In the fs-path i use: hdfs://namenode:8020/path but geoserver return me this error: "Error creating data store, check the parameters. Error message: org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto cannot be cast to com.google.protobuf.Message". What wrong?
1 reply
JB-data
@JB-data
Dear experts,
I have a huge dataset with point geometries.
Further I have a dataframe with a list of couple thousand polygons DFPOLYGON.
Processing happens in pyspark, so I am loading everything in dataframes and views so I can use sqlContext.sql("<write sql>").
I would like to to get for each poins, all the polygons in which it is contained.
Is the best approach to write a udf based on the st_contains function, and for each point use st_contains and loop through DFPOLYGON?
17 replies
James Hughes
@jnh5y
@JB-data it may be worth mentioning that @geoHeil has used GeoMesa with GeoSpark (before it was Apache Sedona) before. He may have some suggestions, etc.
9 replies
federicaesse
@federicaesse
hi all, any of you have ingested parquet file in geomesa-fs that have multipolygon geometry? which partition schema have you used?
1 reply