Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
James Hughes
@jnh5y
If it is something you can walk with me, let's do that.
Alternatively, if @metasim or @vpipkt can help me out, we can sort out you sending them sufficient beer to cover their effort;) (I mention them since they are in the same town as me:))
8 replies
James Srinivasan
@jrs53
Think I have geomesa_pyspark working with conda pack which ought to make adding extra packages (e.g. sklearn) easier. Will check tomorrow/Monday and hopefully document as PR
dajiang31835008
@dajiang31835008
@dajiang31835008 you need Guava. Have you run any of the install shell scripts that help download the right version of Guava?
Is the installation script executed on the hbase cluster or on the local development machine? In IDEA, through debugging, it is found that the version calling Guava is 11.0.2, and there is guava-11.0.2.jar in the .m2 library
biangbiang66
@biangbiang66
Thanks! Another question is that when injesting a file "file-name.osm.pbf", do I need to add the suffix of ".pbf", the command in geomesa manual is "$ geomesa-accumulo ingest -u USERNAME -c CATALOGNAME -s osm-nodes -C osm-nodes test.osm", which doesn't have a ".pbf" suffix.
Yash Patel
@YashPatel98
Is there anyway I can ingest or load a .h5 file (HDF format file) to geomesa-hbase directly?
James Hughes
@jnh5y
@biangbiang66 you probably need to give the complete filename; try it! (maybe the docs are wrong)
@YashPatel98 what is the data in the HDF file? I think of HDF5 as storing raster data; GeoMesa is focused on vector data.
given that, you'd probably have to write some code to convert the data into SimpleFeatures
@biangbiang66 your .m2 repository is not going to be on the GeoMesa tools classpath. Try copying that jar into the lib/ directory where you've unpacked the GeoMesa tools
biangbiang66
@biangbiang66
@jnh5y Sincerely thanks! I've copy all the jars of program "geomesa-osm-converter" by executing the commands in the manual. but there are still an error of ".NoClassDefFoundError: com/google/protobuf/ProtocolStringList". So have I missed some jars?
Yash Patel
@YashPatel98

@jnh5y Yes, HDF5 has raster data. I converted my .h5 file to .csv file and tried ingesting into geomesa-hbase in distributed mode.But getting a YARN CONTAINER error:

org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Recovery is enabled. Will try to recover from previous life on best effort basis.
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://localhost:9000]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at hdfs://localhost:9000/tmp/hadoop-yarn/staging/yash/.staging/job_.....jhist
WARN [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job history, aborting recovery
java.io.FileNotFoundException: File does not exist: /tmp/hadoop-yarn/staging/yash/.staging/job....jhist
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:72)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:62)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1819)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:692)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:381)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2489)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

..........

James Hughes
@jnh5y
@biangbiang66 ah, that sounds like you are missing a protobuf jar. (or you may have more than one on the classpath)
@YashPatel98 hmm... are you running your MapReduce cluster "locally"? Try with distributed mode first.
When you are trying out distributed mode, you'll need to get some of the Hadoop configuration files in the right spot (since they are not that is why you are having trouble)
biangbiang66
@biangbiang66

Thanks! The missing class can be found in hbase-shaded-protobuf-2.2.1.jar and protobuf-java-2.5.0.jar, which have been included in geomesa/lib. I thought it would be dependency confilction, and I,ve tried to remove anyone of them, but another NoClassDefFoundError appears.
when replacing protobuf-java-2.5.0.jar with protobuf-java-2.6.0.jar in the directory of "geomesa/lib/", osm -nodes can be ingested in local mode successfully in local mode! But when I execute mapreduce ingest in distributed mode, another error occured:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.yarn.proto.YarnProtos$LocalResourceProto.hashLong(J)I
at org.apache.hadoop.yarn.proto.YarnProtos$LocalResourceProto.hashCode(YarnProtos.java:12895)
at org.apache.hadoop.yarn.api.records.impl.pb.LocalResourcePBImpl.hashCode(LocalResourcePBImpl.java:62)
at java.util.HashMap.hash(HashMap.java:339)
at java.util.HashMap.put(HashMap.java:612)

I did a little google search, this is what I found :"This error indicates that the jar file used when the code was compiled is different from the jar file used when it was run. You need to make sure that you're using exactly the same version of the protobuf jar as the code was compiled against. " so should I change the version of the jar in other position?

6 replies
@jnh5y
dajiang31835008
@dajiang31835008
geomesa_hbase-3.1.0, call "datastore.createSchema(sft)" ,Encountered an log"Table zztest_tf_5fvehicle_5fhistory_5fpartition_5ftest2_z3_geom_tm_v6 does not exist". and The program stays at“while(!admin.isTableAvailable(table) && stop.forall(new NamelessClass_7())) {
Thread.sleep(1000L);
}
James Hughes
@jnh5y
@biangbiang66 if your deployment is using a different version of Protobuf than what GeoMesa is compiled against, that could explain the errors you are seeing. If that's the case, you may have to rebuild GeoMesa based on your deployed versions of various libraries.
@dajiang31835008 when you called createSchema, tables in HBase should have been created. Did you see any tables created?
If not, maybe you do not have access to create the tables?
dajiang31835008
@dajiang31835008
@jnh5y Thanks for your reply!After called createScheme,I can see zztest_tf_5fvehicle_5fhistory_5fpartition_5ftest2_z3_geom_tm_v6 table created in Hbase .But i can not insert data into the table.The Hbase log out2021-01-12 22:21:50,148 INFO org.apache.hadoop.hbase.master.assignment.AssignProcedure: Starting pid=363, ppid=360, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=zztest_tf_5fvehicle_5fhistory_5fpartition_5ftest3_z3_geom_tm_v6, region=9648170b519538461797cc3f48b6eae8; rit=OFFLINE, location=null; forceNewPlan=true, retain=false
2021-01-12 22:21:50,151 INFO org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch pid=527, ppid=524, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=zztest_tf_5fvehicle_5fhistory_5fpartition_5ftest5_z3_geom_tm_v6, region=10adad70527be469f54c95e66713dda9
2021-01-12 22:21:50,151 INFO org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch pid=194, ppid=191, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=zztest_tf_5fvehicle_5fhistory_5fpartition_5ftest_z3_geom_tm_v6, region=3223008c745fae3187c864e6ae040626
2021-01-12 22:21:50,151 INFO org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch pid=190, ppid=186, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=zztest_tf_5fvehicle_5fhistory_5fpartition_5ftempzz_z3_geom_tm_v6, region=51b233d02f6eef7c95b379ec4fbae507
2021-01-12 22:21:50,151 INFO org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch pid=177, ppid=174, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=sdbtest1_tf_5fvehicle_5fhistory_5fpartition_5ftemp5_attr_v5, region=6662ec83cc0230af7bea5120500163ec
2021-01-12 22:21:50,151 INFO org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch pid=182, ppid=179, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=sdbtest_gdelt_2dquickstart_z3_geom_dtg_v6, region=a50634f8aa325e520f9cb45cb509712b
2021-01-12 22:21:50,151 INFO org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch pid=526, ppid=524, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=zztest_tf_5fvehicle_5fhistory_5fpartition_5ftest5_z3_geom_tm_v6, region=11e8660e78816e5d15949c8e3a8a984a
2021-01-12 22:21:50,151 INFO org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch pid=189, ppid=186, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=zztest_tf_5fvehicle_5fhistory_5fpartition_5ftempzz_z3_geom_tm_v6, region=79e0279fe2faac6c1a423384b29a7d8d
2021-01-12 22:21:50,152 INFO org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch pid=195, ppid=191, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=zztest_tf_5fvehicle_5fhistory_5fpartition_5ftest_z3_geom_tm_v6, region=750cab63387bccd2ebbe560b3ebd649f
James Hughes
@jnh5y
odd.... maybe HBase is being too slow to bring the tables online? (I do not know why that'd be happening or what the underlying issue is here. I think there may be something going on with your HBase instance:()
dajiang31835008
@dajiang31835008
@jnh5y when write data to the table ,thow exception:Writing test data
java.lang.IllegalStateException: Expected a single table but got:
java.lang.IllegalStateException: Expected a single table but got:
at org.locationtech.geomesa.hbase.data.HBaseIndexAdapter$HBaseIndexWriter
KaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲23.apply(HBaseI…: anonfun$23.apply(HBaseIndexAdapter.scala:575)
    at org.locationtech.geomesa.hbase.data.HBaseIndexAdapter$HBaseIndexWriter
anonfun$23.apply(HBaseIndexAdapter.scala:572)
at scala.collection.TraversableLike
KaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲map$1.apply(Tra…: anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike
anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at org.locationtech.geomesa.hbase.data.HBaseIndexAdapter$HBaseIndexWriter.<init>(HBaseIndexAdapter.scala:572)
at org.locationtech.geomesa.hbase.data.HBaseIndexAdapter.createWriter(HBaseIndexAdapter.scala:349)
at org.locationtech.geomesa.hbase.data.HBaseIndexAdapter.createWriter(HBaseIndexAdapter.scala:62)
at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$TableFeatureWriter.<init>(GeoMesaFeatureWriter.scala:147)
at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$$anon$3.<init>(GeoMesaFeatureWriter.scala:108)
at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$.apply(GeoMesaFeatureWriter.scala:108)
at org.locationtech.geomesa.index.geotools.GeoMesaDataStore.getFeatureWriter(GeoMesaDataStore.scala:427)
at org.locationtech.geomesa.index.geotools.GeoMesaDataStore.getFeatureWriterAppend(GeoMesaDataStore.scala:415)
at org.locationtech.geomesa.index.geotools.GeoMesaDataStore.getFeatureWriterAppend(GeoMesaDataStore.scala:49)
at org.geomesa.data.util.GeoMesaQuickStart.writeFeatures(GeoMesaQuickStart.java:349)
at org.geomesa.data.util.GeoMesaQuickStart$ZZWorkerHandler.launchSyncData(GeoMesaQuickStart.java:323)
at org.geomesa.data.util.GeoMesaQuickStart$ZZWorkerHandler.run(GeoMesaQuickStart.java:274)
at java.lang.Thread.run(Thread.java:745)
Emilio
@elahrvivaz
@dajiang31835008 can you create tables and scan them outside of geomesa, i.e. through the hbase shell?
it seems like your cluster is messed up
dajiang31835008
@dajiang31835008
@elahrvivaz Maybe,I try it, thanks
Yash Patel
@YashPatel98

@YashPatel98 hmm... are you running your MapReduce cluster "locally"? Try with distributed mode first.

@jnh5y No.Local ingestion runs successfully. The errors are while running ingestion in distributed mode.

Yash Patel
@YashPatel98
Moreover, I want to know the difference between running geomesa-hbase data ingestion locally vs running in distributed mode.Both uses HBase underneath to store and distribute data into different tables...right?. HBase itself can run in distributed mode with Hadoop. I may be in misconception though....please enlighten me.
Emilio
@elahrvivaz
running ingest is separate from any hbase setup you have. running a local ingest just uses your current machine, so it's limited in threads, memory and I/O. running distributed submits the ingestion as a map/reduce job, so it's parallelized across multiple machines. the end result is the same - data in hbase
biangbiang66
@biangbiang66
image.png
image.png
When rendering WMS of geomesa osm-nodes with geoserver, I encountered the above error
Emilio
@elahrvivaz
did you try increasing hbase.scanner.timeout like it says?
James Hughes
@jnh5y
@biangbiang66 how large of an area are you scanning for? Also, I haven't seen that error before. Can you remind me which version of HBase you are using?
biangbiang66
@biangbiang66
@elahrvivaz I have added the configuration showed in the picture in Hbase-site.xml. But it didn't work.

<property>
<name>hbase.client.scanner.timeout.period</name>
<value>900000</value> <!-- 900 000, 15 minutes -->
</property>

<property>
<name>hbase.rpc.timeout</name>
<value>900000</value> <!-- 15 minutes -->
</property>

<property>
<name>hbase.regionserver.lease.period</name>
<value>1200000</value> <!-- 15 minutes -->
</property>
</configuration>

biangbiang66
@biangbiang66
Thanks!
@jnh5y The data I scanned is the osm-nodes of south-amarica, and I'm using hbase2.2.3.
The osm-nodes of antarctica (with much smaller size) can be rendered successfully.
@elahrvivaz The error must be caused by the unproper configuration of hbase.scanner. But I have no idea about how to adjust it.
biangbiang66
@biangbiang66
Maybe I should set a larger value of Hbase configuration?
Emilio
@elahrvivaz
did you say what version of geomesa you're using?
biangbiang66
@biangbiang66
I'm using geomesa-hbase_2.11-3.0.0 @elahrvivaz
Emilio
@elahrvivaz
i'm not really sure why you'd be having errors. could you get the explain plan from your query that is failing? that would give us more info
https://www.geomesa.org/documentation/stable/user/geoserver.html#logging-explain-query-planning
Emilio
@elahrvivaz
@pomadchin thanks for the timely +1's!! could you also tag https://dev.eclipse.org/ipzilla/show_bug.cgi?id=22966? :D
Grigory
@pomadchin
@elahrvivaz :+1: I think I missed it
np; always happy to +1 cqs :D
Emilio
@elahrvivaz
tyvm!
babatush
@babatush

Hello everyone, I am using cassandra with geomesa, after updating geomesa-cassandra-datastore dependency to 3.1.1 I keep getting the following runtime exception:

java.lang.NoClassDefFoundError: com/google/common/util/concurrent/FutureFallback
at com.datastax.driver.core.GuavaCompatibility.selectImplementation(GuavaCompatibility.java:136)
at com.datastax.driver.core.GuavaCompatibility.<clinit>(GuavaCompatibility.java:52)
at com.datastax.driver.core.Cluster.<clinit>(Cluster.java:68)
at org.locationtech.geomesa.cassandra.data.CassandraDataStoreFactory.createDataStore(CassandraDataStoreFactory.scala:55)
at org.locationtech.geomesa.cassandra.data.CassandraDataStoreFactory.createDataStore(CassandraDataStoreFactory.scala:27)
at org.geotools.data.DataAccessFinder.getDataStore(DataAccessFinder.java:119)
at org.geotools.data.DataStoreFinder.getDataStore(DataStoreFinder.java:69)
Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.FutureFallback
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at sbt.internal.ManagedClassLoader.findClass(ManagedClassLoader.java:102)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

2 replies
I added guava dependency but I still get the same exception
anubhav tarar
@anubhav100_gitlab
Hi , i have a query like select * from table where name='anubhav' and age>20 where both name and age are secondary index in geomesa will this query leverge the power of secondary index or will it do a full scana and then filter
1 reply