Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jul 14 2017 09:41

    pomadchin on master

    Update README.md (compare)

  • Feb 03 2017 20:24
    pomadchin closed #4
  • Feb 03 2017 20:24
    pomadchin commented #4
  • Feb 03 2017 20:23

    pomadchin on master

    Fix quay.io link Merge pull request #5 from jwal… (compare)

  • Feb 03 2017 20:23
    pomadchin closed #5
  • Feb 03 2017 20:23
    pomadchin commented #5
  • Feb 03 2017 20:10
    jwalgran opened #5
  • Dec 10 2016 21:50
    pomadchin commented #4
  • Dec 10 2016 13:42
    Plippe opened #4
Emilio
@elahrvivaz
you probably need to set up your aws credentials - there are details in the geomesa documentation
Rakesh
@rakeshkr00
@elahrvivaz I was going thru the document at https://www.geomesa.org/documentation/user/cli/filesystems.html#configuration and it suggests to make changes in core-site.xml. I am not sure what value to set for variable HADOOP_MAPRED_HOME. Could you please suggest? I am using EMR 5.18. We have IAM policy for the EMR so wouldn't be setting up credentials. I am using Geomesa 2.2.2 and Hadoop 2.8.4, Accumulo 1.9.2.
Emilio
@elahrvivaz
i think you can use the provided bin/install-hadoop.sh script, and that will download the aws jars as well
you may also need to copy over the core-site.xml from your hdfs cluster, assuming hdfs is not in the docker
i don't think you would need to set HADOOP_MAPRED_HOME in that case
Grigory
@pomadchin
@elahrvivaz hey! we’re spinning up accumulo in dockers on EMR, but how to forward AWS S3 credentials inside? Would it be enough to mount EC2 hypervisor file inside? Or how to you useually forward credentials into the container
Emilio
@elahrvivaz
@pomadchin not sure, sorry. just curious, why do you need s3 credentials?
Grigory
@pomadchin

ah; probably wrote a bit messy;

we want to ingest data from the S3 bucket...

and we used s3://geomesa-docker/bootstrap-geodocker-accumulo.sh to bootstrap the cluster
Emilio
@elahrvivaz
ah, that should use the EMR hadoop right?
Grigory
@pomadchin
Yep
on EMR hadoop picks everything up from the EC2 instance metadata >_>
Emilio
@elahrvivaz
i think that you would only need the credentials where you launch the job, e.g. where you have the geomesa tools installed
you could just unzip them on one of the hadoop nodes instead of inside a docker
Grigory
@pomadchin
Ah… but do we need to do it on EMR?
since we can perform aws s3 commands from the outside of the container
Emilio
@elahrvivaz
no, you could do it anywhere that has access to the cluster
you would probably need to copy down the hadoop *-site.xml files
and probably need to copy/install some aws jars too
Grigory
@pomadchin

ah… so how we do it right now:

after bootstraping, sshing to a master node, going inside the accumulo docker

and performing cli ingest here
Emilio
@elahrvivaz
you could install the cli tools on the master node, outside the docker
Grigory
@pomadchin
do you have a quick tip how to do it?
or a link to docs >_>
Emilio
@elahrvivaz
hadoop master node right?
Grigory
@pomadchin
yep
Emilio
@elahrvivaz
the cli tools should pick up the hadoop conf if you're running it there
https://www.geomesa.org/documentation/user/accumulo/install.html#setting-up-the-accumulo-command-line-tools
you basically just unzip the geomesa tar.gz, and then copy in your accumulo client jars
there's a script bin/install-hadoop-accumulo.sh, but you would want to comment out the hadoop parts if you use that
Grigory
@pomadchin
gotcha
thank you!
Emilio
@elahrvivaz
sure, if things don't work let me know
Grigory
@pomadchin
Thanks :+1: