@earthquakesan oh, I see. I wasn't aware of that. I did try to change a configuration option related to hive query engine, but changing it to spark or tez didn't work very well, it crashed. I should definitely learn more about YARN. I am sure that you are already aware of this, but just for the record, it would really be great to have YARN and "stronger" MapReduce engine in the bde2020 setup.
In BDE we also created Spark images, it is possible to use those with Hive. We don't have Tez.
Hi @waterponey,
I have played around with Spark/YARN integration and have a working setup (not pushed to BDE repos). If you can describe your use case (i.e. why do you need YARN/Mesos) in case of deploying Spark in docker, that would be helpful. In my case, I needed YARN for history manager and that's pretty much it.
Well you can deploy a history manager for spark without yarn. It's just that in terms of ressource usage I'm not sure how you can free resources in this setup. Also, if you're doing streaming do you split the ressources at the swarm level or at the standalone spark level (meaning do you spawn another standalone cluster ?)
in our workflows however, we deploy and destroy the frameworks as necessary
e.g. if it's just a batch job, can spawn cluster with necessary resources, run the job and then destroy it
To have several apps run in parallel, you will need to use YARN.
I did not understand your question regarding streaming, can you elaborate?
I was wondering how it would work if you wanted to have several concurrent streaming application. One possibility would be to statically declare one spark cluster per app and define the resources that swarm would allow to each spark cluster at startup. The other would be to define one specific "streaming spark cluster" in swarm and let the spark ressource manager do the allocation to each streaming app.
I mean you deploy docker container containing a spark cluster on a swarm cluster, or am I missing something ?
The proper setup would be as follows:
- Deploy Spark cluster with YARN/Mesos and restrict CPU/Memory usage in Swarm + configure resource managers to see that limit (unfortunately you need to do it manually/ansible). Let's say you give it 64 cores and 256G of RAM.
- When deploying spark apps inside your cluster restrict CPU usage and memory per application (e.g. number of executors, cores per executor, memory per executor). If you want to run 2 applications, then you will do the setup, where each app consumes 32 cores and 128G of RAM.
that's correct, I am clarifying that you are not using swarm as a general term for a cluster %)
that's just another container
Hey I'm sorry, I'm trying to run https://github.com/big-data-europe/docker-hive and the data nodes are failing with
Datanode denied communication with namenode because hostname cannot be resolved (ip=10.0.0.10, hostname=10.0.0.10): DatanodeRegistration(0.0.0.0:50010, datanodeUuid=d64d014a-4467-4065-95e6-596590148f75, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-ab7b488d-d3c3-470d-aea0-c6e6ac6708b9;nsid=2064164277;c=0)
If you have any clarity I'd appreciate it!
?
when running docker pull bde2020/hive I face this problem:
hive_1 | mkdir: Call From 57c424f72c21/172.22.0.2 to 57c424f72c21:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
hive_1 | mkdir: Call From 57c424f72c21/172.22.0.2 to 57c424f72c21:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
hive_1 | chmod: Call From 57c424f72c21/172.22.0.2 to 57c424f72c21:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
hive_1 | chmod: Call From 57c424f72c21/172.22.0.2 to 57c424f72c21:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
and beeline cannot join to localhost:10000
➜ epam-qa-metrics git:(master) ✗ docker-compose exec hive bash
root@57c424f72c21:/opt# /opt/hive/bin/beeline -u jdbc:hive2://localhost:10000
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://localhost:10000
18/11/14 13:11:30 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
Beeline version 2.3.2 by Apache Hive
beeline> CREATE TABLE pokes (foo INT, bar STRING);
No current connection
hive_1 | mkdir: Call From 57c424f72c21/172.22.0.2 to 57c424f72c21:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
hive_1 | mkdir: Call From 57c424f72c21/172.22.0.2 to 57c424f72c21:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
hive_1 | chmod: Call From 57c424f72c21/172.22.0.2 to 57c424f72c21:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
hive_1 | chmod: Call From 57c424f72c21/172.22.0.2 to 57c424f72c21:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
and beeline cannot join to localhost:10000
➜ epam-qa-metrics git:(master) ✗ docker-compose exec hive bash
root@57c424f72c21:/opt# /opt/hive/bin/beeline -u jdbc:hive2://localhost:10000
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://localhost:10000
18/11/14 13:11:30 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
Beeline version 2.3.2 by Apache Hive
beeline> CREATE TABLE pokes (foo INT, bar STRING);
No current connection
this is my docker version:
➜ ~ docker -v
Docker version 18.06.1-ce, build e68fc7a
➜ ~ docker -v
Docker version 18.06.1-ce, build e68fc7a
can anyone help please?
The issue is resolved, but I think it may be worth to mention on bde2020/hive that this is just part of the whole project, which can be found on https://github.com/big-data-europe/docker-hive
Hi guys! For anyone interested in learning space-efficient data structures and fast algorithms that are extremely useful in modern Big Data applications, take a look at my recently published book "Probabilistic Data Structures and Algorithms for Big Data Applications" (ISBN: 978-3748190486). In this book, you can find algorithms and data structures for Membership querying (Bloom filter, Counting Bloom filter, Quotient filter, Cuckoo filter), Cardinality (Linear counting, probabilistic counting, LogLog, HyperLogLog, HyperLogLog++), Frequency (Majority algorithm, Frequent, Count Sketch, Count-Min Sketch), Rank (Random sampling, q-digest, t-digest), and Similarity (LSH, MinHash, SimHash). Check at Amazon or the book's webpage
It appeared when I tried to write data into remote hdfs.
any ideas?
just should I expose the port?
@ratasxy_twitter ^
I'm also running into problems with that repo. How should I connect a client using the java API?
Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "localhost");
config.set("hbase.zookeeper.property.clientPort", "2181");
HBaseAdmin.checkHBaseAvailable(config);Returns org.apache.hadoop.hbase.MasterNotRunningException: java.net.UnknownHostException: can not resolve hbase-master,16000,1592487871967