These are chat archives for linkedin/pinot

11th
Feb 2016
Hisham Mardam-Bey
@mardambey
Feb 11 2016 01:22
@jfim cheers man, good to be back there! (=
I'm seeing a lot more on the screen now.
(with your latest commit)
including exceptions!!!
YAY!~
I should be able to fix now
hehe
Hisham Mardam-Bey
@mardambey
Feb 11 2016 01:44
kafka.common.KafkaException: fetching topic metadata for topics [Set(facebook_ads-realtime)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,9092))] failed
Which explains why it works outside the container, and not inside (not running on localhost in there, has to use the proper container name)
Jean-François Im
@jfim
Feb 11 2016 02:05
nice
Hisham Mardam-Bey
@mardambey
Feb 11 2016 02:10
I didn't look yet, but I wonder where that localhost is coming from, some config, or the table def.
pinot_1 | Property metadata.broker.list is overridden to localhost:9092
Hisham Mardam-Bey
@mardambey
Feb 11 2016 02:15
Not part of the table def.
Hisham Mardam-Bey
@mardambey
Feb 11 2016 02:29
Can't find that one yet.
Getting a bit late too, I'll continue tomorrow.
Hisham Mardam-Bey
@mardambey
Feb 11 2016 14:40
@jfim if you have any tips on where this is being set (above), lemme know.
Jean-François Im
@jfim
Feb 11 2016 19:29
Nope
Is 9092 used by Kafka?
Hisham Mardam-Bey
@mardambey
Feb 11 2016 19:33
@jfim Yup.
It's the Kafka broker port.
Jean-François Im
@jfim
Feb 11 2016 20:11
I think it's kafka not doing name resolution properly to determine what is the external dns name for localhost
we have a similar issue
I can't run the realtime tests locally when connected to the linkedin vpn because it breaks localhost resolution
kafka just breaks in that case
check if there's a way to specify a hostname for the kafka nodes
Jean-François Im
@jfim
Feb 11 2016 21:10

Why can't my consumers/producers connect to the brokers?

When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in metadata.broker.list in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress. Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip to be registered in ZK by setting the "hostname" property in server.properties. In another rare case where the binding host/port is different from the host/port for client connection, you can set advertised.host.name and advertised.port for client connection.

Hisham Mardam-Bey
@mardambey
Feb 11 2016 22:08
@jfim cheers, I'll try shortly. I've set advertised.host.name, I'll also set hostname, and let you know.
Jean-François Im
@jfim
Feb 11 2016 22:08
Cool
If it works, tell me, I'll fix the integration tests too, I hate the fact that I have to disconnect from vpn all the time :)
Hisham Mardam-Bey
@mardambey
Feb 11 2016 22:09
nod!