These are chat archives for kite-sdk/kite
My application continuously writes to avro file using hive module with partitions. I need to query those using impala. But files are persisted in hdfs only after
writer.close() is called. Before close() is called I only see the temporary file.
writer.sync() didn't work either. What should be done for this?
I tried periodically calling
close(), but ended up creating many avro files, each of very small size, in each partition. If I need to periodically call
close() and initialize new writer, then how often should I call
close() for efficient querying?