tensorflow/sig-jvm

Hi Tensorflow experts, I had a few questions about improving latency.

I am using TF+Java to serve a sequential model in docker, but the latency is a bit higher than I would hope, right now I am only using CPU, and I am wondering how best to improve the latency.

The model is used scores 100 candidates at a time and right now I am using threads to call the SavedModel, each with a single candidate. The Model takes about 2ms on average to score a candidate.

1. Would batching the input and sending all 100 candidates at once to the model have better performance (sending a 100x1 feature instead of a 1x1)? Will the Model use multiple CPUS if they are available?
2. Is adding a GPU to docker the most effective way of decreasing latency?
3. I was reading out TFLite and how that can reduce latency (https://www.tensorflow.org/lite/performance/model_optimization), but it doesn't seem like serving TFLite is supported in JVM unless on Android. Is this true? If it is, is optimizing the model like in https://www.tensorflow.org/model_optimization the best option to improve performance?

Also, my features are exclusively doubles, ints, and vectors of ints, which I convert to Tensors like:

TInt64 createInt64Tensor(List<? extends Number> value) {
    TInt64 tensor = TInt64.tensorOf(Shape.of(1, value.size()));
    IntStream.range(0, value.size()).forEach(i -> tensor.setLong(value.get(i).longValue(), 0, i));
    return tensor;
}

and

TFloat64 createFloat64Tensor(Double value) {
    TFloat64 tensor = TFloat64.tensorOf(Shape.of(1, 1));
    tensor.setDouble(value, 0, 0);
    return tensor;
}

Is there a more efficient way?

Thank you!!