Happy to see that you can use GR for realtime graphics. It is the developers' choice whether to use Plots.jl, the GR.jl convenience layer or GR.jl directly - there should be minimal overhead. I replaced the GR functions by one(!) (MATLAB-like) convenience function:
plot(fs, abs(fft(buf)[fmin..fmax]), xlim=(fs,fs[end]), ylim=(0,100), xlog=true) - and it turns out that your script is still responsive.