Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 07:19
    DanielTimLee starred oetiker/rrdtool-1.x
  • Jan 28 2019 20:24
    c72578 opened #950
  • Jan 28 2019 10:43

    oetiker on master

    Fix _errno inconsistent dll lin… (compare)

  • Jan 28 2019 10:43
    oetiker closed #949
  • Jan 28 2019 09:29
    c72578 commented #949
  • Jan 27 2019 00:12
    c72578 opened #949
  • Jan 25 2019 14:25
    c72578 opened #948
  • Jan 25 2019 12:50
    oetiker commented #947
  • Jan 25 2019 12:50

    oetiker on master

    Fix rc compiler redefinition wa… (compare)

  • Jan 25 2019 12:50
    oetiker closed #947
  • Jan 25 2019 10:11
    c72578 opened #947
  • Jan 25 2019 09:59

    oetiker on master

    Initialize local variable junk … (compare)

  • Jan 25 2019 09:59
    oetiker closed #946
  • Jan 24 2019 16:46
    c72578 opened #946
  • Jan 24 2019 10:52

    oetiker on master

    Include winsdkver.h for _WIN32_… (compare)

  • Jan 24 2019 10:52
    oetiker closed #945
  • Jan 24 2019 10:52
    oetiker commented #945
  • Jan 24 2019 09:05
    loose11 starred oetiker/rrdtool-1.x
  • Jan 23 2019 10:48
    wukuan405 starred oetiker/rrdtool-1.x
  • Jan 22 2019 15:14
Tobias Oetiker
@oetiker
what some do, is to modify the write back timer ... in that way dirty pages do not get written back to disk so quickly thus making the updates much cheeper as long as all active blocks are in ram ...
andrewchambers
@andrewchambers
yeah, i think its quite long on linux already
maybe in the order of a minute or more, i need to double check
Tobias Oetiker
@oetiker
but then you have the problem that stuff is not stored for a long time ... bringing the necessety of writing some sort of journal
in linux I think it is 30s currently
by raising it to 30 minutes performance will skyrocket
Fabien Wernli
@faxm0dem
FWIW what made us move away from RRDTool at the datacenter is scale
we really needed something distributed
andrewchambers
@andrewchambers
well, there are ways to distribute filesystem access and rrdtool already has support for ceph afaik
it mentions librados which is a distributed block store
Fabien Wernli
@faxm0dem
of course, that's what we used to do back in the days, but I like application-based distribution
makes things much more decoupled
andrewchambers
@andrewchambers
hmm
Fabien Wernli
@faxm0dem
what we ended up doing is reimplement RRDTool's RRA concept and use Elasticsearch as underlying storage
In effect it's still not fully applicative, but it's an improvement over what we had before
andrewchambers
@andrewchambers
yeah, I thought of rrdtool as a lower level tool, collectd is the closest thing to a coordinating system
I have used prometheus a bit in the past and did not like having to run an http endpoint for each set of metrics
it can quickly get out of hand
Fabien Wernli
@faxm0dem
yeah same I prefer push for this kind of thing
so what we do is collectd>riemann>elasticsearch
and the rrdtool-like behaviour is implemented in riemann
andrewchambers
@andrewchambers
I have not worked with riemann or elasticsearch before
so don't have too much of an opinion on that, it sounds interesting
andrewchambers
@andrewchambers
w.r.t. rrdtool, i saw things like this filesystem http://www.betrfs.org
to quote it
The advantage of a B ε tree is that it can also ingest small, random writes 1-2 orders of magnitude faster than B-trees and other standard on-disk data structures.
I would be curious if something like this would translate for rrdtool
but i guess it does not address the fundamental problem of a single metric write needing to modify a disk page
Fabien Wernli
@faxm0dem
All I can say is that it's reassuring that smart folk who think about important low-level problems like this still exist and are hard at work
nothing worse than a bloated-software-only-world
Tobias Oetiker
@oetiker
yep that diskblock problem is hard
andrewchambers
@andrewchambers
@oetiker So I just had an idea w.r.t. that
perhaps if there was a shared buffer file/directory
where all updates first go there
then once enough have accumulated they can sent out to each individual rrd
so to read an rrd you might need the rrd file and the buffer path
if you want the up to date data
I guess its a similar concept to rrdcached
but letting the OS be the cache daemon by using a shared file
it could just be an append only file that compressed updates get written to
then periodically it plays the log to rrd files and truncates itself
andrewchambers
@andrewchambers
I guess it is quite similar to rrdcached
eh nvm :P I guess its very similar
I was just trying to do it without a process
just use the OS page cache to do the buffering
Tobias Oetiker
@oetiker
yes, a sort of journal file
rrdcached flushes everything when you try to read which is bad for performance :)
how about a design where the rrdd is keeping a 4k buffer for each rrd in memory and the journal would only be written for recovery purposes
Tobias Oetiker
@oetiker
maybe the format of the individual rrd file could also be optimized ...