Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Adam J.
    @jakon89
    hi guys
    Scott Mansfield
    @ScottMansfield
    hey @jakon89 how ya doing
    Akbar Ahmed
    @akbarahmed
    @ScottMansfield Do you have any rough metrics on a. requests/second handled by Rend (and on what hardware), and b. P95 and P99 latency experienced at the client?
    Scott Mansfield
    @ScottMansfield
    req/sec varies by the type of request and box
    so for noop requests, which are super lightweight
    on the type 3 box on packet.net it can get roughly 3.6 million requests per second
    for more real world workloads it's far lower
    we ran a test with a single i2.xlarge instance using our full setup
    that includes rend, memcached, and our L2 disk-backed solution
    we ran a test with 1k writes per second, 3k reads per second and 10% misses
    it had a 99th percentile read latency of 1.7 ms
    Akbar Ahmed
    @akbarahmed
    @ScottMansfield I'm very realistic about metrics. :smile: I do know that a normal instance on EC2 will likely see saturation of the network long before being CPU or IO bound for a solution like Rend.
    Scott Mansfield
    @ScottMansfield
    I'm getting our real production numbers for a recent deployment
    Akbar Ahmed
    @akbarahmed
    In terms of CPU, the processes running (beyond OS) are 1. rend, 2. memcached, 3. Mnemonic, and 4. I presume some type of sidecar for service discovery / telemetry.
    Is this about right?
    Scott Mansfield
    @ScottMansfield
    that's correct
    Akbar Ahmed
    @akbarahmed
    I'm going to ask some really stupid questions, so brace yourself...
    Scott Mansfield
    @ScottMansfield
    average client-side latency is roughly half a millisecond
    Akbar Ahmed
    @akbarahmed
    That's good.
    Scott Mansfield
    @ScottMansfield
    95th, 1 ms
    99th, 3 ms
    Akbar Ahmed
    @akbarahmed
    With these metrics have you hit any resource limits? Namely, have you topped out CPU or IO, etc?
    Scott Mansfield
    @ScottMansfield
    it's always I/O for real-world workloads
    the packet.net box was saturated on CPU because it has a 20 Gbps pipe
    Akbar Ahmed
    @akbarahmed
    to be clear, IO in this case we're referring to being network bound, right?
    Another question, have you tried testing with a VM with more vCPUs to allow Rend's, memcached's and mnemonic's multithreading to utilize more cores?
    I totally forgot to ask. Was this classic or VPC?
    Scott Mansfield
    @ScottMansfield
    sorry, stepped away
    yes, network bound
    in EC2 classic, it's sometimes bandwidth and sometimes packet processing power
    in VPC the packets processing ability is much higher (generally)
    but the bandwidth is the same, so we hit bandwidth
    Akbar Ahmed
    @akbarahmed
    It seems you can (easily) get higher requests/second on VPC. That's obvious. What's the request size? 1k?
    Scott Mansfield
    @ScottMansfield
    on boxes with more cores, we can do more work, but it's still hard even with artificial benchmarks to get to CPU limits before bandwidth
    the noop artificial test is 24 bytes for the header
    real world is like 20k
    Akbar Ahmed
    @akbarahmed
    I think I get the numbers now. If we drop the 20k packet size to 1k we should see enough CPU/Disk headroom to achieve 20x the numbers in order to saturate the network.
    Anyway, the numbers look good. Just asking pointless questions.
    Is there a timeline for when mnenomic will be OSS'd?
    Scott Mansfield
    @ScottMansfield
    not really right now
    hopefully by the end of the year, but anything that speaks memcached can be L2 as well
    Akbar Ahmed
    @akbarahmed
    Thanks.
    Enrico Candino
    @enrichman
    hi all! the last message seems to be a bit old :smile:
    just wanted to know if this project is still maintained