Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Jonathan Coleman
    @jondcoleman
    seems to be more of a discreet event in my experience
    Andrew Newdigate
    @suprememoocow
    Thanks, thats very helpful @jondcoleman
    Mike Bartlett
    @mydigitalself

    how should we report it when it is happening

    just pop in here and say "it's happening now!" or something like that, then we have a record of the timestamp

    Jonathan Coleman
    @jondcoleman
    kk
    rphares
    @rphares
    not big, but backfilled messages appearing atm
    Jonathan Coleman
    @jondcoleman
    yeah, same thing here
    yep happening in multiple chatrooms now
    rphares
    @rphares
    Potentially finished, 4/5 of the last messages appeared in sequence again :)
    Mike Bartlett
    @mydigitalself
    cool, thanks folks, got a good theory on this
    Andrew Newdigate
    @suprememoocow
    Okay, brilliant. That really helps. Thanks guys
    I've got a very good idea of what's happening here.
    Rex Schrader
    @SaintPeter
    FYI, I've got a very solid internet connection and I'm also coming in from two different locations - my home in Roseville, CA and my work which tunnels through Palo Alto, CA. While I couldn't rule out larger internt congestion, it seems very unlikely that both my connections and ones from India would be experiencing issues. I guess it could be on the Gitter's host end, though.
    Andrew Newdigate
    @suprememoocow
    Hi @SaintPeter , I don't think it's internet connectivity
    Rex Schrader
    @SaintPeter
    I've read the conversation above - if I see it or get reports of it, I'll chime in here.
    Andrew Newdigate
    @suprememoocow
    I've got a fairly good idea of what the problem may be, I'm trying to dig a bit deeper but it also looks like it's definitely related to a server issue we've been seeing over the past few weeks
    Rex Schrader
    @SaintPeter
    I don't know if I should apologise or say "You're welcome" for the FreeCodeCamp Org being such a stress test ;)
    Andrew Newdigate
    @suprememoocow
    It looks like it might be related to read-by notifications from such a big room. If you guys switch to lurk-mode (/lurk) it may help relieve the situation while I craft a better solution
    rphares
    @rphares
    Appears to be lagging again, 2 mentions plus my jobs . Not a bad one, though. every third message-ish
    Andrew Newdigate
    @suprememoocow
    Okay, thanks @rphares I've got a fix fairly close to being done
    rphares
    @rphares
    Oh-- awesome, won't report again, wasn't sure you knew all you needed :blush: sorry for the interruption!
    Andrew Newdigate
    @suprememoocow
    No, please do continue to report for now. It's been really really useful
    rphares
    @rphares
    Will do, thanks again for the support :)
    sorted again, things appear in order
    Abhisek Pattnaik
    @abhisekp
    @suprememoocow just wondering .... is the readBy feature really required?
    or if you have found a fix, then that's perfect :D
    Andrew Newdigate
    @suprememoocow
    @abhisekp there's a few changes we're making. The first, which I've just rolled onto next, means that we send a lot less traffic to all the users when somebody reads a message. The second change, which is a little bit further off, but I'm endevouring to get in ASAP will mean that new users will lurk in a room when they join by default.
    For some rooms, mostly private small channel rooms and one-to-one chats, having readBy can be really useful, but it's completely useless in a room with 40k users :D
    Abhisek Pattnaik
    @abhisekp
    @suprememoocow nice. It will also benefit low bandwidth connections like me i.e. 65 KBps. I just got a gitter reload (using next).
    And about the lurk, could it be based on the amount of traffic a room gets i.e. number of messages/day.
    btw, the lurk hides the unread count which i think is useful for small rooms?
    Andrew Newdigate
    @suprememoocow
    @abhisekp we're probably going to just make the default lurk for everyone but make it much easier to switch (we're also changing the way we describe the different modes)
    With regards to staging, unfortunately you'll still get all the traffic until I roll this change into production, since the majority of readBy messages you receive come from the production servers, which don't have my new patch.
    Abhisek Pattnaik
    @abhisekp
    k. cool :sunglasses:
    Mike Bartlett
    @mydigitalself
    @suprememoocow deployed those changes this morning, give us a shout if you still get some lag issues and we can dig deeper
    Abhisek Pattnaik
    @abhisekp

    @mydigitalself When I scroll previous msgs, why does my chat window jumpy?
    I might be able to reproduce this weird behaviour.

    P.S. I've 2 features enabled - Haley and Caching

    Mike Bartlett
    @mydigitalself
    i'd turn off caching, it's still experimental
    Abhisek Pattnaik
    @abhisekp
    k.
    Mike Bartlett
    @mydigitalself
    but when you scroll up, we're obviously loading more messages from the server and, as you know, if you insert content into a DOM above, it will basically push the content you're seeing on the screen down, and we have to do all sorts of magic to try and correct and stabilise that. it will be more pronounced if there are embedded pens or images in the new content that's loaded.
    Andrew Newdigate
    @suprememoocow
    In addition to what @mydigitalself said above, there is one more change I'd like to roll in, but I think that this first change should make quite a big difference already. We've also added a lot more monitoring, so we'll be able to get some more insight into whats causing the problem.
    Abhisek Pattnaik
    @abhisekp
    ok. cool :sunglasses:
    Thanks @mydigitalself and @suprememoocow
    Mike Bartlett
    @mydigitalself
    how's it been so far today?
    rphares
    @rphares
    So far have not noticed any instances of strong lag, thank you so much!
    (though i haven't been on quite as much as usual)
    @abhisekp ?
    Andrew Newdigate
    @suprememoocow
    Cool. That's really good to hear. We'll be rolling out some more changes tomorrow hopefully once we fully understand the difference this change has made
    Abhisek Pattnaik
    @abhisekp
    I've also not experienced any lags yet but I will wait for more 12-24 hrs before giving any conclusions.
    Andrew Newdigate
    @suprememoocow
    That's really good to hear! We'll also be rolling out some more Redis servers fairly soon, which should alleviate some of the load on the current ones
    rphares
    @rphares
    We had some minuscule hiccups around 17 hours ago and 15 hours ago, respectively, but didn't want to message as they weren't causing real issues (on my end) and only two other users confirmed. I could see a few examples of backfilling messages, but the delays were only a few seconds, and only one or two messages out of sequence in a fast convo. One user, in particular, seemed to notice it quite strongly, where i hadn't at all until he mentioned it. @suprememoocow . From my experience, things seem amazing again :)
    (imo , sorted enough :) . but if the data helps shrug)
    Andrew Newdigate
    @suprememoocow
    This graph shows the effect of the code change we made. It was rolled into production at ~10am yesterday http://cl.ly/3M0E0a0e0H3K.
    Jonathan Coleman
    @jondcoleman
    Wow that's dramatic
    Abhisek Pattnaik
    @abhisekp
    haven't seen any laggy lags yet. :+1: