These are chat archives for praw-dev/praw

9th
Apr 2017
ifasina
@ifasina
Apr 09 2017 01:51
Nevermind, looks like if you pass the comment object that you get from the stream to functions, after awhile something weird happens. @bboe
Bryce Boe
@bboe
Apr 09 2017 03:25
@ifasina can you explain more?
Perhaps provide an as-simple-as-possible example that demonstrates the issue you're experiencing.
ifasina
@ifasina
Apr 09 2017 16:41
for comment in reddit.subreddit('sub').stream.comments(): myFunc(comment) #this causes issues eventually @bboe
Bryce Boe
@bboe
Apr 09 2017 16:41
Does it raise an exception, or simply stop returning?
ifasina
@ifasina
Apr 09 2017 16:41
Stops returning
Bryce Boe
@bboe
Apr 09 2017 16:42
How long did you wait?
And what sub are you running it on?
ifasina
@ifasina
Apr 09 2017 16:43
Longest I've let it run is a few hours and after the first 15ish minutes it stops
I'm running it on r/RocketLeagueExchange
Is this function not meant to run for an extended periods of time?
Bryce Boe
@bboe
Apr 09 2017 16:44
I have a bot that runs on submissions for weeks at a time.
So it is meant to run indefinitely.
Occasionally exceptions occur, but those can be handled.
What version of PRAW do you have?
ifasina
@ifasina
Apr 09 2017 16:45
Wow
Um, is there a simple way I can check?
Bryce Boe
@bboe
Apr 09 2017 16:45
print(praw.__version__)
ifasina
@ifasina
Apr 09 2017 16:46
4.4.0
Bryce Boe
@bboe
Apr 09 2017 16:46
That is the most recent release.
There have been some updates to the development version
You could give that a try.
pip install --upgrade https://github.com/praw-dev/praw/archive/master.zip
Also, if you want to verify that it is still making requests you can do the following:
import logging                                                                           

import praw
reddit = praw.Reddit(...)

logging.getLogger('prawcore').setLevel(logging.DEBUG)
logging.getLogger('prawcore').addHandler(logging.StreamHandler())
logging.getLogger('prawcore').handlers[0].setLevel(logging.DEBUG)
Then you'll get the debugging output for each request. If you don't see requests, I would guess that there is something up in your code.
You could replace myFunc(comment) with print(comment) and see if the behavior goes away too.
ifasina
@ifasina
Apr 09 2017 16:51
Oh that will be really helpful. I was trying to pdb it but you can imagine how useful that was.
Currently I've commented out most of the contents of myfunc and all it does now is run a regex on the comment.body
Bryce Boe
@bboe
Apr 09 2017 16:53
Well hopefully the log is more revealing then.
ifasina
@ifasina
Apr 09 2017 16:57
been a few minutes and it's only made 1 request
Bryce Boe
@bboe
Apr 09 2017 17:01
Do you have multiple bots running?
Also you'll want to reset your authentication token because you copied your bearer token in there.
I should probably have prawcore not log headers since they often don't change.
What are the contents of myfunc?
I'm not convinced that your function is returning properly.
ifasina
@ifasina
Apr 09 2017 17:13
for comment in reddit.subreddit(sub).stream.comments():
       if (comment.submission.id == tid and comment.is_root):
             if (self.hasTable(comment.body) == True):
                    print("Comment", comment.id, "has table")
             elif (self.hasOtherFormat(comment.body) == True):
                    print("Comment", comment.id, "has other format")
             else:
                    print("not parsable")
Bryce Boe
@bboe
Apr 09 2017 17:31
Well that looks fine. Other instances of the bot?
You can print reddit.auth.limits at the end of your loop.
Perhaps you're at the end of your rate limit.
ifasina
@ifasina
Apr 09 2017 17:34
No just that one. reddit.auth.limits returned {'remaining': 516.0, 'reset_timestamp': 1491759600.0988455, 'used': 84}
Bryce Boe
@bboe
Apr 09 2017 17:34
Well that is rather interesting.
ifasina
@ifasina
Apr 09 2017 17:35
I think it's definitely one of my functions. If I remove the hasTable check its sending multiple requests
Bryce Boe
@bboe
Apr 09 2017 17:35
What does that function look like?
ifasina
@ifasina
Apr 09 2017 17:35
def hasTable(self, comment):
        for line in comment.split("\n"):
            if any(re.search(reg, line) for reg in table_regex):
                return True
        return False
Bryce Boe
@bboe
Apr 09 2017 17:35
Also, fyi, regarding blah(...) == True: the == true part is redundant.
ifasina
@ifasina
Apr 09 2017 17:36
Thanks
Bryce Boe
@bboe
Apr 09 2017 17:36
comment.split("\n")
I'm not sure what that will do.
Nevermind.
You already pass body in.
I suggest printing line before your if statement.
Maybe your regex is incredibly complex.
Perhaps it's stalling on one of them with one of the lines.
ifasina
@ifasina
Apr 09 2017 17:38
Maybe...
table_regex = [":{1}-+", ":{1}-+:{1}", "-+:{1}", "-+\|{1}-+"]
Bryce Boe
@bboe
Apr 09 2017 17:39
Regardless, if not calling hasTable gets it to work, it does seem outside of PRAW's scope.
ifasina
@ifasina
Apr 09 2017 17:40
Yeah, thanks for all your help! I just need to figure this out now lol
Bryce Boe
@bboe
Apr 09 2017 17:40
Good luck!
George Schizas
@gschizas
Apr 09 2017 17:50
@ifasina your table_regex seems a bit too complicated. You don't need the {1} bit after :. :{1}-+ is the same as :-+, and they both mean "one ':' followed by 1 or more '-'"
ifasina
@ifasina
Apr 09 2017 17:58
Thank you @gschizas
ifasina
@ifasina
Apr 09 2017 18:50
In case anyone cares, it seems the culprit was python's re.search() method. When called it doesn't return sometimes.
Example:
wow
George Schizas
@gschizas
Apr 09 2017 19:57
It's not that it doesn't return, it's that it returns None when it doesn't find your RegEx.
I ran this regular expression in regex101.com ((\w+'?\s*)+[-|~]\s*((\d+\.?\d+\$?)|(\$?\d+\.?\d+)) if I haven't mistyped anything) and I got an error I never saw before!
"Catastrophic backtracking has been detected and the execution of your expression has been halted. To find out more what this is, please read the following article: Runaway Regular Expressions"
George Schizas
@gschizas
Apr 09 2017 20:09
It's not a re module bug, it's your regular expression that's at fault.
George Schizas
@gschizas
Apr 09 2017 20:14
@ifasina You have effectively an (almost) infinite loop with your regular expression :)
Learned a cool phrase out of it though. "Catastrophic backtracking"