Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Terminator X
    @jarun
    awesome! you mean in Set-Cookie conn.request, right?
    stoneluo86
    @stoneluo86
    some time work,Occassionally fail.I use chrome works fine through proxy,so i add all request header from chrome to conn.request
    Terminator X
    @jarun
    Can you please share the details here... From different countries and config it may be different. Will help us to investigate this and optionally add some ourselves.
    Terminator X
    @jarun
    For*
    stoneluo86
    @stoneluo86

    def google_get(conn, url):
    """Send a GET request to Google with the appropriate headers.
    url can be relative (to the appropriate Google domain).
    """

    global ua
    conn.set_tunnel("www.google.com")
    conn.request("GET", url, None, {
        "Accept-encoding": "gzip",
        "User-Agent": ua,
        "Cookie":cookie,
        "Accept":accept,
        "Accept-Encoding":acceptencoding,
        "Accept-Language":acceptlanguage,
        "Cache-Control":cachecontrol,
        "Connection":connection,
        "Host":host,
        "Upgrade-Insecure-Requests":UpgradeInsecureRequests,
    })
    return conn.getresponse()

    def new_connection(domain=None):
    """Returns a new connection to the given domain with appropriate options.
    When the given domain is absent, the global variable server is used instead.
    """

    global server
    #return http.client.HTTPSConnection(domain if domain else server, timeout=45)
    return http.client.HTTPSConnection("proxy_host",proxy_port)
    Terminator X
    @jarun
    thanks!
    Zhiming Wang
    @zmwangx
    Just a FYI: Google is experimenting with pure JavaScript rendering. Here's the source I just got: https://gist.github.com/fc8535bfab1a4319bf7c7de37d062a30. And here's what it looks like:
    Screen Shot 2016-07-02 at 2.43.34 PM.png
    Terminator X
    @jarun
    thanks for the heads up!
    Sasikanth
    @Sasikanth3
    howto integrate it with w3m
    Zhiming Wang
    @zmwangx
    @Sasikanth3 In principle you should be able to do BROWSER=w3m googler query and it should work when you open URLs inside googler, but in reality, we currently have trouble integrating with text-based browsers due to stdin and stdout redirected to /dev/null in https://github.com/jarun/googler/blob/dfc988f16c7e17bb691b3ef4e154f17ae5436588/googler#L93-L109. See jarun/googler#125. Rest assured, it's not a hard technical problem; we just need to figure out the most user-friendly solution after some discussion. A temporary workaround can be found in jarun/googler#125 too; see "Temporary workaround".
    Zhiming Wang
    @zmwangx
    @Sasikanth3 Please test jarun/googler#126 if you'd like to. BROWSER=w3m googler query.
    Zhiming Wang
    @zmwangx
    @Sasikanth3 Text-based browser integration has been restored in the master branch. See the new FAQ: https://github.com/jarun/googler#faq
    Zhiming Wang
    @zmwangx
    @jarun What about enabling daily/weekly Travis builds in case Google introduces breaking changes? New Travis feature: https://blog.travis-ci.com/2016-12-06-the-crons-are-here https://docs.travis-ci.com/user/cron-jobs/ You can schedule builds on https://travis-ci.org/jarun/googler/settings. (For some reason the feature doesn't work for me and I have emailed support.)
    Terminator X
    @jarun
    :+1: Awesome! I have scheduled weekly.
    Zhiming Wang
    @zmwangx
    cool :+1:
    I changed it to "do not run if there has been a build in the last 24 hrs"
    Terminator X
    @jarun
    :+1:
    Ahmad Mahfouz
    @eln1x_twitter
    hello folks
    Terminator X
    @jarun
    hi there!
    Emil Aasa
    @emilaasa
    Cool project! =)
    Terminator X
    @jarun
    Thank you! Please leave a star if you find it useful. :)
    Zhiming Wang
    @zmwangx
    @jarun I'm looking at googler's GitHub topics: google-news, google-search, google-site-search, google-terminal and googler-at. googler is the sole project in the last three, and IMO they don't make good topics (topics are for discoverability, they are not really short summaries of the project's capabilities). What about removing the last three (well, maybe google-site-search can be kept, it's a more neutral one), and add "google" to the mix?
    "command-line" could be a good, relevant topic too.
    Terminator X
    @jarun
    Done! Retained site search.
    Christian Hänsel
    @chrishaensel
    Hi there
    Christian Hänsel
    @chrishaensel
    Maybe someone can help... I am getting the error "[ERROR] Connection blocked due to unusual activity." And yes, I have been running a few (100+) queries today... Does anybody know what the limit is? Is there a way to set a cutom user agent?
    Terminator X
    @jarun
    Google detects tons of automated queries, even when you run them from the browser. The last time we checked it was indeed around 100.
    You can change the user agent in the code, it's Python, so quiet hackable. Just make sure your don't use an Android one, the last time I checked it didn't work... probably because they moved to JS results, but I didn't dive much wither.
    either*
    Zhiming Wang
    @zmwangx
    @jarun I turned on auto cancellation on Travis to save us a few queries when pushing corrections rapidly. https://blog.travis-ci.com/2017-03-22-introducing-auto-cancellation
    Terminator X
    @jarun
    :+1:
    Jamsheed Mistri
    @JamsheedMistri

    Hi, I'm having an issue. I'm trying to use googler on my raspberry pi. I've tried it on my mac and it works perfectly, and I recently updated my pi to use python 3.6.0. When I do googler, I get this error:

    Traceback (most recent call last): File "./googler", line 29, in <module> from http.client import HTTPSConnection ImportError: cannot import name 'HTTPSConnection'

    Could anyone help me please?
    Terminator X
    @jarun
    This isn't a googler issue. Please take a look at: https://docs.python.org/3/library/http.client.html. Specifically Note HTTPS support is only available if Python was compiled with SSL support (through the ssl module).
    If you are trying to port googler on Raspberry Pi that's nice. But you should be ready to figure out minor import related issues yourself. We can't do that for you.
    Jamsheed Mistri
    @JamsheedMistri
    Cool, thanks
    Jamsheed Mistri
    @JamsheedMistri
    @jarun When I execute googler in PHP, I get this error. How can I fix this?
    stdout encoding 'ascii' detected. googler requires utf-8 to work properly. The wrong encoding may be due to a non-UTF-8 locale or an improper PYTHONIOENCODING. (For the record, your locale language is and locale encoding is ; your PYTHONIOENCODING is not set.) Please set a UTF-8 locale (e.g., en_US.UTF-8) or set PYTHONIOENCODING to utf-8.
    Terminator X
    @jarun
    Please set your locale to utf-8. You can refer to https://wiki.archlinux.org/index.php/locale
    Jamsheed Mistri
    @JamsheedMistri
    Thanks!
    steackfrite
    @steackfrite

    @jarun Hi there!
    First, thanks for your work, it's a great idea!

    Now I'm trying to get the same results as the google web version using keywords but i'm quite unlucky so far.

    Here is the google search : site:*.dualthegame.com -site:www.dualthegame.com
    And here is the googler search I made: googler site:*.dualthegame.com site:" -www.dualthegame.com" -n 30 --np --unfilter -l en

    The results are quite different. With the WebUI version I can find several different domain (board, community, etc.) but with googler I can find only 1 domain by search.
    So I need to add a filter after each new search.

    Is there a way to have the same result or at least to be very similar?

    Zhiming Wang
    @zmwangx
    googler "site:*.dualthegame.com -site:www.dualthegame.com"
    exactly the same results
    steackfrite
    @steackfrite
    @zmwangx Ouuuh sir, thank you!
    I did not think I could right it literally like that, it's perfect!
    Naveen Kumar
    @naveenCotha
    Hey @jarun thanks for the googler library. Is there a python sdk ?
    Terminator X
    @jarun
    Nope.