Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Terminator X
    @jarun
    Can we have the numbers with fetching results without gzip compression?
    Zhiming Wang
    @zmwangx
    I'm in a meeting right now. Will get back to you later.
    Zhiming Wang
    @zmwangx
    Not much different without gzip:
    ( repeat 10; do; googler --np hello > /dev/null; done; )  0.86s user 0.22s system 19% cpu 5.572 total
    Zhiming Wang
    @zmwangx

    Actually I made a mistake when measuring "no IO and no parsing" time. I only commented out the reading from socket (downloading) part; opening connection and waiting time were included. The correct time is something like

    ( repeat 10; do; googler --np hello > /dev/null; done; )  0.53s user 0.21s system 93% cpu 0.797 total

    so IO time is even longer, almost 500ms. In fact, I verified with Chromium's network tab, and indeed on my current Wi-Fi it takes about 500ms in total (waiting time included, ~100ms) to finish the GET request.

    Terminator X
    @jarun
    OK... in that case we are fine with gzip.
    Zhiming Wang
    @zmwangx
    By the way, in the end I still omitted the "card result" (https://github.com/jarun/googler/blob/master/googler#L317-L319) because sometimes it is duplicated (Wikipedia result could be duplicated, for instance), plus the fact that it doesn't have an abstract.
    Did you test for any regressions? I already did randomized tests, but given that Google doesn't serve the same thing to everyone, we might still need more testing.
    Terminator X
    @jarun
    By the way, in the end I still omitted the "card result"
    okies
    Did you test for any regressions?
    I will in the coming weekend.
    too busy with office due to an ongoing release :)
    Terminator X
    @jarun
    Can you check with -d switch if your query is being redirected?
    OK leave it...
    Zhiming Wang
    @zmwangx
    Redirected to?
    Terminator X
    @jarun
    if i run a google search, it redirects to server for India
    in regular browser
    wanted to check something
    figured it out anyway
    stoneluo86
    @stoneluo86
    how can i use proxy in googler?
    Zhiming Wang
    @zmwangx
    You either need to use a global/command level proxy (that is, outside googler), or see jarun/googler#37
    Terminator X
    @jarun
    Use tsocks
    The last time I heard, googler works fine on it.
    stoneluo86
    @stoneluo86
    does that mean my proxy was blocked by google
    Zhiming Wang
    @zmwangx
    Yes, Google blocks most free proxies you can find on the web. Not exactly surprising.
    stoneluo86
    @stoneluo86
    i understand,tks
    Zhiming Wang
    @zmwangx
    No problem.
    stoneluo86
    @stoneluo86
    i add cookie header and proxy works
    Terminator X
    @jarun
    awesome! you mean in Set-Cookie conn.request, right?
    stoneluo86
    @stoneluo86
    some time work,Occassionally fail.I use chrome works fine through proxy,so i add all request header from chrome to conn.request
    Terminator X
    @jarun
    Can you please share the details here... From different countries and config it may be different. Will help us to investigate this and optionally add some ourselves.
    Terminator X
    @jarun
    For*
    stoneluo86
    @stoneluo86

    def google_get(conn, url):
    """Send a GET request to Google with the appropriate headers.
    url can be relative (to the appropriate Google domain).
    """

    global ua
    conn.set_tunnel("www.google.com")
    conn.request("GET", url, None, {
        "Accept-encoding": "gzip",
        "User-Agent": ua,
        "Cookie":cookie,
        "Accept":accept,
        "Accept-Encoding":acceptencoding,
        "Accept-Language":acceptlanguage,
        "Cache-Control":cachecontrol,
        "Connection":connection,
        "Host":host,
        "Upgrade-Insecure-Requests":UpgradeInsecureRequests,
    })
    return conn.getresponse()

    def new_connection(domain=None):
    """Returns a new connection to the given domain with appropriate options.
    When the given domain is absent, the global variable server is used instead.
    """

    global server
    #return http.client.HTTPSConnection(domain if domain else server, timeout=45)
    return http.client.HTTPSConnection("proxy_host",proxy_port)
    Terminator X
    @jarun
    thanks!
    Zhiming Wang
    @zmwangx
    Just a FYI: Google is experimenting with pure JavaScript rendering. Here's the source I just got: https://gist.github.com/fc8535bfab1a4319bf7c7de37d062a30. And here's what it looks like:
    Screen Shot 2016-07-02 at 2.43.34 PM.png
    Terminator X
    @jarun
    thanks for the heads up!
    Sasikanth
    @Sasikanth3
    howto integrate it with w3m
    Zhiming Wang
    @zmwangx
    @Sasikanth3 In principle you should be able to do BROWSER=w3m googler query and it should work when you open URLs inside googler, but in reality, we currently have trouble integrating with text-based browsers due to stdin and stdout redirected to /dev/null in https://github.com/jarun/googler/blob/dfc988f16c7e17bb691b3ef4e154f17ae5436588/googler#L93-L109. See jarun/googler#125. Rest assured, it's not a hard technical problem; we just need to figure out the most user-friendly solution after some discussion. A temporary workaround can be found in jarun/googler#125 too; see "Temporary workaround".
    Zhiming Wang
    @zmwangx
    @Sasikanth3 Please test jarun/googler#126 if you'd like to. BROWSER=w3m googler query.
    Zhiming Wang
    @zmwangx
    @Sasikanth3 Text-based browser integration has been restored in the master branch. See the new FAQ: https://github.com/jarun/googler#faq
    Zhiming Wang
    @zmwangx
    @jarun What about enabling daily/weekly Travis builds in case Google introduces breaking changes? New Travis feature: https://blog.travis-ci.com/2016-12-06-the-crons-are-here https://docs.travis-ci.com/user/cron-jobs/ You can schedule builds on https://travis-ci.org/jarun/googler/settings. (For some reason the feature doesn't work for me and I have emailed support.)
    Terminator X
    @jarun
    :+1: Awesome! I have scheduled weekly.
    Zhiming Wang
    @zmwangx
    cool :+1:
    I changed it to "do not run if there has been a build in the last 24 hrs"
    Terminator X
    @jarun
    :+1:
    Ahmad Mahfouz
    @eln1x_twitter
    hello folks