Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Anton Vorontsov
    @symbios-zi
    Do you have any idea? I tried a lot of proxies
    and the same result
    Riyan Bemachadan
    @riyan04314_gitlab
    Hi, need to scrap of google search result ( video page), its possible using this plugin ?
    Cat's Plugins
    @CatsPlugins
    I think GG just updated their result.
    I mean HTML structure
    since sometime the results return all null
    Soufiane Ghzal
    @gsouf
    @symbios-zi if you use public proxies that's the expected behavior. If you use private proxies that means there is something else
    Hi @riyan04314_gitlab if you want to get video results, for instance from this page : https://www.google.fr/search?q=simpsons+opening&rlz=1C1CHBF_frFR777FR777&oq=simpsons+ope&aqs=chrome.1.69i57j0l5.5980j0j7&sourceid=chrome&ie=UTF-8 then yes it's possible
    @CatsPlugins I'll be looking if I see anything
    Igor Lugis
    @dragonattack
    Hi guys. Could you please tell how you get data-sitekey from GoogleCaptchaException?
    @gsouf is there a method for that or this should be done by parsing DOM for now?
    Cat's Plugins
    @CatsPlugins
    @gsouf mate, I think the new version is unstable, sometime it returned all null, after awhile, it returned data, same proxy....hmmm
    Do you think what I can help you help you debug faster?
    Should I send you my bot?
    MeeQi
    @condy123
    Hello, all, I will use proxy to report,and return curl error_code:56.. I confirm that proxy IP can be used.
    Cat's Plugins
    @CatsPlugins
    I think gg spider new version is somehow unstable. Same IP, I can access and crawl amazon freely.
    but gg return 503
    Soufiane Ghzal
    @gsouf
    @CatsPlugins I confirm the latest version works fine. Http code 500 are internal errors from google https://support.google.com/gsa/answer/2686272?hl=en
    could be captcha as well
    @condy123 did you make sure curl is updated to the latest version? Error code 56 means "Failure with receiving network data." as explained on curl's doc https://curl.haxx.se/libcurl/c/libcurl-errors.html
    @dragonattack there are nothing for solving captcha or getting captcha id for now. It was available when captcha were easy to solve, but removed since google updated to recaptcha v2
    Cat's Plugins
    @CatsPlugins

    @gsouf my bad, I didn't read the update notice, I have changed the old method of build google to new one, all fine nows, results return within 5s <3 <3 <3

    Cheerrrrrrrrrrrrr!

    @dragonattack man, you can use nonoCaptcha to solve captcha via sounds.

    But it would increase the time to return data to customer, my bots right now, set timeout 5s, if there's no data, I will try the other proxy. So the total time to return data, incase of captcha will be around 15s, it's acceptable, > 20s would be toooooo long to wait.

    But it's also depend on what's the purpose, if use for rank tracker, and run silently, you won't need too much proxy.
    But like SERP checker, your user will need the proper result at no time.
    Cat's Plugins
    @CatsPlugins
    @gsouf next month, I will spend sometime to help you to crawl Bing.
    MeeQi
    @condy123
    @gsouf php-curl version 7.54.0
    ResCloud.com
    @ResCloudinc_twitter
    Hey guys hows it going?
    LarsSalazar
    @LarsSalazar
    Hi to all, i'm triyng to create a serp tool, but in the last search result i didn't get anymore the related search someone have the same issue?
    This is my call
    $googleUrl = new GoogleUrl();
    $googleUrl->setSearchTerm($searchterm);
                $googleUrl->setPage($page);
                if ($page != 0){
                    $googleUrl->setResultsPerPage(100);
                }
    
                $response = $googleClient->query($googleUrl);
    I Appriciate any help
    gaurav kumar singh
    @gksingh35_twitter
    image.png
    Pls help me out
    Ok got it, it should written without any quotes.
    SIMON Tiandray Henintsoa
    @simthax_gitlab
    Hello, I am trying to create a serp tool which is intended to make more than 20000 searches at once (loop with data from csv file). Is it possible while avoiding being blacklisted by google? Thanks in advance
    Harry Robinson
    @HarryRobinsonUK

    Hi All,
    I'm having an issue at the moment with SERP Spider v2.x, the crawler was running well until I tried it today, I got this error:

    Fatal error: Uncaught TypeError: Argument 2 passed to Serps\Core\Serp\BaseResult::__construct() must be of the type array, null given, called in C:\wamp64\www\crawler-admin\vendor\serps\search-engine-google\src\Parser\Evaluated\Rule\Natural\Classical\ClassicalResult.php on line 146 and defined in C:\wamp64\www\crawler-admin\vendor\serps\core\src\Core\Serp\BaseResult.php:17 Stack trace: #0 C:\wamp64\www\crawler-admin\vendor\serps\search-engine-google\src\Parser\Evaluated\Rule\Natural\Classical\ClassicalResult.php(146): Serps\Core\Serp\BaseResult->__construct(Array, NULL) #1 C:\wamp64\www\crawler-admin\vendor\serps\search-engine-google\src\Parser\AbstractParser.php(83): Serps\SearchEngine\Google\Parser\Evaluated\Rule\Natural\Classical\ClassicalResult->parse(Object(Serps\SearchEngine\Google\Page\GoogleSerp), Object(Serps\Core\Dom\DomElement), Object(Serps\Core\Serp\IndexedResultSet)) #2 C:\wamp64\www\crawler-admin\vendor\serps\search-engine-google\src\Parser\AbstractParser.php(78): Serps\SearchEngine\Google\Parser\AbstractPa in C:\wamp64\www\crawler-admin\vendor\serps\core\src\Core\Serp\BaseResult.php on line 17

    Any idea what could be causing this?
    Luis
    @neutronixx
    Hello, I have this error, with only 2 or 3 requests to googlr, is it normal? Serps\SearchEngine\Google\Exception\GoogleCaptchaException Object ( [captcha:protected] => Serps\SearchEngine\Google\Page\GoogleCaptcha Object ( [googleError:protected] => Serps\SearchEngine\Google\Page\GoogleError Object ( [url:protected] => Serps\SearchEngine\Google\GoogleUrlArchive Object ( [hash:protected] => [path:protected] => /sorry/index [scheme:protected] => https [query:protected] => Array ( [continue] => Serps\Core\Url\QueryParam Object ( [name:protected] => continue [value:protected] => https://www.google.com/search?q=vestidos [raw:protected] => ) [q] => Serps\Core\Url\QueryParam Object ( [name:protected] => q [value:protected] => EhAqAsIHIAlndAAAAAAAAAABGKjH7-UFIhkA8aeDS5EpOFm0H_T65_RL7RQI6WtalgnbMgFy [raw:protected] => ) ) [host:protected] => www.google.com [user:protected] => [pass:protected] => /sorry/index [port:protected] => ) [dom:protected] => DOMDocument Object ( [doctype] => (object value omitted) [implementation] => (object value omitted) [documentElement] => (object value omitted) [actualEncoding] => utf-8 [encoding] => utf-8 [xmlEncoding] => utf-8 [standalone] => 1 [xmlStandalone] => 1 [version] => [xmlVersion] => [strictErrorChecking] => 1 [documentURI] => [config] => [formatOutput] => [validateOnParse] => [resolveExternals] => [preserveWhiteSpace] => 1 [recover] => [substituteEntities] => [nodeName] => #document [nodeValue] => [nodeType] => 13 [parentNode] => [childNodes] => (object value omitted) [firstChild] => (object value omitted) [lastChild] => (object value omitted) [previousSibling] => [nextSibling] => [attributes] => [ownerDocument] => [namespaceURI] => [prefix] => [localName] => [baseURI] => [textContent] => https://www.google.com/search?q=vestidos var submitCallback = function(response) {document.getElementById('captcha-form').submit();}; À propos de cette pageNos systèmes ont détecté un trafic exceptionnel sur votre réseau informatique. Cette page permet de vérifier que c'est bien vous qui envoyez des requêtes, et non un robot. Que s'est-il passé ? Cette page s'affiche lorsque Google détecte automatiquement des requêtes émanant de votre réseau informatique qui semblent enfreindre les Conditions d'utilisation. Le blocage prendra fin peu après l'arrêt de ces requêtes. En attendant, la saisie de l'image CAPTCHA ci-dessus vous permettra de continuer à utiliser nos services.Des applications malveillantes, un plug-in de navigateur ou un script qui envoie des requêtes automatiques peuvent être à l'origine de ce trafic. Si vous utilisez une connexion réseau partagée, demandez de l'aide à votre administrateur. Il est possible qu'un autre ordinateur utilisant la même adresse IP soit en cause. En savoir plusVous pouvez être invité à saisir les caractères de l'image CAPTCHA si vous utilisez des termes avancés auxquels les robots ont recours ou si vous envoyez des requêtes très rapidement. Adresse IP : 2a02:c207:2009:6774::1Heure : 2019-04-21T03:29:45ZURL : https://www.google.com/search?q=vestidos ) [xpath:Serps\Core\Dom\InternalDocumentWrapper:private] => Serps\Core\Dom\DomXpath Object ( [documentWrapper:protected] => Serps\SearchEngine\Google\Page\GoogleError Object RECURSION [document] => (object value omitted) ) ) ) [message:protected] => [string:Exception:private] => [code:protected] => 0 [file:protected] => /home/comcazadm/public_html/serp-spider/search-engine-google/src/GoogleClient.php [line:protected] => 81 [trace:Exception:private] => Array ( [0] => Array ( [file] => /home/comcazadm/public_html/serp-spider/search-engine-google/prueba1.php [line] => 30 [function] => query [class] => Serps\SearchEngine\Google\GoogleClient [type] => -> [args] => Array ( [0] => Serps\SearchEngine\Google\GoogleUrl Object ( [hash:protected] => [path:protected] => /search [scheme:protected] => https [query:protected] => Array ( [q] => Serps\Core\Url\QueryParam Object ( [name:protected] => q [value:protected] => vestidos [raw:protected] => ) ) [host:protected] => www.google.com [user:protected] => [pass:protected] => [port:protecte
    ahmedrageh
    @ahmedrageh
    hi
    Muhammad Elgendi
    @Muhammad-Elgendi
    Hi, I was wondering is there a way to use this parser on raw html pages without using provided clients ?
    hsnelamir
    @hsnelamir
    Hello @Muhammad-Elgendi
    yes you can,
    $browser = new Browser(new CurlClient(), $userAgent, $browserLanguage);
    $googleUrl = GoogleUrl::fromString('https://www.google.com/search?q=Coffee&oq=Coffee&hl=en&gl=ye&sourceid=chrome&ie=UTF-8');
    $googleClient = new GoogleClient($browser);
    $response = $googleClient->query($googleUrl);
    //$dom = $response->getDom();
    $results = $response->getNaturalResults();
    i have a working example on my server, call me if you needed anything
    LeoXu
    @saselovejulie
    Hi could this parser work well on the newest google result page? thanks
    chandu-code
    @chandu-code
    how can i get the google news search result?
    David Johnson
    @dmjio
    hey
    does Google change its HTML often?
    Nico Grienauer
    @Grienauer
    hey! short question: is there a docker setup available for serp-spider which is trimmed perfectly to the needs of the tool? so that is is slick. someone has this already done and lying around in some repo? :P thx
    Cypher Shrugged [BTC⚡️LN]
    @CypherShrugged_twitter

    hi guys, did google change the DOM recently?
    I get error:
    "Fatal error: Uncaught Serps\SearchEngine\Google\Exception\InvalidDOMException: Unable to check javascript status. Google DOM has possibly changed and an update may be required"

    The maps url I am trying to parse is:

    $googleUrl = GoogleUrl::fromString('https://www.google.com/maps/search/?api=1&query=pharmacy&@-25.9590862%2C28.1005628');

    Alex Garcia
    @alexgarciab
    Hello, I think that Google has updated their DOM and the SERPS implementation it is not working correctly anymore. Related Issue: serp-spider/search-engine-google#131
    Cypher Shrugged [BTC⚡️LN]
    @CypherShrugged_twitter
    Thx for the confirmation. Lets hope someone smarter than me can tweak the code for us