Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Jon Gjengset
    @jonhoo
    no, that seems like a reasonable addition!
    out of curiosity -- why would you rather have the examples in a file than in the docs?
    lazypassion
    @lazypassion

    Well it's not to remove the examples from the docs, but I guess it's to have definite working versions of the doc comments. Something that a user could
    go back to if they're using it as reference for their code. I recognize that it isn't necessary, but having workable examples that can be executed I think would be nice.
    Especially since some can demonstrate different workflows like scraping, downloading a file and so on.

    Part of what I had in a mind:

    • Some examples like the ones in the doc comments
    • Maybe an example on setting up webdriver capabilities
    • Scraping example
    • Also adding examples to the doc comments for the different methods for Client, Element, and Form

    All of these examples aside from the scraper would only be one file. They aren't a full cargo project on their own.

    Jon Gjengset
    @jonhoo
    Seems reasonable to me!
    lazypassion
    @lazypassion
    I tried the clear method for Element and it seems to only work on user input. In other words, if the element already had a default value then clear didn't work. If i use send_keys then call clear, it would clear out my input but wouldn't change the original value. I'm currently trying this on airbnb, so I'll try on other websites and see how it goes.
    Jon Gjengset
    @jonhoo
    Hmm, interesting.. It might be that they're using some JavaScript magic there
    lazypassion
    @lazypassion
    Yeah, I feel like that's the case.
    lazypassion
    @lazypassion
        let client = Client::new("http://localhost:4444");
        let price_button_xpath = r#"//button[@class="_1i67wnzj" and @aria-haspopup="true" and @aria-expanded="false" and @aria-controls="menuItemComponent-price_range"]"#;    
    let price_apply_span = r#"//span[@data-action="save"]"#;
        let results_url = "https://www.airbnb.com/s/San-Francisco--CA--United-States/homes?refinement_paths%5B%5D=%2Fhomes&toddlers=0&search_type=filter_change&zoom=10&search_by_map=true&sw_lat=37.567223764294454&sw_lng=-122.58102479003907&ne_lat=38.10949381266281&ne_lng=-122.13470520996094&place_id=ChIJIQBpAG2ahYAR_6128GcTUEo&checkin=2019-08-15&checkout=2019-08-19&adults=2&room_types%5B%5D=Entire%20home%2Fapt&s_tag=GeEBH5Z6"; 
    
        let apply_price = client
            .map_err(|error| unimplemented!("failed to connect to WebDriver: {:?}", error))
            .and_then(move |client| client.goto(results_url))
            .and_then(move |client| {
                    client.wait_for_find(Locator::XPath(price_button_xpath))
                    .and_then(|element| element.click())
            })
            .and_then(move |client| {
                client.wait_for_find(Locator::Id("price_filter_min"))
            })
            .and_then(|mut element| {
                element.clear()
                    .map(|_| element)
            })
            .and_then(|element| {
                Ok(element.client())
            })
            .and_then(|mut client| client.persist())
            .map_err(|error| panic!("webdriver command error: {}", error));
    
        tokio::run(apply_price);
    Ok that should be all the relevant code hopefully
    lazypassion
    @lazypassion
    Seems like airbnb is doing javascript magic so I'll have to wait for more functionality in fantoccini. Or actually try to add some of the webdriver functionality. Though I don't know how to start with that.
    Actually does the send_keys function support sending key strokes like ctrl + a?
    Jack
    @jh0l
    Yeah wd has got special character sequences for each modifier key, its stateful
    So it waits for an alphanumeric character to be pressed in combination
    If the input is in a form you can set its value with the Form::set_by_name fn
    lazypassion
    @lazypassion
    The reason why clear wasn't working in my case is because the input on airbnb was expecting key events and clear doesn't activate any key events. So i guess a nice addition would be key events to the library
    Jon Gjengset
    @jonhoo
    I think you can do that with SendKeys, no?
    lazypassion
    @lazypassion
    Hm well i tried using "\b" for backspace in send_keys and it only backspaced keys that I sent but not the one that is already there. Maybe it's something that selenium is necessary for. So maybe there isn't a way for this library to work around that.
    Jon Gjengset
    @jonhoo
    I think the trick is going to be to add support for PerformActions: https://docs.rs/webdriver/0.39.0/webdriver/command/enum.WebDriverCommand.html#variant.PerformActions
    which lets you do things like key presses
    lazypassion
    @lazypassion
    ah i see, cool
    NickVermaas
    @NickVermaas
    I tried to find out how you could get the html source via lib.rs, but I can't get it working I want to get the html from the page with a headless browser. I figured out how to get a headless client. I hope someone could help, Rust is not my main programming language I come from PHP and Python, I have done a couple of Rust tutorials I really like the language and I am going to dive deeper into the language. If there are also any books or online courses that are recommended, then feel free to let me know, below is the code I currently have. Thank you in advance!
    // use fantoccini::error::CmdError;
    use fantoccini::{Client};
    use futures::future::Future;
    use tokio;
    
    pub fn run() {
        // expects WebDriver instance to be listening at port 4444
        // let client = Client::new("http://localhost:4444");
        let mut caps = serde_json::map::Map::new();
        let opts = serde_json::json!({ "args": ["--headless"] });
        caps.insert("moz:firefoxOptions".to_string(), opts.clone());
        let client = Client::with_capabilities("http://localhost:4444", caps);
        let get_html = client
            .map_err(|error| unimplemented!("failed to connect to WebDriver: {:?}", error))
            .and_then(|client| client.goto("https://www.wikipedia.org/"))
            .and_then(|mut client| {
                // find the source for the Wikipedia globe
                let html = client.source();
                println!("{}", html);
            })
            .map(|_| ())
            .map_err(|error| panic!("a WebDriver command failed: {:?}", error));
    
        tokio::run(get_html);
    }
    lazypassion
    @lazypassion
    do you need the source after you finish executing get_html?
    The problem in this example is that client.source returns a future, so you don't get the value until you wait for it to complete.
    NickVermaas
    @NickVermaas
    Do I need to add a clien_wait or is there a wait on complete loading function or something like that? So what I want as result is when a page has javascript and other things to load afterwards the source html of the entire page.
    lazypassion
    @lazypassion
    Ah yes I understand, I wanted to know if you wanted the page source to be accessible after tokio is finished executing.
    so in your example you should another and_then with a closure that accepts a String
           .and_then(|mut client| {
                // find the source for the Wikipedia globe
                client.source()
                // println!("{}", html);
            })
            .and_then(|source| {
                println!("{}", source);
                Ok(())
            })
    the above link shows how you can get a value after tokio finishes executing all futures
    NickVermaas
    @NickVermaas
    Thank you very much for the explanation it works now!!
    Jon Gjengset
    @jonhoo
    @lazypassion thanks for helping out while I wasn't around! <3
    Dirkjan Ochtman
    @djc
    is there a known-good way/example for running fantoccini in CI?
    Jon Gjengset
    @jonhoo
    It's a little awkward, but take a look at fantoccini's .travis.yml for how you might do it
    I think you should be able to do something similar (headless browsers) on most CI platforms
    Dirkjan Ochtman
    @djc
    looks reasonably straightforward
    Paul-Sebastian Manole
    @brokenthorn
    Is there any way to scroll browser window?
    I want to loop over some images and it seem they are dynamically loaded into the DOM.
    Just selecting all the images and print-ing their src attribute value yields a proper link for just a few, the ones at the top of the page, the visible part of the page, while the rest print base64 stuff like data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR4nGP6zwAAAgcBApocMXEAAAAASUVORK5CYII=
    Paul-Sebastian Manole
    @brokenthorn
    sorry, forgot about base64 encoded inline images
    Jon Gjengset
    @jonhoo
    For what it's worth, you can scroll by executing JavaScript on the page and using scrollTo :)
    Paul-Sebastian Manole
    @brokenthorn
    Oh you can execute js? Cool.
    That's potentially quite enabling.
    that's the idea :p
    Garrett Graves
    @GravesG_gitlab
    hello world
    there are so many things i wish this library could do and i have no clue how to implement them. Is anyone open to me bouncing ideas off of them
    Paul-Sebastian Manole
    @brokenthorn
    @GravesG_gitlab I believe this library is not meant to do anything more than what the webdriver specification allows.
    Jon Gjengset
    @jonhoo
    @brokenthorn yeah, that was the intention -- a fairly low-level browser automation library for Rust. I think it totally makes sense to build a higher-level library on top of it, but think that should be separate :)
    Paul-Sebastian Manole
    @brokenthorn
    :thumbsup: