Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
Repo info
    Nick Tomlin
    Branden Byers

    Now that I have experienced flake working, what I’d like to try next is outputting one final xunit report from the multiple reports being run. Do you see this as being something in the scope of flake?

    For example:

    with —max-attempts=3 the custom xunit reporter will write three xml files. Before each xunit report overwrites the last, I’ll parse the xml and store it for comparison once all tests are finished.

    If all tests pass by the final run, then I will replace the first xunit’s failed test data with the newer passing data (duration, etc). And I’ll do the same with the second xunit report, replacing any remaining failed tests in the first report with the passing results from the second one.

    Then I will write these to one final xunit report that is required for our Jenkin’s builds.

    I could do all of this outside of flake if necessary, but given that I assume other people will have need for one final report of all passing tests, this is something that I could also adapt for other results requirements other than xunit. What do you think?
    Nick Tomlin
    Hmm. It seems a little outside the scope to me since parsing XML is only going to affect a subset of users but would mean (potentially) a lot of added code
    Branden Byers
    Sure, understandable. I’ll just do it outside of Flake.
    Branden Byers

    Hi @NickTomlin, after trying a few different ideas around parsing the final XML output in Jenkins, I realized that we are able to take an even easier approach than protractor-flake.

    I now have a custom XML reporter that writes separate XML files for each suite. Then in our Jenkins bash script, after the tests have run (and the xml files have been written), I have a very short script that parses the xml looking for failures > 0.

    That script then returns failing suites as a string of comma separated failed suites (failed).

    If that script returns any failed suites, then the Jenkins bash script re-runs protactor as we normally do with —suite=$failed

    But I may never have come to this solution if it weren’t for all your work with protractor-flake. So thanks for the inspiration!
    Nick Tomlin
    @brandenbyers sorry for the late reply. That's great to hear! Simpler is better :)

    Hey @NickTomlin, thanks for getting back to me thru e-mail. We can discuss the issue I am facing on here as you had asked. Here is my original question to you:

    I have been looking for a way to retry failed tests in protractor and your tool is the best one I have come across so far! I configured it with my sample test project and got it to work, but it seems to be retrying all the tests in my suite instead of just the failed ones. I saw that this is listed as one of the caveats with the tool when you are using a custom reporter. So I wanted to know which reporter will it work with?

    I’m using Jasmine 2.3.4 and ‘Jasmine Reporters’ which publishes the results in an xml. I saw that one of the solutions was to set the displayStackTrace to ‘spec’ or ‘all’ if you are using jasmine-spec-reporter. But I don’t think the reporter I am using as such an option. So I would like to know the best way to get this to work (1) without a custom reporter (2) with a custom reporter (which one?)

    in your reply you asked me if the xml from the reporter gets sent out to STDOUT. I am not sure how to check this. Can you tell me how?
    Nick Tomlin
    @amistry03 does the reporter capture the input and write it to a file, or does it pass it on, e.g. does protractor protractor.conf > output.out right the test output to output.out my guess is that it just writes it to an XML file and does not pass it on?
    @NickTomlin that's correct, it writes it into an xml file (the one which I emailed to you)
    Hello @NickTomlin , I am currently testing the protractor-flake and it's working great but I have one question, when I get a single test fail I see in log "Re-running the following test files: (...)" and it reruns all tests present in that file. Is it possible to change it to run only the one test that has failed? I am using Protractor 2.0.0. Thanks for the great job so far :)
    @Martis89 sounds like the same problem as mine! @NickTomlin let us know if you need any additional info from us!
    Nick Tomlin
    @Martis89 a few questions: 1. What OS are you using? 2. What version of jasmine are you using? 3. Are you using a custom reporter?
    @amistry03 you will need to print the output of your tests to STDOUT in addition to writing the XML file otherwise protractor-flake will not work. You may be able to just use multiple reporters (the standard one and the XML one)
    @NickTomlin 1 - I am using Ubuntu 14.03.3 LTS desktop. These tests are ran via Jenkins CI.
    2 - Jasmine version 2.0.
    3 - Karma-junit-reporter 0.2.2
    Nick Tomlin
    @Martis89 I believe you have the same problem as Amistry. You'll need to make sure you are also printing the test results to stdout
    Dmitry Sasov
    Hi @NickTomlin I just found out about protractor-flake and it looks really great, but I wasn't able to install it. Please see the error attached below. Am I right that it doesn't work with Protractor 3rd version and I have to downgrade to the protractor@2.x? Thank you!
    dima@lenovo-g50-70:~$ npm i -g protractor-flake
    npm WARN peerDependencies The peer dependency protractor@2.x included from protractor-flake will no
    npm WARN peerDependencies longer be automatically installed to fulfill the peerDependency 
    npm WARN peerDependencies in npm 3+. Your application will need to depend on it explicitly.
    /home/dima/.npm/bin/protractor-flake -> /home/dima/.npm/lib/node_modules/protractor-flake/bin/protractor-flake
    npm ERR! Linux 3.13.0-74-generic
    npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "protractor-flake"
    npm ERR! node v4.2.2
    npm ERR! npm  v2.14.7
    npm ERR! code EPEERINVALID
    npm ERR! peerinvalid The package protractor@3.0.0 does not satisfy its siblings' peerDependencies requirements!
    npm ERR! peerinvalid Peer protractor-flake@1.0.2 wants protractor@2.x
    npm ERR! Please include the following file with any support request:
    npm ERR!     /home/dima/npm-debug.log
    Chris Nguyen
    Hi @NickTomlin I'm running protractor-flake with multiCapabilities but it seems to be running all of tests on failure
    Sri Harsha
    Hi @Nick. Same happening to me , running all the tests on failure. [feat(protractor-flake)]: rerun only failed test on failure.
    Rohit Kadam
    @NickTomlin how to use protractor-flake with grunt, any sample example, documentation link will be useful?
    hi there, is the issue of running all of tests on failure solved? it is still happen to me unfortunately.
    Uma Mahesh
    How to handle this exception even if i am trying browser.wait and expected conditions still it raises exception
     Failed: unknown error: Element is not clickable at point (1269, 20). Other element would receive the click: <div modal-render="true" tabindex="-1" role="dialog" class="modal fade ng-isolate-scope full-screen-busy-spinner in" modal-animation-class="fade" ng-class="{in: animate}" ng-style="{'z-index': 1050 + index*10, display: 'block'}" ng-click="close($event)" modal-window="modal-window" window-class="full-screen-busy-spinner" index="0" animate="animate" modal-animation="true" style="z-index: 1050; display: block;">...</div>
    Uma Mahesh

    @NickTomlin I have followed your read me and added code to the protractor conf and trying to run using protractor-flake but i see the below text

    Using standard to parse output
    Re-running tests: test attempt 3

    Tests failed but no specs were found. All specs will be run again.

    **you must either specify a configuration file or at least 3 options. See below for the options:

    Usage: protractor [configFile] [options]
    configFile defaults to protractor.conf.js
    The [options] object will override values from the config file.
    See the reference config for a full list of options.```

    protractor-flake protractor-local.config.js I have customized config
    Jan-Niklas W.
    Hi everyone, is it possible to rerun protractor-flake just with failing tests due to timeouts? I think this would be a great possibility to improve the test quality.
    Luis Pablo
    Hello! :) Is somebody here?
    I have a little problem, I was using shardTestFiles: true in my protractor config because in one test I link an account with Facebook being on a dashboard interface, the next test unlinks account of facebook, and the next spec that is another file to test mobile interface I do this again, but if I'm not using shardTestFiles: true it obviously use the same browser so the process of linking a facebook account by second time is different and it doesn't display the log in pop up of facebook, that's why I was using shardTestsFiles, to avoid this issue and always pop up the facebook log in form, how can I solve my problem so I can use protractor-flake properly? Also, how I can make that it stops when the first failure happens and restart the test running?
    Hi, has anyone try to use this seemingly awesome plugin with gulp? The project I work already uses gulp-protractor to run protractor tests and I can't see an easy path to integrate it so wondering if someone can share me some example gulp task that uses protractor-flake (if you use gulp-protractor, that would be bonus but that is probably too much to ask) - Thanks for your help in advance!
    Khaja Moinuddin Mohammed
    Hello everyone. I was given protractor-flake a test but getting the following error.
    (function (exports, require, module, __filename, __dirname) { @IF EXIST "%~dp0\n
    ode.exe" (
    SyntaxError: Invalid or unexpected token
    I have tried various combinations of passing the protractor arguments but still facing the same issue
    node_modules\.bin\protractor-flake.cmd --protractor-path=node_modules\.bin\protractor.cmd --parser standard --node-bin node --max-attempts=3 --color=magenta -- suite=synthetic params.env syn_sat protractor.conf.js
    By the way the normal protractor tests run when i run from the command line as below
    node_modules\.bin\protractor.cmd protractor.conf.js --suite=synthetic --params.env syn_sat
    I have opened an issue against this here NickTomlin/protractor-flake#64
    Khaja Moinuddin Mohammed
    Not sure if it helps, but am using ES5 and not ES6 or Typescript for my protractor js tests
    Hello. Is it possible to use protractor-flake with Angular-CLI?
    Wilson Palkuri
    Hi @NickTomlin - I have same problem as @Martis89 and @amistry03. All my tests are being re run (both failed and passed). Where do you suggest to print test results ? i see that 'standard' parser is parsing the entire 'output'. Please advice or provide a written solution. Thanks.
    Wilson Palkuri
    @NickTomlin - Cn we use protractor-flake with --suite ? is there a fix for this yt ?

    @NickTomlin I added a flakey test output file for jasmine-spec-reporter, and the following unit test:

        it('isolates individual failed specs from jasmine-spec-reporter output', () => {
          protractorFlake({maxAttempts: 3})
          expect(spawnStub).to.have.been.calledWith('node', [pathToProtractor(), '--params.flake.retry', true, '--specs', '/tests/a-flakey.test.js'])

    but it's failing with this error:

      1) Protractor Flake failed specs isolates individual failed specs from jasmine-spec-reporter output:
         AssertionError: expected stub to have been called with arguments node, ["/Users/brine/protractor-flake/node_modules/protractor/bin/protractor", "--params.flake.retry", true, "--specs", "/tests/a-flakey.test.js"]%D
          at Context.<anonymous> (test/unit/index.test.js:103:38)

    What am I doon wrong?

    nvm :)
    Anyone know if I can use protractor-flake if I am using mocha as my framewrk? @NickTomlin
    Wim Selles

    Hi @faisalAvizia

    There is a PR for that one, see NickTomlin/protractor-flake#60. I asked @NickTomlin to merge it.

    So yep, there is a possibility, you just need to wait ;-)

    Thanks @wswebcreation . Hopefully it'll get merged soon. I am starting a new project and would love to use this.
    Trisha Chetani
    Trisha Chetani
    i need a suggestion on how to implement
    have added this file named as flake

    const protractorFlake = require('protractor-flake');
    // skip first two passed args (node and self)
    let protractorArgs = process.argv.splice(2);

    protractorPath: 'node_modules/.bin/protractor',
    maxAttempts: 2,
    parser: 'multi',
    nodeBin: 'node',
    protractorArgs: protractorArgs
    }, (status, output) => {

    but how does this file which read my config file , on success ful run it will know no need to re run the test
    and on failure it will re run the test