WIP -- feat: parallel scheme examples

[iamkenos]

Branch for #175

Breaking Change

• --parallel-scheme=scenario will run scenarios in parallel on Scenario level. e.g. 1 process for each Scenario / ScenarioOutline as opposed to 1 process for each example in the outline (current behavior as of writing)

Feature

• new scheme --parallel-scheme=examples which is set to default - this does what --parallel-scheme=scenario before the change.

General Approach Taken

• if parallel scheme is scenario,

  • we execute the tests in parallel on scenario level; we do this by passing all the scenario outline lines that matches the execution tags as part of the behave arguments.
    • ex:
      • Scenario 1, Examples 1, 2 and 3: behave features/foo.feature:21 features/foo.feature:22 features/foo.feature:23
      • Scenario 2, Examples 1, 2 and 3: behave features/foo.feature:31 features/foo.feature:32 features/foo.feature:33
    • in the example above, you get 2 processes running in parallel, 1 for each scenario and will run all examples from within the respective process

• if parallel scheme is examples,

  • we execute the tests in parallel on examples level; we do this by passing each of the scenario outline line that matches the execution tags as part of the behave arguments.
    • ex:
      • Scenario 1, Example 1: behave features/foo.feature:21
      • Scenario 1, Example 2: behave features/foo.feature:22
      • Scenario 1, Example 3: behave features/foo.feature:23
      • Scenario 2, Example 1: behave features/foo.feature:31
      • Scenario 2, Example 2: behave features/foo.feature:32
      • Scenario 2, Example 3: behave features/foo.feature:33
    • in the example above, you get 6 processes running in parallel, 1 for each example. - this is the current behavior of parallel-scheme=scenario

Pending Issues

• Breaking tests, we'll need to fix those
• Still couldnt figure out the reporting part. 🙃

[iamkenos]

@hrcorval , i ended up not using scenario_outline_line - im not sure what that was intended for in your POC.

but just looping over the scenario lines works. (see the loop on line 623) then i just pass a single item list as scenario_lines=[line]

i can see behave runs the tests properly this way but i can't figure out why the execution_codes and json_reports generated is incomplete when i do this approach.

given a feature file:

Feature: Example Table Tags

  Background:
    Given I have something

  Scenario Outline: Foo From Example 1
    Then I expect something

    Examples:
      | data |
      | 1    |
      | 2    |

  Scenario Outline: Foo From Example 2
    Then I expect something

    Examples:
      | data |
      | 1    |
      | 2    |

when i run parallel-scheme=examples

then i expect this on

Actual

This only contains information on the last example on the example tables.

[iamkenos]

would appreciate if you can provide some guidance here. thank you! i also updated the PR description to describe the general approach of the implem

[hrcorval]

Hi @iamkenos , Sorry for late reply, I have been really busy and I could not take care of the library changes as I would like to.
I have been reviewing in details the changes in the PR and everything looks good so far. I still need to analyze why we are not getting the expected execution_codes and json_reports, just give me more time to go over this.
Regarding the scenario_outline_line argument, this is not necessary anymore, as I created the branch as fast as possible to provide feedback on the expected approach, I forgot to remove that. So that variable is not needed anymore in the proposed solution.
I'll create a new release candidate as you suggested, just to start playing with that version while we continue iterating on the improvements.
Thanks a lot for all the improvements you are providing, we all benefit from that :)

[iamkenos]

Hi @iamkenos , Sorry for late reply, I have been really busy and I could not take care of the library changes as I would like to. I have been reviewing in details the changes in the PR and everything looks good so far. I still need to analyze why we are not getting the expected execution_codes and json_reports, just give me more time to go over this. Regarding the scenario_outline_line argument, this is not necessary anymore, as I created the branch as fast as possible to provide feedback on the expected approach, I forgot to remove that. So that variable is not needed anymore in the proposed solution. I'll create a new release candidate as you suggested, just to start playing with that version while we continue iterating on the improvements. Thanks a lot for all the improvements you are providing, we all benefit from that :)

No worries at all, I'm patiently keeping an eye out for updates and I'm real excited about this. Thanks so much for your time! 🤝

[hrcorval]

Hi @iamkenos, I hope you are doing fine. If you agree, I believe we are in conditions of merging your PR into version 4.1.0, and continue making all the improvements/changes using that release version.
We did a new release Today with version 4.0.9, which contains lot of fixes for Windows OS. So, We created the branch 4.1.0 to start iterating over this PR.
Please, let me know your thoughts. Thanks!

[iamkenos]

Hi @iamkenos, I hope you are doing fine. If you agree, I believe we are in conditions of merging your PR into version 4.1.0, and continue making all the improvements/changes using that release version. We did a new release Today with version 4.0.9, which contains lot of fixes for Windows OS. So, We created the branch 4.1.0 to start iterating over this PR. Please, let me know your thoughts. Thanks!

Hey @hrcorval , it's been a while. Yes that's fine, better that we don't rush this and make sure it's prod ready before we ship. I'm still flushing out possible issues and use cases for this, on and off. Progress has been slow since it's been hectic at work and the holidays are just around the corner.

If you can share a rebased version of this branch that includes the latest 4.0.9 release, that would be real great. I'll continue working from there.

Again, thank you so much for the time and patience. Advanced happy holidays to the behavex team! 🎄