Tests

Tests#

How to start#

Find something to start with#

You can either add a new test case that you think is missing in an existing plan, or you can develop new automated test for a feature or its aspect that is not covered yet.

Some tips as to what tests would come in handy can be found on this Trac page listed under “Tests” component.

You can also create a test plan for a new feature - to do that, just pick an RFE ticket and check whether new test plan is needed. To check this, there are test related fields that indicate the status:

  • Test case - links to a test plan/case in case it is already done

  • Test by - QA contact the issue is assigned to

  • Test coverage - can have four values:

    • yes - the test is already completed, the Test case and Test by fields should already be filled

    • no - the test is deemed unnecessary or unreasonably laborious to create (but you can still create it, if you really want to)

    • wanted - the test is deemed necessary, but not done yet. Check the Test by field - if it’s filled, the test plan is in progress, if it’s empty, it’s free to take

    • empty field - this issue needs review

Update Trac ticket#

Should you choose to contribute a test with an existing ticket, click Modify ticket and select accept in box Action. The owner of a ticket will be changed to your name and next status of the ticket will be assigned.

If you’re working on yet unassigned test for RFE ticket, simply fill out the Test by field.

If you prefer to work on a test with no existing ticket, create a new ticket for the purpose.

Get the source#

See Contribute code section.

Design a test or a test plan#

The first step to create a new test is designing a test plan. Prepare a test plan proposal using the test plan template. If the test plan is designed for a new feature (RFE ticket), publish the test plan on www.freeipa.org/page/V4/feature_name/Test_plan, where feature_name is the name of the new feature you’re testing. Then send the test plan for comments and approval to freeipa-devel mailing list.

Create a test or test plan#

The test should be written in Python and using pytest.

When creating a test, you can follow

Prior to starting the implementation#

  • Check if the patch/test effort is not duplicated by someone else and/or already covered

  • Check if the patch/test requires coverage in other component areas, e.g WebUI, upgrade, migration, backup-restore, Trust, DNS, external-CA, Multiarch, etc.

  • Check if the change take into consideration availability of packages for all distributions

Prior to committing#

  1. Check the logic of the patch/test and its relevance to the original issue

  2. Check for spelling errors and common mistakes like (not exhaustive):

    1. Exception handling (too broad or too specific)

    2. Unconditional xfails

    3. Hardcoded values

    4. Relying on sleep when there are other mechanism to determine readiness (e.g. Selenium’s WebDriverWait)

    5. Checking the proper logs using their respective paths

    6. Commented out code

  3. If needed, include PR-CI definitions

  4. Execute the bundled code checks using make fastlint. This will check for linting errors and PEP8 standards

Submit a patch#

When the test is completed, submit a patch much the same way as when contributing code. Then the patch goes through code review process. If the changes in the patch are approved, they get pushed to the source repository. The process is also summarized bellow.

Committing (doesn’t apply for the temp commit)#

  1. Follow the commit message guidelines listed in Commit message requirements

  2. Signoff and sign your commit

  3. All commits must include Pagure link

  4. Include separate temp commit (see below)

(lowercase) ipatests : short summary within 80 chars (no dot)
This is the particular reason for a change and or addition of a new test. Be as specific as possible. Use imperative language (fix bug, not fixed bug nor fixes bug) and present time.
Fixes (or Related): Pagure ticket link
Signed off by: (use –signoff and -S when committing)

Note: “Fixes” is used when providing a fix for the raised issue. “Related” is used when providing a test for the fix.

Temp commit#

When making changes to the tests, also include a separate temp commit. The purpose of the temp commit is to execute the new changes in the PR-CI.

The template of the temp commit is available at temp_commit.yaml

Integration tests#

When changing e.g. test_sudo.py, the author would need to edit Line#71 to read

test_integration/test_sudo.py

Line#L68 should read

``RunPytest ``

WebUI tests#

When changing, e.g. test_loginscreen.py, edit Line#71 to read

``test_webui/test_loginscreen.py ``

and change Line#L68 to read

RunWebuiTests

Also make sure, the topology selected reflects the topology needed for the test (e.g. Line#L87 requires 3 replicas, therefore the topology would be

*master_3repl_1client

You can see the list of available topologies at the top of temp_commit.yaml

Last but not least, link the PR-CI definition with the temp commit definition, i.e.

$ ln -sf ipatests/prci_definitions/temp_commit.yaml .freeipa-pr-ci.yaml

executed from the repository root

Note: Don’t execute just the changed test case/test class, but rather the whole suite.

Creating Pull Request#

  1. Provide the same title and summary as in the commit message

  2. Add yourself to the list of Assignees

  3. Set up the proper labels for backports and state of the work (WIP, Needs review)

    1. Ask for a review by assigning a reviewer, if known in advance

Reviewing the pull request#

  1. Review all the steps from Prior to starting the implementation and Prior to committing

  2. Check the results of PR-CI and make sure the intended test scenario was actually executed by checking the tests and the output of executed commands.

  3. Provide comment/s with your suggestions and/or final statement. Be professional and respectful in your comments. When the # review is finalized, provide an appropriate label (e.g. ACK).

Merging#

  1. Copy the link to the successful temp commit and include it in the comments

  2. Delete the temp commit within the PR, so that only the main commit with the patch remains.

  3. Don’t use the Merge button within GitHub! If you have the appropriate project permissions, use the ipa tool to merge the pull request, e.g.

``ipatool pr-push 3406 -r reviewer1 -r reviewer2  -B ipa-4-8 -B ipa-4-7 ``

If you don’t have the permissions, e.g. as an external contributor, merging will be taken care of, usually by the reviewer.