OpenStack QA
Walkthrough of Tools, Code and Processes
Most Important Things to Remember
Participate
The more you participate, the more people you meet and the more knowledge you gain The more knowledge you gain, the more you can share that knowledge with others
Ask questions Don't be afraid to ask questions
There's no such thing as a stupid or silly question
Be public Prefer public questions and discussion over private
email threads Cast a wide net, catch more fish
Things all contributors should have
Launchpad account
Upload your SSH keys to Launchpad Gerrit imports your SSH keys from Launchpad Subscribe to main openstack mailing list
IRC
[Link] #openstack and #openstack-dev Typically 340+ on #openstack, 140+ on #openstackdev Best place to find help #openstack-meeting for weekly meetings Wednesdays at 16:00 UTC
devstack installed locally
Running with devstack
Always run your test code against a real system devstack makes things easy for you Running [Link] "resets" a development OpenStack environment for you Point tests like Tempest at your local devstack environment Great for one-off tests or monkey-testing
Running devstack ([Link])
$> ./[Link] <snip lots of output!> horizon is now available at [Link] keystone is serving at [Link] examples on using novaclient command line is in [Link] the default users are: admin and demo the password: 4f9a953e98ee57d922d9 This is your host ip: [Link] [Link] completed in 79 seconds.
Tempest
Tempest is a project that contains functional integration tests intended to be run against actual OpenStack deployments Contains a functional testing framework that uses the unittest2 and nose Python libraries as a base Test cases execute a series of API calls against OpenStack service endpoints and validate the response from the endpoint Uses a simple config file that describes test environment
Easy way to generate a test config
jpipes@librebox:~/repos/tempest$ ./tempest/tools/conf_from_devstack -D ../devstack/ -o etc/tempest. conf Output file already exists. Overwrite? [y/N]Y jpipes@librebox:~/repos/tempest$ cat etc/[Link] [nova] host=[Link] port=5000 apiVer=v2.0 path=tokens user=admin Bolded values api_key= 4f9a953e98ee57d922d9 were gathered tenant_name= admin from the devstack ssh_timeout=300 build_interval=10 configuration build_timeout=600 [environment] image_ref= 3712ca26-f926-4725-9132-08ec1f6e452e image_ref_alt=4 flavor_ref=1 flavor_ref_alt=2 create_image_enabled=true resize_available=true
Running Tempest
Run Tempest with nosetests
jpipes@librebox:~/repos/tempest$ nosetests -v tempest List of all extensions ... ok <snip> ---------------------------------------------------------------------Ran 61 tests in 2283.166s FAILED (SKIP=2, errors=5) jpipes@librebox:~/repos/tempest$ nosetests -v [Link].test_flavors The expected flavor details should be returned ... ok flavor details are not returned for non existant flavors ... ok List of all flavors should contain the expected flavor ... ok Detailed list of all flavors should contain the expected flavor ... ok ---------------------------------------------------------------------Ran 4 tests in 0.722s OK
Running a single test case...
jpipes@librebox:~/repos/tempest$ nosetests -v [Link].test_flavors: FlavorsTest.test_list_flavors List of all flavors should contain the expected flavor ... ok ---------------------------------------------------------------------Ran 1 test in 0.502s OK
Top tip: Separate test module from test class with colon, not dot!
What errors look like...
====================================================================== ERROR: The server should be rebuilt using the provided image and data ---------------------------------------------------------------------Traceback (most recent call last): File "/home/jpipes/repos/tempest/tempest/tests/test_server_actions.py", line 79, in test_rebuild_server [Link].wait_for_server_status(rebuilt_server['id'], 'ACTIVE') File "/home/jpipes/repos/tempest/tempest/services/nova/json/servers_client.py", line 150, in wait_for_server_status raise [Link](message) TimeoutException: Request timed out Details: Server 8bd71a0c-64d7-4ffc-aeb0-227bf4bbb80c failed to reach ACTIVE status within the required time (600.0 s).
Machine-readable output with xUnit
jpipes@librebox:~/repos/tempest$ nosetests -v --with-xunit [Link].test_server_actions The server's password should be set to the provided password ... ok The server should be power cycled ... ok The server should be signaled to reboot gracefully ... ok The server should be rebuilt using the provided image and data ... ERROR The server's RAM and disk space should be modified to that of ... ERROR The server's RAM and disk space should return to its original ... ERROR <snip> ====================================================================== ERROR: The server's RAM and disk space should return to its original ---------------------------------------------------------------------Traceback (most recent call last): File "/home/jpipes/repos/tempest/tempest/tests/test_server_actions.py", line 112, in test_resize_server_revert [Link].wait_for_server_status(self.server_id, 'VERIFY_RESIZE') File "/home/jpipes/repos/tempest/tempest/services/nova/json/servers_client.py", line 150, in wait_for_server_status raise [Link](message) TimeoutException: u'Server d7f1d3e3-1d3a-4c22-b3ef-1fe6660fe0b5 failed to reach ACTIVE status within the required time (600.0 s).' ---------------------------------------------------------------------XML: [Link] ---------------------------------------------------------------------Ran 6 tests in 1918.131s FAILED (errors=3) jpipes@librebox:~/repos/tempest$ cat [Link]
<?xml version="1.0" encoding="UTF-8"?> <testsuite name="nosetests" tests="6" errors="3" failures="0" skip="0"> <testcase classname="[Link].test_server_actions.ServerActionsTest" name="test_change_server_password" time="22" /> <testcase classname="[Link].test_server_actions.ServerActionsTest" name="test_reboot_server_hard" time="22" /> <testcase classname="[Link].test_server_actions.ServerActionsTest" name="test_reboot_server_soft" time="22" /> <testcase classname="[Link].test_server_actions.ServerActionsTest" name="test_rebuild_server" time="617"> <error type="[Link]" message="u'Server ac4a2f89-f905-42e5-ba14-801a3146cf9c failed to reach ACTIVE status within the required time (600.0 s).'"> <![CDATA[Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/unittest2/[Link]", line 340, in run testMethod() File "/home/jpipes/repos/tempest/tempest/tests/test_server_actions.py", line 79, in test_rebuild_server [Link].wait_for_server_status(rebuilt_server['id'], 'ACTIVE') File "/home/jpipes/repos/tempest/tempest/services/nova/json/servers_client.py", line 150, in wait_for_server_status raise [Link](message) TimeoutException: u'Server ac4a2f89-f905-42e5-ba14-801a3146cf9c failed to reach ACTIVE status within the required time (600.0 s).' ]]> </error> </testcase> ... </testsuite>
Tempest Code Walkthrough
Tempest directory structure
$src_dir/ etc/ <-- contains sample Tempest config tempest/ common/ <-- common code like rest_client services/ nova/ <-- client for compute tests/ <-- test cases tools/ <-- helpers scripts
Tempest test case class
Contains a set of related tests
For instance, a test case may test operations that list servers in various ways
Resources shared by methods of the test case should be created in the setUpClass() method and destroyed in the tearDownClass() method
setUpClass() method is used to create resources (in this case a pair of server instances and a few snapshot images) that are referenced by test methods in the test case What's wrong with this picture? hint: [Link]
net/tempest/+bug/899701
Pattern you will see is that instead of making HTTP calls directly, you use the various client objects attached to the tempest.
[Link]
Tempest test case methods
Test methods should validate a related set of actions Test methods may be decorated with the [Link] decorator to indicate that a method contains a particular type of test If a test method creates any resources, it should always clean up after itself
Tempest test case methods (cont'd)
Test methods should not modify any shared resources -- doing so may create ordering dependencies When using various assert methods, include a corresponding failure message that is descriptive and provides the tester with details they may need to diagnose an issue
Example: writing a good test case
Add a test case that does the following: 1. Create a new server from a base image 2. Change the name of the server 3. Validate the new name of the server appears when showing details about the server
def test_scenario(self): name = rand_name('server') resp, body = [Link].create_server(name, self.image_ref, self.flavor_ref) resp, body = [Link].update_server(server['id'], name='newname') [Link](200, [Link]) resp, server = [Link].get_server(server['id']) [Link]('newname', server['name'])
@attr(type='smoke') def test_update_server_name(self): """The server name should be changed to the the provided value""" name = rand_name('server') resp, body = [Link].create_server(name, self.image_ref, self.flavor_ref) resp, body = [Link].update_server(server['id'], name='newname') [Link](200, [Link]) resp, server = [Link].get_server(server['id']) [Link]('newname', server['name'])
@attr(type='smoke') def test_update_server_name(self): """The server name should be changed to the the provided value""" name = rand_name('server') resp, server = [Link].create_server(name, self.image_ref, self.flavor_ref) [Link](201, [Link]) # Update the server with a new name resp, body = [Link].update_server(server['id'], name='newname') [Link](200, [Link]) # Verify the name of the server has changed resp, server = [Link].get_server(server['id']) [Link]('newname', server['name'])
@attr(type='smoke') def test_update_server_name(self): """The server name should be changed to the the provided value""" name = rand_name('server') resp, server = [Link].create_server(name, self.image_ref, self.flavor_ref) [Link](201, [Link]) server_id = server['id'] # Update the server with a new name resp, body = [Link].update_server(server_id, name='newname') resp_code = [Link] fail_msg = ("Failed to update server %(server_id)s. " "Got HTTP response code %(resp_code)d with " "body %(body)s") % locals() [Link](200, [Link], fail_msg) # Verify the name of the server has changed resp, server = [Link].get_server(server_id) fail_msg = ("Failed to find updated server name. Expected 'newname' " "Got %s") % server['name'] [Link]('newname', server['name'], fail_msg)
@attr(type='smoke') def test_update_server_name(self): """The server name should be changed to the the provided value""" name = rand_name('server') resp, server = [Link].create_server(name, self.image_ref, self.flavor_ref) [Link](201, [Link]) server_id = server['id'] # Update the server with a new name resp, body = [Link].update_server(server_id, name='newname') resp_code = [Link] fail_msg = ("Failed to update server %(server_id)s. " "Got HTTP response code %(resp_code)d with " "body %(body)s") % locals() [Link](200, [Link], fail_msg) # Verify the name of the server has changed resp, server = [Link].get_server(server_id) fail_msg = ("Failed to find updated server name. Expected 'newname' " "Got %s") % server['name'] [Link]('newname', server['name'], fail_msg) # Clean up after ourselves... [Link].delete_server(server['id'])
Submitting Code
Code Submission Guidelines
Be consistent
Consistency in your code -- style, comments, documentation, etc -- shows you care
Respond in a timely manner to reviews Ensure commit messages are proper A Launchpad bug number or blueprint is referenced
when appropriate First line is short description of patch More detailed description of patch follows Do not put successive "fixup messages" in commit message
Basic contribution process
1. 2. 3. 4. 5. 6. 7. 8. 9. Assign yourself to an unassigned bug Create local topic branch Make code changes Run Tempest against standing environments If all tests pass, commit code changes Write descriptive commit message Propose for review Address any review comments Amend commit and re-propose
Common Scenarios
Use git-review to propose patch
Scenario: You have assigned yourself to a bug, created or modified code that addresses the bug, run tests and now want to propose your changes for review. Solution: Commit local changes and then call git review
Basic code submission
1) git commit -a 2) Write a descriptive commit message 3) Save and close your editor 4) git review
Use --amend for small fixups
Scenario: You have pushed a patch and gotten one or more reviews that call for some minor fixups. You make the fixes on your local branch and need to push the changes for review. Solution: Commit local changes, but amend the original commit.
Amending a commit after fixes
1) git commit -a --amend 2) Optionally edit the commit message 3) Save and close your editor 4) git review
Oops! You forgot to use --amend!
Scenario: You made fixups based on review and then did a git commit -a and then git review, but you forgot to use --amend. This generated a new patchset to Gerrit instead of updating the original commit. Solution: First, take a deep breath and remember that every contributor has done this before. After that, Abandon the incorrect new changeset in Gerrit and then use git reset to undo your mistake.
Undoing your --amend mistake
1) Go to your newly-created Gerrit changeset and click the "Abandon" button to mark the changeset as obselete 2) git reset HEAD^ 3) git commit -a --amend 4) Optionally edit the commit message to indicate any major changes you may have made during fixups 5) git review
jpipes@librebox:~/repos/tempest$ git commit -a # Ooops! Forgot to --amend! [bug912596 08439dc] Bad Jay! 1 files changed, 1 insertions(+), 0 deletions(-) jpipes@librebox:~/repos/tempest$ git reset HEAD^ Unstaged changes after reset: M tempest/[Link] jpipes@librebox:~/repos/tempest$ git commit -a --amend [bug912596 9e320c0] Fixes LP Bug #912596 - image_ref_alt not found Author: Jay Pipes <jpipes@[Link]> 3 files changed, 89 insertions(+), 1 deletions(-) create mode 100644 tempest/tests/[Link]
Use git stash to save changes
Scenario: You have a bunch of uncommitted code changes locally. You want to pull code that just made it into trunk. Solution: Use git stash to save your uncommitted code changes, pull changes from master, rebase your local branch against master and then reapply your stashed code changes.
git git git git git git
stash checkout master pull checkout <BRANCH> rebase master stash pop
Pull a colleague's code from Gerrit
Scenario: You are reviewing a colleague's code and want to pull the code to your local machine for testing. Solution: Use git review -d <PATCH_NUM> to clone the remote branch locally and automatically put you into the branch.
Here is the patch number...
Troubleshooting
If [Link] fails to build and gives errors about domains failing to be removed, manually check the status of VMs that may have been left around after a test run using virsh list --all Manually virsh destroy <INSTANCE_NAME> and virsh undefine <INSTANCE_NAME> all instances and try rerunning [Link]