SEI Load Test Planning Process

Mark McWhinney
Portata, Inc.

Introduction

As with any project, the planning stage is the most critical to a project's success. For load testing projects, the planning stage is even more critical due to technical and project challenges.

Load testing involves more parts of an organization than any other activity in application development. The people involved often include business owners, developers, architects, QA, database administrators, system administrators, network administrators, data center support staff, various managers, third party organizations and sometimes customers.

Load testing projects usually occur as the last step in application development just before applications are deployed. In practice with schedule slips there is usually very little time to plan and execute a load test.

We create models of user behavior making many assumptions about what the users will do, how they do it and how often they do it. We create large result sets with no definitive method for determining the correctness or meaning of the results.

To handle these problems with load test planning, Portata developed the SEI Load Test Planning Process. It is an empirical model that evolved over several years from observations in our load test consulting practice.

The planning methodology is a step by step procedure that examines six key areas of information that are critical to the success of a load test project. This paper describes these six steps and provides tips for implementing the planning activities.

The Planning Process

Often development and functional testing phases are late, so load testing gets squeezed in to the schedule just before application deployment. As a result, there is usually little time to plan and execute a load test.

To handle the time compression, we designed the SEI Load Test Planning Process such that it can be executed within one day for most projects. The morning of the first day is spent collecting information. The afternoon is spent documenting the plan and analyzing it for omissions and errors.

The planning day begins with a meeting in the morning to determine the project’s goals and gather information. The meeting attendees include representatives from the application’s development team, deployment team, and production support as well as people who are most familiar with how the end users are likely to use the application. People from different parts of the project may have very different ideas as to what the goals are or what load/stress/performance is all about. It is important that they all contribute to the planning process and have a common understanding of what the project will achieve.

During the meeting, the facilitator (usually the load tester) writes the goals, user types, etc. on a whiteboard to keep the meeting focused. This will also help the documentation effort later. It is best to use a whiteboard instead of flip chart paper, so that we can modify or remove information as needed during the meeting.

Once we collect all of the information from the planning meeting and any subsequent meetings, we write the information into the Load Test Plan document.

The Load Test Plan is a simple document that contains only the core information needed to build and execute the load test – namely the information gathered in the planning meeting. The six main sections of the document correspond to the six areas discussed in the meeting, so that the documentation effort is basically just copying the information from the whiteboard into the specification document.

The plan includes only the technical and business issues not the schedule and other project-specific information that should be in the application’s development plan or overall test plan. Likewise, the plan does not contain a lengthy dissertation on the types of testing as is often found in a functional test plan.

When we complete the plan, we distribute it to all stakeholders for review then correct it as needed.

The plan review and update activities can take a day or two beyond the initial planning day, but while that is happening, we can begin the technical investigation and test environment set up.

The Six Steps

The intent of the SEI Load Test Planning Process is to

make information gathering quick and painless
make plan document generation easy
make the plan objective and verifiable
be simple enough for technical and non-technical people to understand and participate

To this end, we identified six areas for which we need to gather, analyze and document information. The six areas are the project goals, users, use cases, production environment, test environment and load test scenarios. We refer to the areas as “steps” in that the planning process goes through the areas in a specific order. Each of these six steps is described below.

Goals

The load test project goals are the business needs of the project, not the technical objectives.

We start the planning by defining the load test project goals with the over all business needs not the technical objectives. While questions such as “will we max out the CPU under load” are important, they are only the contributing issues in the bigger questions such as “we will be able to support our business activities” such as selling products on the Web or processing financial transactions.”

So that we can objectively verify that a project has met its goals, we express the goals in the form of questions that can be answered objectively with either a yes/no or a number. For example, “How many users can the system handle” can be answered objectively with a number. “How will the application do under load conditions” is not good in that there is no objective definition of what it means to “do”.

Goals should also have constraints to bound the questions. It is not enough to ask how many users can run on an application; we must also specify what the acceptable response time performance is. For example, “how many users can run on the application while maintaining an 8-second response time or better for 90% of their transactions” is a good, objective question.

Users

Users are any person or process that may generate load or use system resources.

We create the list of users by brainstorming all the types of people and processes that could possibly affect the application or system. We start by defining the types of primary end users of the application or system such as purchasers, claims processors, and sales reps. We then add other types of users such as system administrators, managers, and report readers who use the application or system but are not the primary users.

We add types of non-human users such as batch processes, system backups, bulk data loads and anything else that may add load or consume system resources. We include significant applications that will be collocated on the same hardware that will compete for system resources such as CPU and network bandwidth. What qualifies as a significant application is somewhat subjective. An application server or database is usually significant. A print spooler generally is not.

Once we identify all the types of people and processes that can put load on the application or system, we go through a winnowing process to eliminate the user types that are unlikely to put appreciable load on the system. This could be due to too few users of a particular type and/or too little system activity. For example, system backups performed at night when there is little or no user activity may be winnowed out. System administrators, managers and background processes are often eliminated.

Since the number or mixture of users may vary from one scenario to another, we do not attempt to define the number of users until the Scenario step (the last step).

Use Cases

Use cases are tasks or business processes that users perform such as entering an order, generating a report, or approving a request.

For each type of user we identified in the previous step, we identify the tasks that the user performs. Again, we use a brainstorming approach to come up with a good list.

As before, we go through a winnowing process to eliminate the use cases that are unlikely to put appreciable load on the system. This could be due to tasks that are performed too infrequently or tasks that do not consume much system resources.

During the morning planning meeting, we just list the use cases for each user type. If time permits, we define the steps in each use case. If time is too short as is often the case, we work with the user representatives after the meeting to define the steps for each use case.

Since the number or mixture of use cases may vary from one scenario to another, we do not attempt to define the number of users executing a particular use case until the Scenario step (the last step).

Production Environment

The production environment is the configuration of hardware and software components in the data center that support the application when it is in operation.

In that performance and capacity of an application is significantly affected by the hardware and software components on which it executes, it is important to understand what they are and what their individual capabilities are.

During the morning planning meeting we draw a diagram of the production environment on the whiteboard. The diagram includes the servers, clients, and network elements. Each item on the diagram includes speed, capacity, IP address and name, version numbers and other significant information. For example, a server would include the model number, OS version, number of CPUs and their speed, and memory. Databases would include version number and type of connections e.g. ODBC, SQL*Net, etc.

Often the not all the information is available during the morning meeting, so we gather and diagram as much of the information as time permits then work with the production support staff after the meeting to get the details as needed.

The information gathered in the meeting is documented in tabular format in the Load Test Plan document. If time permits, we draw a diagram of the environment using a tool such as Visio and annotate each component with the information. (The diagrams that the production support staff typically has are usually too crowded to annotate with the information that we need.)

Test Environment

The test environment is the configuration of hardware and software components in the test lab that support the application when it is being tested. It also includes the hardware and software components needed by the load generation and measurement tools.

As with the production environment we draw a diagram of the test environment on which the test will run. It includes the same elements as the production environment along with the equipment needed for the load test software. There may be multiple variations of the test environment depending on the business goals and the scenarios.

It is important for the test environment to be as similar to the production environment as is possible to be able to get meaningful performance results. Often for financial reasons it is not possible to duplicate the exact production environment. If so, during the planning meeting we look at various alternatives for setting up the test environment. They may include the use of the production environment after normal usage hours, using some production tiers, leasing or renting equipment, short-term licenses from software vendors, etc.

The lead time to plan and set up the test databases can be significant, so we start the planning in the initial meeting. It is important that the databases be set up with the same amount of data in the same proportions as the production environment as that can substantially affect the performance.

Scenarios

A scenario is a description of the test environment and the use cases, number of users per use case and their frequency.

For each scenario, we

Select the use cases to include
Determine how many instances of each use case will run concurrently
Determine how often the use cases will execute per hour
Select the test environment

We list the users, their use case, and frequency in a table for each scenario.

Typically there is one scenario for each business goal.

Portata

Portata is the premier provider of affordable load test services. Our experienced staff can measure your user capacity and identify your application bottlenecks at a substantially lower cost.

Portata has refined its knowledge of the critical success factors for planning and managing application performance management projects. Our experienced test project managers and engineers maintain deadlines within budget, while managing the risks inherent in supporting today’s complex business processes.

For more information about load testing or other services, contact Portata by e-mail at info@portata.com or visit our web site at www.portata.com.

Also available from Portata is the Critical Success Factors for Load Test Projects white paper.

Author

Mark McWhinney is President of Portata in Mountain View, CA. Mark is the author of several papers on load testing and on software quality assurance processes.