Critical Success Factors for Load Test Projects

Mark McWhinney
Portata, Inc.

The intent of load testing is the same as functional testing or any other type of testing: to find problems before the end users do. However, load testing is not the same as functional testing. It has its own unique issues and its own critical success factors. This white paper describes some of the critical success factors that are unique or are particularly meaningful to load testing.

Planning

As with any project, the planning stage is the most critical to the project's success. For load testing projects, the planning stage is even more critical. Load testing involves more parts of an organization than any other activity in application development. The people involved often include business owners, developers, architects, QA, database administrators, system administrators, network administrators, data center support staff, various managers, third party organizations and sometimes customers. Planning, coordinating and communicating must be done well to ensure the project's success.

To further complicate the planning process, load test activities and load test results tend to be nebulous by their nature. Without good planning, it is all too easy to reach the end of a load test, find that the answer is "42", and realize that you really do not understand what it means.

Goal Setting. The first step in a successful load test planning process is to define the specific goals of the project in objective terms. We find it useful to express the goals in the form of questions whose answer can only be a number or a yes/no. For example, "how many users can run order entry while maintaining 8 second response time or better for 90% of their transactions?"

SEI process. The goal setting step is the first step in the overall planning process that produces the load test plan. Portata uses the SEI Load Test Planning process. See our white paper on this process.

Staff Skills

Load testers need a variety of skills including many skills that functional testers usually do not have.

Experience. Load testers need in-depth knowledge about load testing and how to use the load test tools. The load test team should have at least one member who has run several successful load test projects.

Test Tool Certifications. Some tool vendors have load test tool certifications. Though the certifications are not a substitute for experience, they indicate that the load testers at least understand how the tools work.

Application. Load testing does not require as much knowledge of the application as functional testing does. In functional testing the testers cover every function, but in load testing the testers will execute just a few common paths through the application. However, the load testers should have some operational knowledge of the application to be tested or at least have access to a knowledgeable application user.

Application Usage. The intent of load testing is to emulate what real users will be doing with the application when it is deployed. Therefore, the load testers must also have access to someone who knows how the application is actually used in production or can at least make an informed estimate. That person will supply information about how users typically accomplish a particular task, how often tasks are performed, and how many users perform the tasks.

Systems. Load testers must have a good working knowledge of the software and hardware components of the application. These components may include web servers, application servers, database servers, operating systems, networks and network elements such as load balancers. The load testers need not have "guru" level knowledge of each of the components but should have operational knowledge and an understanding of the performance issues associated with the components. For example, a load tester should know what multi-way joins, indexes and spin counts are and what affect they have on a database server.

Protocols. Load tests are usually driven at the protocol level instead of the GUI level. This allows load generation machines to emulate many users. Load testers should understand the protocol(s) used at the point where the load test tool interfaces with the application to be tested. For web applications this is typically between the web browser and the web server. For load testing web applications a load tester should know both HTTP and HTML. Other commonly used protocols are SQL*Net, ODBC, and DCOM.

Communication. Load testers will work with many parts of an organization to coordinate activities, schedules and resources. Load testing is not a heads down coding exercise. Daily interaction with a variety of people requires good oral and written communication skills as well as good people skills.

If the load testers do not have sufficient communication and people skills, the project will require more management time to handle the planning, coordination and communication.

Support Staff

The load testers must have access to DBAs, application server administrators, network administrators, and system administrators. They are used on an as-needed basis to set up the test environment, provide access to the components, solve problems, and monitor the load test runs.

Load Test Lab

Test Servers. Management usually will not allow load tests to be performed on productions systems since the high load levels would interfere with production activities. Also, production activities may skew the load test results.

Unlike a lab for functional testing, a load test lab must be configured the same as the production environment in terms of component capacity and speed, otherwise the load test may not be accurate or even meaningful. For example, if a production machine has four CPUs then the test lab should be configured with a four CPU test machine.

Often it is cost prohibitive to duplicate the full production system in the test lab. Some possible workarounds are to use the production equipment in the test lab prior to deployment or to use the production infrastructure during off-business hours. Equipment can be rented or leased but there is typically a long leadtime for acquisition and set up so plan accordingly.

Databases. The database(s) in the test lab must be preloaded with either a copy of production data or dummy data that is similar in size and content to the production data. Databases that are too small will tend to give erroneously fast results and can obscure table scan and index problems.

Load Test Tools. Load test tools range from the low-end URL pingers to commercial-grade multi-protocol load generation and monitoring systems. Most applications that are worth load testing include business critical functions and dynamic data, which require the higher end tools.

The important features to have in a load test tool are:

Ability to parameterize data.
Ability to capture dynamic data and use on subsequent requests.
Application infrastructure monitoring.
Support for the application's protocols.

The load test tools are not inexpensive, so allocate sufficient budget during the budget planning process.

Load Test Monitoring

Response time measurements indicate whether you have a problem and how bad it is, but they do not tell you why you have a problem, what the root cause is, or how to get rid of it. For that information you need to monitor the individual components in the application architecture. The components may include a web server, application server, database server, network and network elements such as firewalls, operating systems and server hardware.

The better load test tools have built-in monitors which measure the various components in real-time and report results verses the load levels applied at the time. If the load test tool that you will be using has the capability to monitor, the tool may require that monitoring agents be installed on the components and/or authorizations be granted to access the components in real time. If the tool does not have this capability, you must make provisions for staff to manually monitor the key components and integrate the results.

Portata

Portata is the premier provider of affordable load test services. Our experienced staff can measure your user capacity and identify your application bottlenecks at a substantially lower cost.

Portata has refined its knowledge of the critical success factors for planning and managing application performance management projects. Our experienced test project managers and engineers maintain deadlines within budget, while managing the risks inherent in supporting today’s complex business processes.

For more information about load testing or other services, contact Portata by e-mail at info@portata.com or visit our web site at www.portata.com.

Also available from Portata is the SEI Load Test Planning Process white paper.

Author

Mark McWhinney is President of Portata in Mountain View, CA. Mark is the author of several papers on load testing and on software quality assurance processes.