|
Automating Software Configurations
in Test Labs
by Roberta Gonzalez, EdenTree Technologies
and Rob Spade, SASA Solutions
The importance and pervasiveness of software are driving all types of networking
and computing systems. Systems developers are finding that with the right
software they can leverage Moore’s law to deliver more functionality than a
dedicated ASIC with comparable performance.
Witness, for instance, the evolution of the PBX from a hardware-based system to
a software application that can be installed on a general-purpose computer. This
trend is driving a greater need to automate the software test lab with an eye on
the most time-consuming and manual processes in the test process: the management
and configuration of the growing number of computers used in the testing.
Consequently, highly trained employees and a significant amount of hardware and
software are underutilized for hours or days depending on the complexity of the
system. Add to this the difficulty of managing this process in collaborative
test labs. These labs may depend on off-site and offshore colleagues, partners,
or customers who must duplicate this testing infrastructure or wait for onsite
personnel to configure the systems.
At the same time, technology and economic pressures have amplified the
ever-present pressures to do more with less and shorten test cycles, spawning an
increased interest in a solution to the configuration challenge.
But new capabilities now are available to target automating the configuration
process, dramatically reducing the amount of time required and opening up the
test lab in a secure way for offsite developers and customers. The potential
impact of these systems can be dramatic. To help explain the functionality and
the results, a large maker of IP multimedia subsystem (IMS) software that
automated its performance and interoperability test lab will serve as an
example.
The Test Environment
The company’s IMS software is designed to deliver multimedia and voice services
across wireline and wireless networks. The software is deployed on Sun Netra
servers, and the test equipment and ancillary devices include a mix of
electronic equipment and computer-based applications from multiple vendors.
The company faced the same challenges as would any software company developing
an application: support for a large number of computers, frequent new releases,
heterogeneous test environments, and complex computer configurations.
With each new product release, the configuration problem grows exponentially.
Figure 1 shows the series of sequential steps, ordered from the bottom to the
top, that are required. Each step is assigned an average percentage of the
effort required for the entire process.
|
Figure 1: Typical Computer Setup Process for a Software Test |
The absolute time required for the complete process to be performed will vary
from platform to platform, operating system to operating system, and application
to application. Typically, it requires anywhere from four to eight hours
although it’s not unheard of for a configuration of a complex system to take
several days. Any lab that must test computer-based products faces the
accelerating problem of testing new releases against a growing number of
existing releases and possible deployed environments, each requiring a new
server configuration.
In the example IMS test lab, the engineers must verify interoperability and
performance levels for a collection of integrated products, requiring a computer
setup process that is repeated as many as 224 times per release test cycle. The
process includes the following elements:
1. Approximately 14 computers, along with various switches and traffic
generators and analyzers, make up a specific test topology, and each is reset to
a known state just prior to running required tests. The computers are configured
with standard product releases for a given deployment scenario. Different
traffic patterns are generated to emulate real-world conditions.
2. The process is repeated for each of approximately 16 deployment scenarios.
These variations include all the possible product release permutations the new
release will be required to interact with once released into production.
3. Whenever a new release of a product has completed development, about six to
eight times per year, steps one and two are repeated.
In this lab, each test must be run on a specific test topology with known
configured elements. Testers use traditional automated test scripts to start the
test and initiate traffic patterns with test suites executing overnight or over
an entire weekend in some cases.
However, the testers must first set up the test topologies and manually restore
complete computer configurations, including operating systems, patch levels,
applications, product provisioning parameters, and test data, that are required
for each automated test script. It takes a tester an average of three hours to
perform the configuration steps for each computer in the topology.
Once the tests are completed, the results are queried to determine if the test
has produced the expected results. While the test-results investigation is
performed, the topology is maintained, and no other tests are performed. This
process can consume two days of a tester’s time for each topology.
Key Challenges
In a lab environment where products and systems are growing in complexity, it
becomes exceedingly difficult to perform all mandatory and desired testing
within reasonable cost and in the required amount of time. As a result, the cost
of testing increases, and the testing capacity and quality decrease.
There are several key challenges in our example lab:
• Manual and time-consuming computer configurations are bottlenecks to automated
test scripts. Each test topology setup requires two testers working manually for
2.5 workdays before any automated test can be run.
• The lab is significantly underutilized due to lengthy, manual reconfiguration
periods.
• There are no efficiencies of scale. There is a linear relationship between the
number of new product releases and the time and cost of testing each required
topology.
• Remote collaborators in an offshore facility depend on their onsite
counterparts to set up test topologies for them.
Solution Alternatives
To complete test cycles within given time constraints, the options are to add
test beds and testers, reduce the quality of testing, or improve the test
process so current resources can handle increased test throughput.
 Improving the test process is the desired approach. But how?
A new technology is becoming a popular choice for quickly creating computer
configurations in production environments: virtualization. However, in a test
environment where lab-configured systems must exactly replicate deployed
systems, virtualization is not an option. The behavior and performance of
virtual machines are not identical to those of physical machines, which
eliminates virtual machines as a viable option for improving the process of
configuring the DUT.
For the test lab, the required characteristics of a solution are the following:
• Configure computer equipment with operating system, applications, and test
data.
• Provide a programmatic interface to support test automation scripts that
select the configuration target and provisioning options and perform the
configuration process.
• Store and set up any number of saved configurations on demand.
• Not require any deviation from the installation and upgrade procedures when
creating a new configuration.
• Complete transparency to the product under test. Do not incorporate any
additional software in the environment under test.
• Able to scale as new computers are added to the test environment.
• Not require the use of the test topology for test-results analysis, allowing
other tests to be performed while analysis takes place.
• Be cost-justified by measurable savings in current test resource costs.
Automated Computer Configurations
A new class of solutions available today automates the computer setup process.
It has a flexible architecture that accommodates high-end UNIX, Linux, and
Windows platforms. The solution incorporates a software application managing
storage infrastructure made up of a storage device for the computer image
archive and a storage area network (SAN) or LAN switch that connect to each of
the managed computers and the storage device. Computers to be managed are
connected to the solution using a host bus adaptor (HBA) or preboot execution
environment (PXE) card.
Figure 2 represents the deployed solution architecture for the example test lab.
|
Figure 2. Architecture for Automated Reconfiguration of 14 Computers |
The testers in the example lab initially configure each of the 14 computers as
usual. Each computer is connected to the Fibre Channel network through an HBA,
which provides standard disk I/O capability over a serial fiber bus. Using the
configuration management graphical user interface (GUI) or application
programming interface (API) commands, the testers save the computer
configurations to the storage array that, in this case, provides 7 TB of disk
storage.
 For each test topology, the configurations are automatically restored or pulled
back onto the computers in the topology through API commands integrated with the
test scripts. The solution also allows for configurations to be mounted from the
storage device and used virtually. However, this method is not used for the
computers that host the system under test because the virtual state affects the
performance of the system. This method is used only for ancillary devices in the
test topologies.
Important features of the solution for this lab include the following:
• Restores the mirrored disk drives that are in the servers hosting the product
under test. The solution reconstructs the mirrored contents of all the disk
drives on each server with the specified images and then reboots all the servers
using the new images. The solution restores mirrored drives simultaneously.
• Configures multiple computers at the same time. In an environment where a test
topology includes multiple computers, the capability to perform concurrent
configurations exponentially increases the benefits of the solution.
• Restores a computer image to a different computer than that used to create the
image. This feature provides several important process advantages, including the
capability for one computer to be used to create master images for other
computers of the same type and new computers that are added to the solution.
Additionally, computers dedicated to test-results analysis can be used to access
mounted (used virtually from the storage array) images. This aspect frees the
computers in the primary test environment to be used for continued testing.
In summary, the key benefits of this automated solution for the IMS lab include
the following:
• The manual process of configuring computers is replaced by a completely
automated process that eliminates the previous two-day bottleneck.
• Off-site and offshore personnel can remotely manage and control computer
configurations.
• New release configurations are easier to create by recalling an existing
configuration, performing required changes, and saving the new variant.
• Test results can be saved and recalled on any computer, allowing analysis to
be delayed or performed on alternative systems while automated tests continue
uninterrupted.
• Existing computer resources are optimized and leveraged, allowing additional
tests to be performed using the existing test investment.
Quantitative Results
The automated configuration solution has achieved two key process improvements
for the IMS lab. First, the amount of time needed to perform each test has been
greatly reduced by decreasing the amount of time required to reconfigure the
computers and set up the test topology. Secondly, the capability to perform
off-line analysis for each test further shortens the total test cycle as shown
in Figure 3.
|
Figure 3: Performance Improvements of Post-Solution vs. Pre-Solution Test Cycle |
Quantitative benefits for the lab can be summarized as follows:
• The average setup time for each computer was reduced from three hours to 12
minutes or just 7% of the time.
• The capacity of the lab to support release test cycles increased from three to
eight or more releases per year.
• The amount of equipment required to support the company’s release development
schedule decreased from two sets of equipment to one, eliminating a $700,000
test bed and a $150,000-per-year tester to configure the equipment.
Conclusion
As the reliance on sophisticated software applications increases, more
manufacturers will find their product development schedules increasingly at the mercy of the software test lab and
its manual configuration process. As the IMS example shows, test lab automation
offers a way out of this scenario without compromising test performance as a
virtualization-based solution would. Automation is customized to the needs of
software test labs and increases productivity and product quality while reducing
costs.
About the Authors
Roberta Gonzalez co-founded EdenTree Technologies in 2002. During 20-plus years
of experience in the communications networking industry, she has held vice
president and director level roles in marketing and business development at
Spirent Communications, Hewlett-Packard (Agilent), and Network General. Ms.
Gonzalez also has delivered technical seminars throughout the United States,
Europe, Asia, Australia, and Latin America and holds a B.S. in engineering from
Duke University. EdenTree Technologies, 810 Lawrence Dr., Suite 210, Newbury
Park, CA 91320, 805-499-4555, e-mail:
rgonzalez@edentreetech.com
Rob Spade, the founder and principal consultant for SASA Solutions, has more
than 30 years of experience in network test engineering. He has held CTO, vice
president, director, and principal development positions at Optimems, Motorola,
Idea, OMEX, and NCR. Mr. Spade also has been affiliated with the IEEE Network
Management Committee, the TIA Committee on Satellite Transmission of IP and ATM,
and the Corporation of Open Systems and co-founded the Communication Test
Automation Group. SASA Solutions, P.O. Box 51807, Phoenix, AZ 85076,
480-361-8482,
e-mail: rspade@edentreetech.com |