Mobility Performance Testing
by Richard Lu, Azimuth Systems
The rich availability of Wi-Fi connectivity
and the universal demand for constant access to voice and data
communications are driving consumer and enterprise demand for
converged Wi-Fi/cellular applications and services. Under the
banner of fixed-mobile convergence (FMC), these applications
promise better access to voice and data services as well as
lower communications costs.
Successful delivery of converged services is
based on the assumption that wireless IP networks and
802.11a/b/g/n devices can deliver the underlying quality of
service necessary to guarantee a satisfying end-user experience.
Critical FMC performance metrics that directly affect user
experience are measured by voice quality, dropped calls, and
battery life.
These user-experience metrics are directly
influenced by Wi-Fi metrics such as data rate, error rate,
packet loss, and roam time, which define mobility performances
like range, roaming, and hand-in/hand-out. The increasing
adoption and deployment of this technology have created the need
for a reliable means to ensure the delivery of carrier-grade
voice over Wi-Fi to end users.
As a result, extensive testing is required to
guarantee that FMC devices and the networks on which they
operate can deliver the necessary performance. For mobility
performance testing to be effective from quality, coverage, and
cost perspectives, test methodologies must accurately recreate
the real world in an efficient, repeatable, cost-effective, and
scalable manner.
What Is Fixed-Mobile Convergence?
FMC means different things to different
groups. For end users, it is the promise of quality fixed
services and applications of voice, video, and data being
delivered seamlessly over mobile wireless networks to handsets
or endpoint devices. For infrastructure vendors, FMC represents
a large commercial opportunity to deliver back-end integration
products and services, and for service providers, it is an
opportunity to develop additional revenue from existing cellular
and Wi-Fi network infrastructure investments. To drive increased
adoption by end users, the industry must deliver compelling FMC
applications and services beyond what are currently available.
One major benefit that FMC provides is the
convenience of having one device and phone number for both the
mobile network and the home network. With the voice call
seamlessly transferred from the cellular network to the home or
corporate network, users enjoy continuous communications and
benefits from improved coverage of the Wi-Fi network while
reducing cellular minute usage.
Location-independent access to data
applications also represents a huge draw for potential users.
Whether stationary or mobile in any location covered by a
cellular or Wi-Fi network, users will have access to
Internet-based applications such as e-mail, Web browsing, and
online services like banking, news, and entertainment. This
ability to use data applications where and when they want will
enable significant improvements in personal and professional
productivity.
Consumer Satisfaction
In addition to seamless voice and data
communications, sustainable consumer demand for FMC services
also will be driven by the capability to provide cost savings
not available from alternative Wi-Fi and cellular-only services.
Existing cellular services have established user experience
expectations that new converged services must meet or exceed.
For service providers, this means FMC must deliver carrier-grade
voice and data services characterized by:
• Optimal voice quality.
• Few or no dropped calls.
• Reliable and high throughput data
connectivity.
• Long handset battery life in the standby
and active states.
• Seamless voice and data roaming/handoff.
Wi-Fi Mobility Usage Scenarios
The growth in consumer demand for long-range
cellular network services has been primarily driven by
improvements in mobile voice services and value-added data
services like text messaging and Web access. FMC broadens the
range of these services indoors where cellular coverage may be
less reliable. By moving coverage onto the Wi-Fi network, users
have access to potentially much higher-speed services.
However, this Wi-Fi connectivity must at
least sustain the mobility and quality of service that cellular
users have come to expect. Additionally, converged
Wi-Fi/cellular services must support the same or better quality
voice and data services while the user is in motion and
transferring among the networks. This mobility service requires
adequate performance in three key usage scenarios: Wi-Fi range,
Wi-Fi roaming, and hand-in/hand-out network handover.
Wi-Fi range, or coverage, is defined as
quality voice and data services while the user moves closer to
and away from a Wi-Fi Access Point (AP) and is a significant
measure of Wi-Fi performance. In contrast to cellular
technology, Wi-Fi technology varies the transmission data rate
to minimize packet errors.
A Wi-Fi transmitter on the AP or client uses
dynamic rate adaptation algorithms to control the transmission
data rate on a packet-by-packet basis. These algorithms consider
many different network and environmental variables including
receive signal strength and packet error rate in deciding to
increase or decrease the transmission rate.
If these algorithms are poorly implemented,
movement within a Wi-Fi AP will significantly impact data
throughput and voice quality. Proving that the rate adaptation
algorithms have been implemented properly requires extensive
range testing of the client and AP devices.
Wi-Fi roaming, the capability to provide
quality voice and data services while moving between APs, is
another critical measure of Wi-Fi device performance. Both
cellular and Wi-Fi networks use algorithms to transfer
connectivity from one infrastructure device, such as cellular
base station or Wi-Fi AP, to another as connection conditions
require.
Cellular networks utilize roaming algorithms
that are implemented on the base station and managed by the
network, which make roaming decisions dependent on real-time
network conditions like base-station loading or base-station
service outage. Wi-Fi networks, in contrast, use roaming
algorithms implemented on the client, which make roam decisions
without consideration of real-time network conditions.
This means that Wi-Fi clients could make
decisions to roam to an AP that is overloaded or not functioning
properly. This would result in significantly reduced data
throughput and
voice quality or the session being
terminated completely.
In addition to the differences in the
implementation of the roaming algorithm, cellular and Wi-Fi
networks execute the roam differently. Cellular networks roam
using a make-before-break approach. This establishes the
connection between the handset and the base station being given
control of the session prior to terminating the connection
between the handset and the base station that is giving up
control of the session.
In contrast, Wi-Fi clients use a
break-before-make approach to roaming, in which the client
terminates connectivity with one AP prior to establishing
connectivity with a new AP. A poorly implemented roaming
algorithm on
the client can result in a significant amount of time to
establish the connection with the new AP or even failure to make
the connection entirely, which can severely impact data
throughput, voice quality, and call continuity. Given the
differences in the roaming algorithm implementation and
execution, maintaining data throughput, voice quality, and
satisfactory call continuity requires extensive testing.
Transferring session connectivity between
Wi-Fi and cellular networks also is much more complex than a
Wi-Fi-to-Wi-Fi roam or a cellular-to-cellular handover. With the
differences between cellular and Wi-Fi handover algorithm
implementations, engineering efficient Wi-Fi-to-cellular
handovers requires complex changes to the back-end cellular
network as well as significant changes to decision-making
algorithms. With this increased complexity, extensive testing
must
be done to validate the impact of handover on throughput, voice
quality, and call continuity.
Real-World Wi-Fi Mobility
When discussing the performance of Wi-Fi
devices, the impact of the real-world environment in which the
devices operate must be recognized. Unlike a cellular network,
which is the sole occupier of the licensed RF spectrum in which
it operates, a Wi-Fi network runs in an unlicensed spectrum that
it shares not only with other
Wi-Fi networks but also potentially with other RF networks like
Bluetooth or even other RF devices such as cordless phones and
microwave ovens.
The presence of RF devices and Wi-Fi networks
competing for spectrum causes RF interference that can
significantly impact the performance of a Wi-Fi network. In
addition, solid obstacles such as walls and furniture as well as
the movement of objects like vehicles can create RF signal
conditions, known as multipath and fading, that impact the
performance of Wi-Fi devices.
Most FMC solutions will use the Internet, a
nondedicated IP network, as a primary carrier of the voice data,
which can directly affect voice quality as the traffic load
varies. To maintain
the best possible FMC services, converged Wi-Fi/cellular
handsets must deliver the best Wi-Fi mobility performance in all
these different types of real-world conditions.
Performance Testing
To determine the impact of Wi-Fi performance
on user experience, it is important to first identify the
critical mobile performance scenarios that directly impact the
user experience. Table 1
identifies critical mobile performance scenarios.
Table 1. Critical Mobile Performance Scenarios
Wi-Fi mobile scenarios establish the
fundamental performance metrics including data rate, packet
loss, error rate, and roam time that must be tested. A set of
essential tests for the evaluation of the Wi-Fi mobility
performance can be developed (Table 2).
Table 2. Required Wi-Fi Mobility Performance Tests
Effective Test Methodologies
In over-the-air (OTA) testing, engineers
replicate the actual conditions of the environments in which
devices will operate. This is accomplished by renting or buying
empty office buildings and homes or even testing on live
networks. Mobility and roaming are tested by placing Wi-Fi
devices on mobile carts, moving these carts to various locations
in the test space, and manually configuring tests and recording
test results at each location.
Due to the uncontrollable nature of OTA
environmental conditions, the majority of testing is done
manually. The effectiveness of Wi-Fi mobility performance
testing using OTA methods also is limited by two critical
factors:
• The time-consuming manual test setup and
execution typical of OTA tests limit the capability of this
testing method to scale.
• Consistent, repeatable test measurements
are nearly impossible in open-air environments, limiting the
capability to reliably repeat the tests in the future.
In addition, the RF interference may vary with each test iteration even at the same location,
which could make reproducing results and issues nearly impossible.
The alternative to OTA testing is using a
controlled RF environment such as a screen room that filters out
external RF interference. This method is expensive because of
the large installation and maintenance costs of the screen
rooms. In addition, the size of the screen room severely limits
the effectiveness of testing distance, roaming, and mobility.
A more advanced method of controlled RF
testing involves device isolation. In device isolation, each
test-bed device is placed in an individual isolated enclosure
and connected via cables to programmable RF attenuators,
combiners, and switches. This test methodology replicates the
Wi-Fi network in a controlled, cabled environment that
stabilizes the RF connection by removing the variability of
open-air systems.
The device-isolation approach provides a
completely controllable RF environment to conduct repeatable
mobility testing. Test solutions that use a controlled, cabled
RF environment eliminate the need to design, build, and maintain
homegrown test beds and costly RF screen rooms.
Another benefit of this methodology comes
from the programmable test bed and tools that enable automated
test configuration and execution. To analyze the effect of
mobility on both device and network performances, users can
automatically configure any network device and dynamically
position any network node. Automated test configuration also
allows for repeatable test execution over time.
Using programming, scripts can be created
which require little human intervention and automatically run
multiple iterations of different configurations in a fraction of
the time required for manual testing. This repeatability reduces
the time spent on quality assurance and benchmark test processes
as well as time to market and testing costs.
Test scalability is an additional important
benefit of the device-isolation approach. If the controlled RF
environment is properly architected, system designers can scale
Wi-Fi testing from a single device to the entire network. Users
can configure an entire Wi-Fi network and provide system-level
testing of actual APs, clients, and other wireless devices.
Networks can be tested under a variety of traffic and client
load conditions. Client and traffic load emulations enable the
development of test setups that recreate a busy network
environment for the DUTs.
Lastly, this approach provides the capability
to test Wi-Fi mobility performance. Engineers can assess the
impact of one or a combination of real-world conditions,
including RF multipath and fading, background Wi-Fi traffic, RF
interference, and IP network delay, on the mobility performance
of 802.11a/b/g/n devices.
Another critical test is conducted by systems
engineering groups within service-provider organizations that
will use these same test scenarios to validate interoperability
of devices from different suppliers, benchmark performance of
different configurations to make purchasing decisions, and
certify devices for deployment. For the engineers tasked with
selecting FMC handsets that will operate on service-provider
networks, an effective performance benchmarking process provides
a means of performing an apples-to-apples comparison of the
Wi-Fi mobility performance delivered by FMC handsets from
different manufacturers.
Summary
Converged Wi-Fi/cellular promises benefits
ranging from continuous access to voice and data applications
and services to reduced costs. Sustainable consumer demand will
be dependent on carrier-grade FMC services that deliver reliable
and fast data throughput, good voice quality, few dropped calls,
and long handset battery life.
As cellular-only services provide voice and
data services with carrier-grade quality, the capability to
deliver FMC services of similar quality is expected. This will
directly be impacted by the performance of Wi-Fi-enabled devices
as users move within and between Wi-Fi networks as well as
transfer between Wi-Fi and cellular networks.
Critical FMC performance metrics that
directly affect user experience are functions of Wi-Fi data
rate, error rate, packet loss, and roam time. Among the several
test methodologies available today, the most effective method
for testing the mobility performance of 802.11a/b/g/n devices
uses a controlled, cabled RF environment. In addition to
providing accurate and repeatable test results, such an approach
can use programmable test tools and external test components to
analyze mobility performance in controlled, real-world network
conditions and leverage automation to reduce test time and
overall costs.
About the Author
Richard Lu is the Wi-Fi product line manager
at Azimuth Systems and has 10 years of experience in test and
measurement for the telecommunications and networking
industries. Before joining Azimuth Systems in 2003, Mr. Lu was
at Coppercom, where he started the company’s design verification
lab, and was an applications engineer at Zarak Systems, which
later was acquired by Spirent Communications. Mr. Lu graduated
from the University of California with a B.S. Azimuth Systems,
31 Nagog Park, Acton, MA 01720, 978-263-6610, e-mail:
richard_lu@azimuth.net