- 1 -
An Approach to Testing X Window System Servers at a
Protocol Level
Michael Lee Squires
Len Wyatt
Sequent Computer Systems, Inc.
15450 SW Koll Parkway
Beaverton, OR 97006
(503) 626-5700
ABSTRACT
An approach to testing display servers for
the X Window System is presented which depends
only on an underlying communications mechanism.
The server tester is intended to provide
coverage for those server specifications which
cannot be tested from the Xlib level. It also
ensures that the basic facilities needed to run
the Xlib tests are operational. All testing is
performed by sending messages to the server and
evaluating the responses from the server.
1. INTRODUCTION
The purpose of this document is to present a set of
requirements for a server tester and to outline a software
structure which will satisfy those requirements. This
document is intended to guide not only the server tester
developers, but other X Window System developers and testers
also.
1.1 Background
There are a variety of interfaces, services, and
protocols specified as part of the X Window System. These
are implemented by a set of components which may be
produced by different organizations, running on different
hardware and using different operating systems.
In order to promote a standard, it is not sufficient to
merely publish an interface specification. Examples of
software which implement the standards and which illustrate
the use of the services must be made available. Test
procedures and test suites which verify that specific
component implementations adhere to the specifications must
be defined and made generally available.
- 2 -
The MIT X effort has addressed the interface
specification, software implementation, and example use. It
has not, thus far, addressed the issue of testing and
specification compliance.
In the interests of promoting the X Window System as an
industry standard, a number of organizations have formed a
consortium for the purpose of defining and generating an
appropriate set of test tools and procedures for X. This
document addresses a specific facet of X testing which has
been identified by the consortium.
2. PROBLEM STATEMENT
2.1 Testing Goals
2.1.1 General
The goals for testing X implementations are to:
-- promote portability of applications and application
libraries by ensuring a consistent set of interfaces
across disparate computer systems and operating
systems. That is, if we have an application program
which compiles and runs on system A, we should be able
to transfer the source to system B and, within the
constraints of language and operating system
portability, compile and run that program
satisfactorily. Since most applications are currently
written to the procedural interface provided in C by
the Xlib library, this interface is the first target of
the testing effort for portability.
-- promote interoperability of clients and servers which
may be running on disparate computer systems. That is,
if we have a client which runs on system A and uses the
display server available on system B, then that client
should be able to operate successfully with a server
running on system C (ignoring protocol extensions and
client device dependencies).
2.1.2 Server Tester Goals
The primary goal of the server tester is to address
those areas of interoperability which cannot be adequately
covered by tests at the Xlib level. A secondary goal is to
facilitate the Xlib testing by ensuring that the basic
server functions are operational. Another secondary goal is
to provide a low-level diagnostic tool which would be useful
to server developers.
2.2 Definitions
- 3 -
The following definitions form a basis for
understanding the requirements of the server tester.
We define an interface as an implementation dependent
functional boundary between two adjacent components. Taking
Xlib as an example, the function names, calling sequences,
and data types necessary to interact with Xlib are its
interface.
We define a service as an implementation independent
set of capabilities that a particular layer of software
provides. Again using Xlib as an example, we could define
an equivalent level library for another language which
provided the same services via a different interface.
We define a protocol as the specification of
interaction between two peer processes. This specification
consists of:
-- a message catalog
-- sequencing rules for message exchange
-- the specifications of the actions of the individual
processes in response to the message sequencing
When we examine the X Window System, we see that the X
protocol specification encompasses three components: the
client, the server, and the display hardware. Thus a
complete test of a server's adherence to the protocol
specification would include observing its effect on the
display hardware as well as its ability to adhere to the
rules for message sequencing.
We also notice that the "XLib Interface Specification"
actually consists of both an interface and a service
specification. Furthermore, the service specification
depends on the adherence of a particular server to the
protocol specification.
Because Xlib was designed to provide fairly direct
access to the underlying X protocol, many of the messages
defined in the protocol specification and their
corresponding actions have a direct mapping at the Xlib
level. This concept of a mapping between the Xlib
specification and the protocol specification is a key
element in determining the requirements for a server tester.
3. APPROACH
3.1 General
Because many of the actions specified in the protocol
- 4 -
specification can be tested at the Xlib level, because the
Xlib level will be more heavily used by the majority of X-
based programmers, and because DEC is mounting a substantial
effort to rigorously test the Xlib interfaces and services,
we will defer the bulk of the testing of server actions to
the Xlib testing effort.
We will design and implement a server tester which will
concentrate on testing those features of the protocol which
are not testable at the Xlib level or which are
prerequisites to the Xlib level testing. Thus, we will test
the ability of the server to accept all legal message types
and respond appropriately. We will ensure that server
capabilities which the Xlib testing will depend on work
within some bounds (e.g ensure that the server can send a
pixmap to the client upon request). We will test to ensure
adherence to the canonical byte stream of the protocol,
independent of the host byte sex or compiler structure
packing.
3.2 Determining Specific Requirements
We have developed a decision tree which allows us to
determine which server actions should be addressed by the
server tester.
The first decision in the tree is "is the mapping of
this action (and the stimulus to cause it) to the Xlib level
direct enough to allow it to be tested at the Xlib level?"
An example of an action which can be tested at the Xlib
level is drawing a line within a window. Initializing the
server connection with an incorrect protocol version number
is an example of a test which cannot be performed at the
Xlib level.
If the mapping is direct enough to allow testing at the
Xlib level, then we need to ask "is this capability
fundamental enough that the Xlib tests cannot be reasonably
performed if the capability is deficient?" An example of a
fundamental capability is the server's ability to send error
responses.
If the mapping is not direct enough to allow the
testing at the Xlib level, then we ask "can the action be
inferred from other data available at the Xlib level?" An
example of something which could be inferred is the
initialization of the default color map. By reading the
color map immediately following opening the display, an Xlib
level test can determine if the color map was properly
initialized.
- 5 -
+-----------------------------------------------------------+
| Decision #1 Decision #2 Result |
+-----------------------------------------------------------+
|Is the Can the NO TEST |
|mapping NO action be YES DON'T TEST |
|direct inferred? |
|enough to |
|test at Is the NO DON'T TEST |
|the Xlib YES action YES TEST |
|level? fundamental? |
+-----------------------------------------------------------+
We will apply this set of decisions to the protocol
specification to generate specific tests to perform in the
server tester. Generating this list is a specific
deliverable on the development schedule (see section 6).
4. SPECIFICATIONS
4.1 General Requirements
The server tester must satisfy the following
requirements:
-- It will not use X as a basis for its user interaction.
(This provides a new server developer the ability to
run the tester using standard character terminals.)
-- It will verify that the server can accept all legal
message types.
-- It will verify that the server will reject invalid
message types.
-- It will perform all tests which are derived from
applying the decision criteria in section 3.2 to the
protocol specification. The complete list of tests is
presented in section 4.2.
-- It will be portable to different computer systems and
operating systems. (That is, any machine/operating
system dependencies will be isolated. We are not
committing to port it to any/all systems.)
-- It will adapt the tests to different display hardware.
(That is, a mechanism for modifying test parameters
based on display characteristics will be provided.)
-- It will provide the capability to continue a test suite
after errors are encountered in a particular test.
-- It will provide the capability to test a server's
ability to deal with clients running on different byte
sex hosts. (That is, the tester will be capable of
- 6 -
'masquerading' as a host with a different byte sex than
it really has. This allows an organization to more
completely test a server with a single host system.)
Additionally, we will provide user documentation and
internal documentation for the server tester.
4.2 Specific Tests
This subsection is currently TBD. The detailed list of
tests is the first deliverable discussed in section 6.
5. SOFTWARE STRUCTURE
The basic approach to developing the server tester and
the test cases which use it is to provide a layered set of C
routines which constitute the bulk of the test mechanism.
Individual test cases will be implemented as C programs
which make use of this layered set. We will explore
development of some sort of front-end tool (e.g.
preprocessor, interpreter) which will facilitate test case
generation, but given the resource and schedule constraints
we are unwilling to commit to that at this time.
We will structure the test cases as a series of C
programs, each one designed to test a specific capability or
set of capabilities. This allows incremental development of
the test cases, selective use of individual tests during
development or diagnoses, and ease of test suite
configuration via use of command scripts.
The software will consist of 5 software layers:
-- Communication Manipulation - lowest level - contains
the machine and operating dependent software -
responsible for byte swapping, buffering, and ensuring
the canonical byte stream.
-- Message Handling low level utilities (which look very
similar to the various internal utilities within the
current Xlib implementation) - expose more of their
internal data structures than Xlib does to provide test
case writers the capability to "damage" messages
-- Message Generation - essentially a subset of the
current Xlib with many parameters omitted in favor of
their defaults - (i.e.
Generate_Standard_Create_Window_Request would be a
candidate routine)
-- Adaptation - all routines necessary to adapt the tests
to specific display parameters - (e.g.
Generate_Safe_Rectangle_Set)
- 7 -
-- specific test cases
Note that this software really consists of two major
components - the "tools" (levels 1-4) and the test cases
themselves. This division is used to generate the
milestones in section 6.
+--------------------------------------+
| TEST CASE |
+--------------------------------------+
| | | |
| | | |
v | | |
+------------+ | | |
| ADAPTATION | | | |
+------------+ | | |
| | |
| | |
v | |
+----------------------+ | |
| MESSAGE GENERATION | | |
+----------------------+ | |
| |
| |
v |
+-------------------------+ |
| MESSAGE HANDLING | |
+-------------------------+ |
|
|
v
+--------------------------------------+
| COMMUNICATION MANIPULATION |
+--------------------------------------+
6. DEVELOPMENT SCHEDULE
+---------------------------------------------+
|Deliverable Date |
+---------------------------------------------+
|Complete list of server tests 7/17/87 |
| |
|Tools + initial set of test cases 10/9/87 |
| |
|Remainder of test cases 12/18/87 |
+---------------------------------------------+