Usability Evaluation

The main purpose of the tests was to assess the limitations of ink-enabled user interfaces. SketchUML is not only an e-learning tool that facilitates learning of the UML syntax, it also a novel approach for only ink-input accepting computer interfaces. The testing session was intended to identify the limitations of ink-only input and evaluate the usability of non-hybrid interfaces (rather than hybrid ink & point-and-click-based interfaces) and the effectiveness of SketchUML as an e-Learning tool for UML diagrams.

Participants. A number of undergraduate students (7 seniors, 3 juniors and 1 sophomore) that were enrolled in a software engineering class have been tested in this study. The students were all novices to UML or had intermediate knowledge, but no student was an UML expert. The participants' experience with computers and user interfaces ranged from 5 to 15 years (M = 12,8, SD = 2.5). No student received any reward for participating in the study and their participation was voluntary. All but two students were new to Tablet PCs and the SketchUML software.

Devices. The environment was a classroom setting with other students, desks and typical supplies. The lighting was also that of a typical classroom. The equipment being used was 6 Hewlett Packard Compaq TC4200 Tablet PC Laptops, with 1.7GHz Intel Centrino Processors, 512MB Ram, running Windows XP Tablet PC Edition. The test software was the latest stable release of SketchUML. For testing, the laptops have been switched to Tablet Mode and participants were asked to use the Pen as the input device.

Procedure. After the participants signed informed consent forms and filled out a pre-test demographic questionnaire, they were given a short introduction to the interaction with Tablet PCs and the software SketchUML. The introduction consisted of a brief overview of the features of SketchUML and the features of the Tablet PC hardware that was used and was presented using a standard classroom projector. All participants were introduced at the same time so every participant received the same introduction. An effort was made to randomly divide the participants into two groups, so that 6 participants were in group one and 5 participants were in group two. Each participant was given a task sheet with a task description; the task the students had to complete was different for both groups. The members of group one were given a class hierarchy description and were asked to draw the class hierarchy using the Tablet PCs, on which SketchUML was running. Group two was given the same class hierarchy description and a pre-loaded class diagram that was drawn incorrectly and does not meet the hierarchy description. The erroneous parts of the diagram were labeled accordingly (e.g. The class name “wrong class” for a class that must be deleted), with exception of incorrectly placed connectors, as connector labeling support is currently not fully supported in SketchUML. Participants in group two were asked to critique the pre-loaded diagram using the Teacher Mode that is built into SketchUML. Figure 1 shows the experimenters ideal solution to the task of the first group. Figure 2 shows the pre-loaded diagram the participants in the second group were asked to critique and Figure 3 shows the ideal solution to the task of group two.