Identifying an effective framework for usability evaluation of Computer Supported Collaborative Learning System in Educational settings

Huang, Eileen
Parry, Dave
Senapathi, Mali
Item type
Degree name
Master of Computer and Information Sciences
Journal Title
Journal ISSN
Volume Title
Auckland University of Technology

A review of previous studies has highlighted a gap between Computer Supported Collaborative Learning (CSCL) system usability evaluation (UE) in educational collaborative teaching and learning settings and groupware UE in general settings. The purpose of the research was to investigate this area further to identify a framework for CSCL system UE in educational settings. The framework should consist of the criteria that reflect the key features of CSCL system usability. Such a framework will then be capable of identifying the advantages and the disadvantages of a CSCL system's usability and its usability problems or issues. Having considered a number of existing UE frameworks, a framework for CSCL system UE in educational settings was developed. The framework consists of 24 criteria grouped into six dimensions: Effectiveness, Efficiency, Collaborativity, Error tolerance, Universal accessibility, and Satisfaction. The methodology for this research was designed as a two-year Case study (completed part-time) with six stages. It included user testing, one-to-one interviews, and questionnaires as the UE methods. The Open Journal System (OJS), a free online academic journal publishing system, was chosen as a collaborative learning (CL) system to test the developed framework. In this study, OJS had been set up for the Collaborative Computing (CC) paper in the School of Computing and Mathematical Sciences (SCMS) at the Auckland University of Technology (AUT). 18 participants took part in the study, and all except two academic staff were recruited from the postgraduate students in the CC class. The two academic staff joined in the Pilot study and 16 students/participants were divided into two groups – a student testing group and a follow-up group. A 60-minute testing session was completed by each participant in the Pilot study group and the student testing group. This included a pre-test questionnaire, an asynchronous collaborative task (a peer review on OJS), a post-test questionnaire, and a one to one interview. 10 testing sessions were conducted. The participants in the follow-up group only completed the two questionnaires. Data was collected through a pilot study and the two groups mentioned previously. The study has found that the defined criteria in the developed framework are important to UE and this framework is able to identify advantages and disadvantages of OJS. Limitations and problems in the research were identified. Future research should ensure that a larger sample size is used and user types are diversified, and that the framework's criteria can be best tested on several CSCL systems which support synchronous teamwork. Further investigation could be focused on how to determinate the weight for each of the six dimensions so that the framework can be improved and developed into an adaptable and effective assessment tool suitable for evaluating the system usability of a range of CSCL systems in educational settings.

Computer Supported Collaborative Learning (CSCL) , Usability evaluation framework , Educational collaborative teaching and learning settings , Usability evaluation (UE) , Framework dimensions , Collaborativity , Effectiveness , Efficiency , Error tolerance , Universal accessibility , Satisfaction , Collaborative Computing , User testing
Publisher's version
Rights statement