Information Technology for Assessing Student Learning

Information Technology for Assessing Student Learning

Salvatore Valenti

DIIGA - Universita’ Politecnica delle Marche

Ancona, Italy

valenti@inform.unian.it

Introduction to the Special Series

Assessment plays a central role in the educational process. In the last few years, the scientific commu-nity has seen the development of new assessment methodologies and novel tools applying Information Technology to the assessment process of student learning.

According to our findings (Valenti, Cucchiarelli, & Panti, 2002), more than fifty commercial off-the-shelf tools for Computer Based Assessment are currently available on the market. This number may rise considerably if we take into account the research tools currently under development and the tools for delivering assessment embedded within most electronic learning environments.

There are a number of reasons for this growing interest by research and industry in applying IT to the assessment of student learning. First, there is the increase in the number of students attending institu-tions of higher education. Most universities offer a portion of their required freshman classes in lectures of hundred or more students. Bjedov (cited in Valenti, Cucchiarelli, & Panti, 2001) reports about teach-ing programming in a class whose enrolment varies between 360 students in spring semesters and 940 students in fall semesters. At the University of Ulster class sizes in excess of 100 are common for popu-lar entry level modules in computer programming. Similar numbers are reported in Italy at the Universi-ties of Ca' Foscari (Venice) and Ancona, and in Brazil at the University of Pernanbuco leading to the conclusion that the rising size and the diversity of teaching groups in high level education is a world-wide problem (Valenti et al, 2001).

Second, the growth of credit-bearing distance learning offerings and enrolments at accredited, degree-granting colleges and universities has been astonishing in the last few years. During the academic year 1997–98, approximately 1.6 million students were enrolled in credit-bearing distance learning courses (whether electronic, television-based, or print- and-mail-based, and including both synchronous and asynchronous instruction) in degree-granting post-secondary colleges and universities in the United States. That year, ,000 college-level credit-bearing distance-learning courses were offered in 1,680 institutions. Thirty-five states currently operate virtual universities or participate in a regional virtual university, typically created by existing degree-granting colleges and universities (Valenti, Panti, & Leo, 2003). This growth raises the demand for reliable and effective tools for Computer Based Assessment.

Material published as part of this journal, either on-line or in print, is copyrighted by the publisher of the Journal of Informa-tion Technology Education. Permission to make digital or paper copy of part or all of these works for personal or classroom use is granted without fee provided that the copies are not made or dis-tributed for profit or commercial advantage AND that copies 1) bear this notice in full and 2) give the full citation on the first page. It is permissible to abstract these works so long as credit is given. To copy in all other cases or to republish or to post on a server or to redistribute to lists requires specific permission and payment of a fee. Contact Editor@JITE.org to request redistribu-tion permission.

This special series of articles is focused on dis-cussing current approaches on applying Informa-tion Technology to the Assessment Process of Student Learning. As stated in the call for papers, the main topics of interest were:

• Placement, Formative, Summative and Self-Assessment: Novel Tools and Ap-proaches

• Innovation in Assessment

Special Series Editor: Salvatore Valenti

Information Technology in the Assessment Process of Student Learning

• • • • • • • Assessment Issues in Virtual Learning Environments Diagnostic Testing

Pedagogical Issues in Question Design And Content Impact on Institutional Teaching and Learning Strategies Quality Assurance Issues

Evaluation of Assessment Engines Assessment & Standards

After the revision process, ten papers authored by researchers working at the Universidade de São Paulo, São Carlos, Brazil; University of Veszprém, Hungary; University of Athens, Greece; Helsinki Univer-sity of Technology, Finland; Northern Michigan University, USA; Concordia University, Canada; Shef-field Hallam University, UK; Zayed University, UAE; and Università Politecnica delle Marche, Italy were accepted.

The first paper, by Aluísio, de Aquino, Pizzirani, & de Oliveira Jr., presents a web-based, fully auto-mated system for English Proficiency tests (CAPTEAP) that checks the students’ ability in the use of English for academic purposes. CAPTEAP employs the resources of a case-based reasoning tool to as-sist non-English speakers in producing scientific papers. It comprises 4 modules through which the stu-dents are evaluated with regard to their ability to: 1) analyze the structure of a given section of a paper; 2) analyze relationships among clauses signaled by discourse markers; 3) recognize conventions and rhetorical functions in English from scientific texts; 4) identify writing strategies for each component of a section. CAPTEAP has been applied in the official proficiency test required from MSc. students at the Computer Science Department at University of São Paulo, Brazil.

Zayed University, based in the United Arab Emirates, has adopted a new educational concept in the re-gion, which is an Outcome-Based learning approach. This new Academic Program Model (APM) is de-signed to continuously improve the curriculum and provide students with the knowledge and skills to succeed in a rapidly changing world. The APM is driven by five critical components: the outcome based curriculum, the e-portfolios, the learning communities, the use of information technology, and the sup-port of the center for teaching and learning assessment. The paper by Bouslama, Lansari, Al-Rawi, & Abonamah introduces a new hybrid APM that is both outcome and grade point average (GPA) based. It shows how assessment is effectively used to make the learning outcomes component work. Also, it dis-cusses how technology can facilitate the learning and assessment processes and describes how learning outcomes are used in the development of an information systems curriculum.

In their paper, Gouli, Gogoulou, & Grigoriadou present a coherent and integrated framework for educa-tional assessment purposes, serving various assessment functions. The AssessToLearn framework uses concept maps as the main tool of the assessment toolbox. The framework provides a basis for the design of assessment activities, and it comprises a three-step process consisting of (1) Ascertaining the Stu-dents’ Prior Knowledge - Activating Knowledge, (2) Promoting Knowledge Construction & Identifying Conceptual Changes - Constructing & Enriching Knowledge, and (3) Assessing Knowledge Construc-tion - Refining Knowledge. The AssessToLearn framework constitutes the basis for the development of a web-based adaptive assessment environment, referred to as PASS-Personalized ASsessment System supporting Collaboration, Adaptation and Learning. PASS aims to support the assessment process in the context of the framework and provide adaptive capabilities as far as the adaptation of the assessment process and the guidance/feedback given to the students during the assessment process are concerned. According to Korhonen, Malmi, Nikander, & Tenhunen., feedback is an essential element of learning. In the paper they consider how exercises with automatic feedback should be designed to support a broader scope of learners. To achieve these results, the Felder-Silverman learning model is discussed. The model categorizes learners with four different axes: sensing vs. intuitive learners, visual vs. verbal learners, ac-tive vs. reflective learners, and sequential vs. global learners. The authors discuss how all dimensions of

182

Valenti

the model can be taken into account when designing assignments and automatic feedback. Two modern automatic assessment systems, PILOT and TRAKLA2 are adopted as example systems to demonstrate the ideas presented.

Active learning and cooperative learning are two alternatives to the traditional lecture/lab approach to teaching software development. Evidence supporting use of these learning strategies in Computer In-formation Systems began to emerge in the mid-1990s as faculty sought ways to improve student under-standing of programming concepts, reduce the level of frustration, and better prepare students for jobs based on teamwork. Poindexter, in “Assessing Active Strategies for Teaching Programming,” reports a study with active and peer learning strategies that was conducted at the Northern Michigan University involving software development. Using literature, empirical research, and focus group discussion re-sults, the paper assesses the value of active and peer learning environments in programming courses. In his paper Saade discusses a research work that investigated the applicability of a modified version of the Technology Assessment Model (TAM) in explaining students’ acceptance of web based technolo-gies for their courses. The target IT presented is a web based Educational Information System for En-hanced Learning (EISEL). The results of the modified version of TAM are discussed. The technology and user group are new to the IT acceptance and adoption research. The TAM constructs were opera-tionalized in the context of the EISEL. This research is a first attempt (using the first version of EISEL) to understand students’ beliefs and perceptions in relation to using a web based learning tool. The results presented suggest that TAM was able to provide a relatively reasonable view of students’ acceptance of EISEL.

Sik-Lányi and co-workers present two papers: “Multimedia Program for the Training of the Vision of Children with Visual Impairment and Amblyopia” and “Using Virtual Reality to Improve Space and Depth Perception”.

The first paper (Sik-Lányi & Lányi) summarizes questions related to partial sightedness, defines the terms of visual training and education of vision, and discusses how the faculty of vision can be devel-oped also in partially sighted persons. Special attention is given to amblyopia, or bluntness of vision. Then the advantages of multimedia software in training the vision of partially sighted children are dis-cussed. Three multimedia programs are described that can be used to train the vision of children with visual impairment. Experiments were conducted in special schools for partially sighted children. The evaluation of the preliminary experiments can be summarized as follows: using the computer tasks and games motivated the children.

In the second paper (Sik-Lányi, Lányi, & Tilinger) a virtual reality program with animations that was used by students to practice space perception tasks and to acquire better space perception is discussed. For testing the programs students were screened for their space perception, and then performed paper tests. One group was permitted to practice using the virtual reality programs; the other used only tradi-tional tutorial methods. A second paper test was then performed. The evaluation of the two tests showed that the possibility of using the virtual reality programs increased the space perception of the students. The paper by Slack, Beer, Armitt, & Green discusses an experiment in which students from four Euro-pean countries collaborated and communicated to carry out problem-based learning in occupational therapy by using an on-line learning environment. The effectiveness of this learning approach was evaluated by means of the final assessments and published learning outcomes. In particular, transcripts from peer-to-peer sessions of synchronous communication were analyzed, using the SOLO taxonomy, to establish the extent to which deep learning had taken place. Results showed differences between the four tutorial groups in their approach to the ‘process’ of doing the case study assignment and its ‘content’. Statements taken from the transcripts were categorized by the SOLO level displayed and it could be seen that the students were achieving the learning outcome as the course progressed. The conclusions of this analysis of learning outcomes and assessment indicate that synchronous communication and on-line

183

Information Technology in the Assessment Process of Student Learning

meetings between course participants can support deep learning, although this does not happen sponta-neously.

The last paper of this special issue by Valenti, Neri & Cucchiarelli presents an overview of current ap-proaches to the automated assessment of essays. Ten systems, currently available either as commercial systems or as the result of research in this field, are discussed: Project Essay Grade (PEG), Intelligent Essay Assessor (IEA), Educational Testing service I, Electronic Essay Rater (E-Rater), C-Rater, BETSY, Intelligent Essay Marking System, SEAR, Paperless School free text Marking Engine and Automark. For each system, the general structure and the performance claimed by the authors are de-scribed. Then, an attempt is made to compare the performances of the described systems. The most

common problems encountered in the research on automated essay grading is the absence both of a good standard to calibrate human marks and of a clear set of rules for selecting master texts. Both these prob-lems represent interesting issues for further research in this field.

References

Valenti, S., Cucchiarelli, A., & Panti, M. (2001). Assessing programming skills over the web. In A. Andronico, A.M. Fanelli,

G. Piscitelli, & T. Roselli (eds), Didamatica 2001 - Informatica per la didattica - Lavori Scientifici, (pp 380-387). Bari, Italy: G. Laterza. Valenti, S., Cucchiarelli, A., & Panti, M. (2002). Computer based assessment systems evaluation via the ISO9126 quality

model. Journal of Information Technology Education, Vol. 1, (3), 157-175. ISSN: 1539-3585. Valenti, S., Panti, M., & Leo, T. (2003). Quality assurance issues for a web-based degree in motor disability assessment . In

F. Albalooshi (Ed.), Virtual education: Cases in learning and teaching technologies, (pp 34-49). Hershey, PA: IRMA Press.

Acknowledgements

I take the occasion to thank the authors of the papers for their contribution. I am very grateful to Eli

Cohen for the support provided in the design and in the realization of this special issue, and for hosting it on this great journal.

Finally, I wish to thank all the following reviewers for their time and extremely valuable work: Alessan-dro Cucchiarelli, Anil Aggarwal, Joanna Bull, Grainne Conole, Federico Flueckiger, Kinshuk, Emanuela Moreale, Maurizio Panti, Jin Qun, Doc Rea, Ali Fawaz Shareef, Ramesh Sharma, Wolffried Stucky, Alexey Tretiakov

184