Computer-Aided Assessment in Statistics - the CAMPUS project

by Neville Hunt, Statistics Group, Coventry University

Introduction

The relentless drive for "efficiency" in higher education and the consequent increase in workloads has given university teachers a compelling incentive to investigate alternative forms of assessment. Some forms of assessment with a clear educational value can no longer be entertained because of the burden placed on the teacher. An added concern is plagiarism (i.e. cheating), which anecdotal evidence would suggest is on the increase, yet is difficult to detect in large modules with more than one assessor. Whilst computer-aided assessment (CAA) has an enthusiastic following, it is not clear to many teachers that it either reduces workloads or reduces the risk of cheating. In an ideal world most teachers would prefer to give individual attention and personal feedback to each student when marking their work. In this sense CAA must be seen as second best and will therefore only be used if it is seen to offer significant benefits in terms of reduced workloads or increased validity.

This paper describes the development of a spreadsheet-based CAA package for use in the teaching and learning of Statistics and other quantitative disciplines. The CAMPUS program (Computer Aided Marking Program Using Spreadsheets) seeks to address the concerns of teachers regarding workload, validity and feedback when assessing students.

Requirements of a CAA package

Fundamentally a CAA package must be easy to use, both by the teacher when setting the assessment and by the student sitting it. If the CAA package requires a large investment of time in setting up the test, it will only be used by the enthusiast. Equally, the process of sitting the test must be straightforward, otherwise students will be quick to seize upon software difficulties as the reason for their under-performance. The package should be able to accommodate a variety of question types, including multiple-choice, text response and numerical answer. There should be a mechanism for giving informative feedback to students, not simply the number of "right" and "wrong" answers. An efficient and secure system for recording marks is required. Security devices are essential to guarantee the integrity of the test, ensuring that students can neither access the test before it is sat nor alter their answers or marks after submitting the test for assessment.

The nature of Statistics as a discipline imposes additional requirements on a CAA package. Many areas of Statistics involve a graphical approach, so the ability to incorporate graphs and charts into questions is essential. Statistical theory is quite mathematical, so there is a need to be able to handle symbolic notation. Most statistical problems involve a mixture of calculation and interpretation. Whilst the calculation element lends itself to questions with a fixed numerical answer, assessing interpretation is more difficult. Using text response is almost impossible in most cases since there are many different forms of words that have the same essential meaning. Multiple-choice items are not an entirely satisfactory solution since they may give the student ideas that they may not otherwise have had.

Most teachers will already possess a battery of assessment items that they have accumulated during their career. Some of these may be paper-based, but many will be stored in a variety of electronic formats. For example, the author has maintained a bank of some 400 tutorial questions stored in text format on a Unix computer. It is clearly desirable that these accumulated resources should be easily transferable into any CAA package that the teacher chooses to use.

Random questions

Software already exists which offers most if not all of the desirable features outlined above. For example Question Mark Designer (ref???) is in widespread use within UK higher education institutions. What motivated the CAMPUS project was the desire to able to set "random" questions. Software such as QMD offers a facility to select a certain number of questions at random from a large library, but it does not allow individual questions to contain random parameters. To clarify the distinction, consider a trivial item designed to test students' ability to carry out simple addition. A large library of questions could be prepared as in Table 1.

Table 1 Library of questions

The alternative is to prepare a single question, as in Table 2, where x and y are numbers that are randomly generated each time the question is loaded. This has several advantages:

Table 2 Random question

1. All students tackle different questions, eliminating any possibility of copying answers.
2. All students tackle questions of equal difficulty, which fairness demands.
3. There is infinite opportunity for drill and practise, which many students need.
4. Statistics is an applied subject and the vast majority of Statistics teachers would endeavour to set assignments and problems in a real-world context. Statistics is taught to students in many disciplines - biology, geography, economics, psychology, to name but a few. Finding different relevant contexts for each of these diverse fields of study is extremely time-consuming. The task of establishing a large library of questions of similar difficulty for each application area would be enormous.
5. With such a trivial example, there is little difference in the effort required by the lecturer in setting up the test. However, in more complex problems even the task of computing the answers would be very tedious, let alone the risk of occasionally getting an answer wrong!

Microsoft Excel

The desire to incorporate random elements, including random graphs, leads naturally to a spreadsheet-based system. Microsoft Excel is the spreadsheet package supported by Coventry University and is widely used elsewhere. An increasing number of students have home-computers on which Excel is installed. As a basis for a CAA package Excel has many advantages:

• it includes a large range of statistical functions;
• it incorporates a random number facility;
• the Chart Wizard offers a wide range of statistical charts and graphs;
• charts, graphs and calculated statistics can be linked dynamically to the source data;
• a limited use of symbolic notation and an ability to import Microsoft Equation Editor objects;
• the Import Wizard allows the lecturer to import questions stored in a variety of electronic formats;
• the software environment is familiar to the students and will not therefore distract from the test;
• the software is familiar to staff who will therefore require little training to use the CAA package;
• the user-interface can be controlled (and secured) using Visual Basic macros.

Simonite et al (1998) have described an Excel-based system for setting personalised coursework assignments for students using a randomisation method. However, the marking of the assignments is still carried out by hand. The CAMPUS program seeks to automate the whole process of setting and marking such assignments.

The CAMPUS program

CAMPUS consists of two Microsoft Excel 5 workbooks, QUEST and TEST. The teacher uses QUEST to construct a test paper by either selecting questions from the existing database, or by entering additional questions that they have devised. Questions in the database are classified by topic (e.g. NORMAL) and application area (e.g. BUSINESS). TEST is a skeleton (or template) test sheet which has already been formatted and equipped with various security features (described later). Once the teacher has completed the selection of questions, upon the click of a button both questions and answers are automatically copied from QUEST and pasted into the TEST template. The answers are immediately hidden from view and the TEST file is then saved under a different filename specified by the teacher.

A student sitting the test simply loads this file into Excel and follows the onscreen instructions (see Figure 1). Visual Basic macros within the test file ensure that any random elements within the test - in both questions and answers - are recalculated on loading, so that each student’s test is different. Having completed the test the student clicks a button to "Submit for Marking". A macro then checks the student’s responses against the (hidden) given answers and compiles a detailed feedback sheet for the student (see Figure 2). Two copies of the student’s responses and feedback sheet are automatically sent to the local printer. The teacher may also have opted for them to be saved electronically to a secret network directory.

Question Construction

There are essentially five different types of question supported by CAMPUS.

Multiple-choice

These are the simplest to construct. Figure 3 shows the layout of a typical question.

Figure 3 Layout of typical multiple-choice question

The question is constructed on the first 17 columns of a standard spreadsheet. On the first row of the question, Columns 1-4 are for reference information. Each of the four lines in the body of the question is actually entered into a single cell in Column 5 - in Excel, cells automatically expand to accommodate up to 256 characters. The question text could be copied in from a word-processed file provided that a carriage return is inserted at the end of each line - if not, the whole question will be pasted into a single cell.

The lines immediately following the question contain the possible answers - a maximum of 9 choices can be given. In Column 5 are the labels (A, B, C, etc.) while Column 6 contains the answers that students are expected to give - both correct and incorrect. In Column 12 a feedback narrative is entered corresponding to each answer. This narrative appears on the student feedback and marks sheet upon completion of the test. If the student’s response is incorrect but not anticipated in the list of wrong answers, a default feedback narrative is invoked, namely "You made an unforeseen error!".

Numerical answer

The structure of numerical answer questions is almost identical to that of multiple-choice questions. The only modification to the layout in Figure 3 is that the letters A, B, C and D, labelling the four choices, are replaced by numbers indicating the maximum acceptable arithmetic error (either way) in the student's answer. For probabilities this might be 0.0005. If omitted these "tolerances" are assumed to be zero. Obviously the four answers are not now displayed to students sitting the test! However, if a student enters one of the foreseen wrong answers they still get the corresponding feedback narrative. Once again, up to 9 possible answers can be entered. One minor restriction is that the "Correct" answer must be listed first.

Multi-part

It is quite common for Statistics questions to comprise several parts, all based on the same real (or at least realistic) context. Suppose that our example question has an additional part "What is the probability of at least one of its ships running aground this year?". The revised layout is shown in Figure 4.

Figure 4 Layout of a multi-part question

Each part is effectively a separate question. The only link is via the reference numbers 5.1, 5.2, and so on. Part (a) must be incorporated into the body of the introduction. When constructing a test, the setter may omit some of the parts, provided that the labelling is edited accordingly.

Random element

The ability to set random questions is probably the key feature of CAMPUS. These are the most difficult to set and require a basic grasp of Excel functions and formula construction. Figure 5 shows a modification of our initial example to incorporate a random element.

Figure 5 Random element in question

In the original question students had to calculate the probability of 2 ships running aground. Now each student calculates the probability of x ships running aground, where x is randomly chosen. The critical line is the fourth line of the question. In Column 5 the setter starts typing the question as normal. At the point where the setter would have typed "2 ships …" they press Return (or Enter). In the next cell, Column 10 here, an Excel formula is entered. It seems reasonable for x to be 0, 1, 2, 3 or 4, so the appropriate formula is =INT(5*RAND( )) . Pressing the F9 (recalculate) key a few times acts as a useful check that the formula is operating correctly.

It remains to sort out the answers. The first answer should be Pr(x) using mean 1, the function for which is POISSON(x,1,FALSE). The location of x must be specified relatively using RC (Row and Column) notation. In the spreadsheet shown this would be R[-1]C[4], meaning back 1 row and forward 4 columns. So the final formula would be =POISSON(R[-1]C[4],1,FALSE) . Similarly, the second answer would be =POISSON(R[-2]C[4],0.5.FALSE), the third
=POISSON(R[-3]C[4],2,FALSE) and the fourth =1-POISSON(R[-4]C[4]-1,0.5,TRUE). Currently CAMPUS does not allow random feedback narratives, so the fourth feedback narrative would have to say, albeit rather vaguely, "Wrong: you calculated at least not exactly".

Graphic element

Provided any graph, chart or picture is embedded into the cells containing the body of a question, it will be transferred across to the test paper in the normal way. If the chart contains random elements then the source data must also be stored in the body of the question. Since embedded charts actually sit on top of the cells, the source data can be hidden in the cells behind the chart if necessary.

Uses of CAMPUS

At Coventry University the CAMPUS program has been used to conduct three different kinds of assessment.

Supervised summative assessments

Here students all sit the test at the same time in a networked computer laboratory under the supervision of a member of the teaching staff. The teacher responsible for the assessment will have previously installed the test file onto the network, which students now load into Excel. On completion of the test the student feedback sheets are retrieved immediately from the local printer, one copy being retained by the teacher and the other presented to the student. Where there is any possibility of fraud, perhaps because the local printer is located in another room or some other insecure location, the teacher may crosscheck the printouts against the saved electronic copy of each student’s work.

Unsupervised summative assessments

The system has been used for marking unsupervised homework exercises, despite the obvious concerns regarding cheating. A student may have attempted the test several times, thereby benefiting from the feedback, or sought assistance from others who have already obtained the correct answers. Provided the homework does not carry any great weight in the overall assessment, the fact that this student has gained an inflated mark of 90% is considered a small price to pay for the reduction in the marking burden. Colleagues who find this unsatisfactory simply ask for the student’s hand-written solutions to be submitted with the CAMPUS printout, as an additional check that the work has been done. This also enables the teacher to identify the cause of any "unforeseen error" feedback given by CAMPUS. Even so, the time taken to check homework in this way is a small fraction of the time taken previously to mark it and give relevant feedback by hand.

Formative self-assessments

The teacher can use CAMPUS to provide students with random revision tests that can be sat time and time again, until the student achieves an adequate level of proficiency. Since every time the test is loaded it contains different values for random elements, in this mode of operation students are allowed to save their test with its current values so that they can resit exactly the same test. Also in this mode of test, the electronic storage of a student’s responses is disabled, to avoid unnecessary file storage - the student simply retrieves the feedback sheet from the local printer.

Security

It is the responsibility of the setter to ensure that the location of the test file is not made public prior to the appointed time for the test to be sat. When a student loads the test several security devices come into operation:

1. All Excel menus and toolbars are disabled and removed from view.
2. The answers and feedback narratives are hidden from view.
3. The test worksheet is locked, protected by a password supplied by the setter, so that only those cells reserved for the student's responses are accessible.
4. Certain keys are disabled, notably the "=" key, which might otherwise allow students to use Excel as a (very powerful) calculator.
5. In a Windows 3.1 environment, the Excel window is frozen so that students cannot switch to other software applications (e.g. electronic mail); unfortunately it appears to be impossible to achieve this in a Windows 95 environment.
6. The cells for a student's responses are formatted with a black font on a red background, which is remarkably difficult to read - even from an adjacent desk!
7. Once a student submits the test for marking, their mark/feedback sheet is sent to the local printer and stored in a secure directory on the network. The permissions are set on this directory to allow students to create new files but not to read, write, view or erase them. As an additional level of security, an Excel facility is used which only allows the file to be opened using the setter's password.

One weakness of CAMPUS is that although each student is required to enter a name before the test is marked, no steps are taken to confirm their identity. In a supervised test it is left to the invigilator to check the identities of students sitting the test.

Conclusion

As more experience of running CAMPUS is gained, additional minor modifications are being made. However, the underlying simplicity of the package is its principal attraction and no further major developments are planned. However, work is ongoing to extend the existing database of questions.

Acknowledgement

CAMPUS was produced by the author under secondment to the Coventry University Teaching, Learning and Assessment Task Force.

References

Simonite, V., Ells, P. and Turner, W. (1998). Using IT to generate individualised coursework questions and solutions for an introductory course in statistics and probability. CTI Maths&Stats, February 1998.