Assessing Proctors: solving problems in grading student performance in a peer support system

Rob Hayler

1. Introduction

Peer support systems and schemes are becoming increasingly attractive to university departments throughout the disciplines and throughout the country. They offer a very cost-effective way to implement various pedagological aims and to train students in useful transferable skills. So successful can these schemes be that the urge to somehow incorporate student participation in these schemes into the final degree classification of the student can become irresistable. At that point however, serious problems arise regarding the assessment and grading of performance in such an academically uncharacteristic project. The proctorial system run by the School of Philosophy at The University of Leeds is just such a scheme and faced just such a problem. In this article I intend to describe what it is we do here, and discuss our solution to the problem detailed above.

2. The Proctorial System

Here in the School of Philosophy at the University of Leeds our students are encouraged to do philosophy. We do not want to create students who just know about philosophy, we want to produce philosophers. Now, doing philosophy includes attending lectures, making notes and reading books but, as an undergraduate at least, it is largely a matter of talking, discussing and arguing with your peers. We find that most personal growth in philosophy comes as a result of such activity. This is exactly what we hope to encourage with the proctorial system .

A proctorial is an hour long meeting, taking place in the first half of the week, of the members of a first year tutor group, usually around a dozen students. The students are set a piece of reading and some related questions by the lecturer whose course is under discussion that week. The idea is that they get a chance to discuss these questions, the reading, and their views on this subject in an informal, non-threatening environment as a group without the presence of an authority figure such as their tutor. Amongst other benefits, this gives the group a chance to gel and individuals the chance to hone the philosophical skills needed for debate. The group presents the product of their deliberations to their tutor at the tutorial, which takes place in the second half of the week, and these results form the basis of initial tutorial discussion. The proctorial is supervised by a more experienced peer: a third year undergraduate who acts as a proctor.

3. The Role of the Proctor

The proctor has certain administrative duties, such as to take a register, but their main job is as a chairperson and a facilitator of discussion. They are not there to teach or answer questions, they are there to aid the learning of their proctorial group. They are also form part of the School's pastoral care network and students are encouraged to ask them general questions about student life.

The proctors, as a group, have to attend six training sessions at the beginning of semester one: two before they start the job and four after, then have another two refresher meetings at the beginning of semester two. During these sessions, attended by all proctors (roughly thirty students) and chaired by myself, proctors are given advice on how to put the specification of their role into practice in particular situations: how to handle unruly students, how to encourage a participative atmosphere and so on. As the proctors' meetings progress they should (and do) come to resemble a proctorial with me talking less and just guiding discussion as proctors gain confidence and answer each others' questions.

Whilst acting as proctors our third years garner and develop certain very useful transferable skills. These include people-management skills such as being able to guide a discussion in a non-intrusive way, attentive listening skills and diplomacy, and project-management skills such as successful time management and sub-division of tasks. They are also expected to take on and take seriously certain responsibilities such as monitoring attendance, regularly liasing with the tutor for their group and dedicating an hour of their time every teaching week for the whole year to the same proctorial group.

The combination of all these factors, that the proctors attend training meetings, that they dedicate themselves to an important task involving some responsibility and that they learn how to put into practise certain skills which are useful, not only in philosophy, but in life and work in general, led to us believing we would be justified in offering participation in this scheme as an assessed module which would count towards the final degree classification of the student. Previously we had relied exclusively on those volunteering. At this point the headache of how to assess a proctor's performance began to throb.

4. Proctoring as a module

Proctoring was made available to take as a 10 credit module for any third year student who had done or was doing 80 or more credits in philosophy in their upper years. To put this in context, all full-time arts students here at Leeds do 240 credits worth of courses, which are usually 10 or 20 credits each, over their upper two year and their degree classification is determined by the average of the marks they receive for all of them. Thus, the Proctoring module would make up 1/24 (one twenty-fourth) of the marks that determined degree classification. Logged as a 'regular' 10 credit module, the proctoring course became subject to certain regulations. The most important and problematic of which being that a 10 credit arts module needs an assessment of the form of two thousand words worth of writing or a two-hour exam. The problem of how to assess proctoring had arisen and needed to be solved.

5. Preliminary solutions

The first solution was not, in my opinion, a solution at all. It was suggested that the proctors just hand in something purely to satisfy requirements. This piece of work could be on the proctoring experience, the pedagogy of peer support and so on and, regardless of its quality, it would be awarded a mark equal to an average of all that student's other marks. The informing idea behind this suggestion was that the students had done enough to earn 10 credits just by participating and lending their time to the scheme. The students taking the module were, unsurprisingly, quite keen on this idea but those like myself and my then colleague Mark Funnell (now teaching in further education in Stevenage) who were actually running the system hands-on felt that this was deeply unsatisfactory for three reasons. Firstly we were not guaranteed any interesting feedback in the form of carefully reflexive written assessments from our proctors. Secondly we felt that the proctors were being effectively sold-out at the last minute by not having a chance to consolidate their experiences in a piece of written work at the end of the year, a positive and useful part of any other course. Thirdly, we felt that by awarding an average of all other grades we were potentially being unfair to those average students who had excelled at proctoring and biased towards the few brilliant students who treated proctoring as a bit of time off.

This third problem was partially addressed by the second proposed solution: that they would do a piece of work on any connected subject that they chose, as before, but this time if it was particularly good they would be awarded their average mark plus five percent and if it was particularly bad they would be awarded their average mark minus five percent. After an initial first glance, it became apparent that this was probably worse than the first solution. The first two concerns Mark and I expressed were still not addressed and the 'index-linking' of the score to the student's overall average meant that the third worry remained but now the system had been complicated. Mark and I believed that a workable and fair solution could be found so we worked and found it.

6. A Written Assessment for Proctors

The assessment that we decided upon takes the following form. The student is asked to call upon their training and experience in responding to a number of ficticious or generalised scenarios. These include such problems as 'the students arrive at their first proctorial unsure what a proctorial is, what do you tell them?', 'certain students never participate, what do you do?' and so on. They are given a choice of eight and asked to choose to write on between three and six. The word limit is approximately 1500 words and they are asked not to write more than 750 words on any one answer. See the appendix for a copy of the complete assessment as the students see it. The students were encouraged both in the meetings and in the rubric to be honest and draw upon their own experience. No marks are deducted from anyone who admits that they did something wrong or that a problem arose due to their own action as long as they show some reflexive understanding of what happened and learnt from the experience. In order to level the playing field , it is made clear to the students that they are being marked more on the understanding of their role than their ability to construct stylish prose. Mark and I felt that we had created a way of fairly judging the participation of a proctor in this system by means of a written assessment. We felt that the structure of the assessment, the choice available within the assessment and the encouragement given in training sessions and in the rubric made it so.

7. Successes and feedback loops

Much to our relief this assessment worked out even better than we planned. After some initial grumbling from the first year to be given the assessment they seemed quite enthusiastic about it. I received some informal feedback from proctors informing me that they had enjoyed the opportunity to do something a bit different and reflect on their experiences. In the first year the marks ranged from low II.ii scores to a high I. In the second year the students were given model answers to look at and discuss in order to help them understand what was expected of them. I am proud to say that, after monitoring by a senior member of staff, none of these proctors scored less than a II.i grade for this course.

This year, the third year that this form of assessment has been in place the feedback loop has been completed. The proctors are given the assessment at the very beginning of the first meeting. It is then used as the basis of for the first four meetings, each problem scenario used as a jumping off point for discussion. Thus the assessment itself now structures the training. At the start of the second semester a wider range of model answers is now available for student discussion. Early indications from feedback I have garnered from students and tutors is that this years' proctors are the best ever. I eagerly await the assessments they are due to hand in this May.

Appendix: The Assessment

PHIL3910 Proctoring Module Assessment

Course Tutors: Jim Parry, Robert Hayler

Rubric

The deadline is Friday 15th May 1998.

The word limit is 1500 words. Answer 3 to 6 questions, writing no more than 750 words on any individual question.

Whilst I wish you to show that you understand the nature of the proctor's role, I am also interested in your views on proctorials and proctoring.

Questions

  1. The students arrive at their first proctorial unclear as to what a proctorial is and unsure of the role of the proctor. What do you tell them?
  2. A mature student has trouble accepting your authority and is disruptive. What do you do?
  3. Your proctees insist on directing logic questions at you. What is the procedure?
  4. The group are not considering their answers carefully. Consequently, the answers they give are insubstantial and they are finishing the set material too quickly. Explain how you rectify the situation with reference to at least one question from the proctorial handouts.
  5. What do you do if the group has made a fair attempt at answering all the questions set and there is still about 20 minutes left of the proctorial?
  6. Describe any measures you could take to deal with either
    1. one student dominating proceedings.
    2. two students dominating proceedings.
  7. How would you attempt to deal with students who do not participate in the discussion?
  8. Describe a problem that you encountered whilst proctoring and its consequences. How did you deal with it? What did you feel that you learnt from this?


Created on: April 17th 2007

Updated on: February 25th 2009