This short paper discusses how Familiarisation procedures as described by the Manual could work for a large scale assessment body such as Cambridge English.
The Manual perceives the Familiarisation procedure as an ‘indispensable starting point’ before a linking exercise to the CEFR can be carried out effectively. It goes on to say that ‘an account of activities taken and results obtained is an essential component of the validation report’. It advocates that ‘participants in the linking process [should] have detailed knowledge of the CEFR’ (2003:1).
There appears to be two underlying assumptions in the above statement: (a) linking examinations to the CEFR is a one-off, time-bound process; (b) participants involved in the linking process are a limited number of people with specific skills perhaps not general within the wider organization.
Let’s consider the first assumption in light of the Cambridge English test development and review model. The model has an ongoing iterative cycle from perceived need to test design to trialling to administration to post exam review (for full discussion of the model see the Cambridge English test development cycle in Weir and Milanovic 2003). The cycle allows for changes in learning, pedagogy and assessment trends as well in the targeted candidature to be incorporated into an examination. Using the Manual terminology, we can relate the test design phase to Specifications, the trialling phase to Standardisation (training and standard setting), the administration and post exam review to Empirical validation. Therefore, by default the linking process is perceived as an on-going activity rather than as a single activity at a given time.
Several issues are highlighted by the second assumption: who are these participants, how many, how familiar are they with the CEF on a familiarity-unfamiliarity continuum, should we take their assessment experience into account or should we start with an assumed zero baseline of knowledge? The picture gets complicated when the participants involved in ensuring that an examination is related to the CEF span across the organization and beyond. Let us look at some figures before we discuss how familiarisation is dealt with. Within Cambridge English, there are 33 subject officers, seven exam managers, eight research & validation officers, and four research & validation managers who work with a network of stakeholders on test specification, item writing, test construction, performance scale construction and on the application and use of performance testing scales. The network includes 30 chairs and 115 item writers, 15000 oral examiners, 700 team leaders, 65 senior/regional team leaders, and 500 writing examiners who do not necessarily reside in the UK and whose roles and responsibilities are at times interchangeable, e.g., a chair of one exam could be an item writer for another (figures are true at Dec 2007). Experienced members of this large community certainly have a close understanding of Cambridge English exam levels. For example, item writers providing materials for a particular examination are familiar with how to interpret the level, e.g., in terms of text difficulty, linguistic features, genre choice etc. Item writer guidelines and examination handbooks provide detailed information on text selection and item writing at a certain level. Through their work with a certain proficiency level, many will also be familiar with the ALTE Can Dos (which Cambridge English, as an ALTE member, helped to develop and validate) and, of course, have come into contact with the CEFR.
The questions we must answer, then, concern the purposes that familiarisation with the CEFR should serve, and the approaches that we should take.
Current processes and documentation specify and operate exam constructs and levels very effectively. They already make use of many things now incorporated into the CEFR but familiar for many years – the Waystage and Threshold specifications, for example. Further explicit reference to the CEFR is being introduced into processes over time where this serves to complement or clarify. Modification of a standard is one possible purpose. The CEFR may also impact at other points in the test cycle, without explicit reference in operational documentation. For example, current studies of the constructs of Reading, Listening, Writing and Speaking, in which a socio-cognitive model is used to match test tasks to CEFR level descriptors, may motivate changes to tasks as exams are revised or updated.
For Cambridge English, Familiarisation with the CEFR should be seen as a part of consolidating and building on existing knowledge. At the same time, the prominence of the CEFR raises the need for a more general awareness-raising, of particular importance for staff just entering the organisation.
Using activities and materials described in Chapter 3 of the Manual, a set of familiarisation procedures are made available to internal staff and to our network and are being implemented where and when applicable as follows:
- as part of a one-day workshop with pre and post activities
- as part of commissioning letters to stakeholders involved for example in performance testing
- as part of the induction and training programme for new staff, e.g. induction worksheet
- as part of a longer training course - a total of 18 contact hours (planned activity)
- as a seminar with pre- and post-activities, for example during annual meetings with Chairs (planned activity)
- as part of Cambridge English seminar series (planned activity)
These activities will be preceded by a short exercise to ascertain level of familiarity with the CEFR.