Our two final year undergraduate project students are making great progress in categorising samples of the PeerWise questions created by students in our first year Physics class.

They've developed and refined a range of information to be captured from each question authored by a student as follows:

*Clarity of Questions: -*

0 – Unclear
(including spelling & grammar that make question unclear)

1 – Clear

*Feasible Distractors*:

0 – None

1 – At least
2 but not all

2 – All
Feasible

*Explanation:*

0 – Missing

1 – Inadequate
or wrong

2 – Minimal/unclear

3 –
Good/Detailed

4 – Excellent
(Describes physics thoroughly, remarks on plausibility of answer, use of
appropriate diagrams, perhaps explains why you would have obtained distractors)

*Quality of Author Comments:*

0 – None

1 – Irrelevant

2 – Relevant

*Correct:*

0 – Obviously
Incorrect

1 – Most
Likely Correct

*Recognised as Incorrect:*

0 – N/A or
not recognised

1 –
Recognised as incorrect by students (or disagreement with author)

*Diagram: *

0 – None

1 –
Contextual picture but not relevant

2 – Relevant
diagram or picture

*Plagiarism: *

0 -
Potentially Plagiarised

1 – Not
obviously plagiarised

*Context of Question*:

0 – None
(formulas, recalling info)

1 –
Irrelevant or extraneous context (entertaining, imaginary)

2 – Physics
(frictionless, idealized situation)

3 – Relevant
real world context (applicable to daily situations cars on racetracks)

*Revised
Taxonomy: *

1 – Remember,
Recognise or Recall OR just plugging in numbers

2 –
Understand, Interpret or predict (No calculation needed, understanding 3^{rd}
law for example)

3 – Apply,
Implement or Calculate (1 step calculation)

4 – Analyse,
differentiate or organise (multi-step calculation, higher analysis)

5 – Evaluate,
Asses or Rank ( Evaluating various options and assessing their validity)

6 – Create,
Combine or Produce (Aske

d to combine various areas of
physics, need to get a structure right to solve whole problem)

This last category maps the question onto levels in the cognitive domain of Bloom's (revised) taxonomy. After doing some tests to establish an acceptable level of inter-rater reliability, we've let the two students loose on their own sets of questions.

They're in progress, but early indications are that in contrast to a recently published study (A participatory learning approach to biochemistry using student authored and evaluated multiple-choice questions, Denny and Bottomley DOI: 10.1002/bmb.20526) we're seeing that only relatively few questions inhabit the lower reaches of this scale, with most in category 3 and 4 and non-negligible numbers in the highest categories. I'll post more results when we have them.

Here's a nice example of a question classified at level 5 in the taxonomy.