How difficult are exams? A framework for assessing the complexity of introductory programming exams

Simple item page
aut.conference.typePaper Published in Proceedings
aut.publication.placeSydney
aut.relation.endpage154
aut.relation.startpage145
aut.relation.volume136
aut.researcherClear, Anthony George
dc.contributor.authorSheard, J
dc.contributor.authorSimon
dc.contributor.authorCarbone, A
dc.contributor.authorChinn, D
dc.contributor.authorClear, Tony
dc.contributor.authorCorney, M
dc.contributor.authorD'Souza, D
dc.contributor.authorFenwick, J
dc.contributor.authorHarland, J
dc.contributor.authorLaakso, M-J
dc.contributor.authorTeague, D
dc.contributor.editorCarbone, A
dc.contributor.editorWhalley, J
dc.date.accessioned2013-02-11T10:31:27Z
dc.date.available2013-02-11T10:31:27Z
dc.date.copyright2013-01
dc.date.issued2013-01
dc.description.abstractStudent performance on examinations is influenced by the level of difficulty of the questions. It seems reasonable to propose therefore that assessment of the difficulty of exam questions could be used to gauge the level of skills and knowledge expected at the end of a course. This paper reports the results of a study investigating the difficulty of exam questions using a subjective assessment of difficulty and a purpose-built exam question complexity classification scheme. The scheme, devised for exams in introductory programming courses, assesses the complexity of each question using six measures: external domain references, explicitness, linguistic complexity, conceptual complexity, length of code involved in the question and/or answer, and intellectual complexity (Bloom level). We apply the scheme to 20 introductory programming exam papers from five countries, and find substantial variation across the exams for all measures. Most exams include a mix of questions of low, medium, and high difficulty, although seven of the 20 have no questions of high difficulty. All of the complexity measures correlate with assessment of difficulty, indicating that the difficulty of an exam question relates to each of these more specific measures. We discuss the implications of these findings for the development of measures to assess learning standards in programming courses.
dc.identifier.citationAustralasian Computing Education Research Conference (ACE 2013) held at UniSA, Adelaide, Australia, 2013-01-29 to 2013-02-01, published in: Proceedings of the Fifteenth Australasian Computing Education Research Conference (ACE 2013), vol.136, pp.145 - 154
dc.identifier.isbn978-1-921770-21-0
dc.identifier.urihttps://hdl.handle.net/10292/5151
dc.publisherAustralian Computer Society (ACS)
dc.relation.urihttp://crpit.com/PublishedPapers.html
dc.rightsCopyright © 2013, Australian Computer Society, Inc. This paper appeared at the 15th Australasian Computing Education Conference (ACE 2013), Adelaide, South Australia, JanuaryFebruary 2013. Conferences in Research and Practice in Information Technology (CRPIT), Vol. 136. A. Carbone and J. Whalley, Eds. Reproduction for academic, not-for-profit purposes permitted provided this text is included.
dc.rights.accessrightsOpenAccess
dc.subjectStandards
dc.subjectQuality
dc.subjectExamination papers
dc.subjectCS1
dc.subjectIntroductory programming
dc.subjectAssessment
dc.subjectQuestion complexity
dc.subjectQuestion difficulty
dc.titleHow difficult are exams? A framework for assessing the complexity of introductory programming exams
dc.typeConference Contribution
pubs.elements-id135293
pubs.organisational-data/AUT
pubs.organisational-data/AUT/Design & Creative Technologies
pubs.organisational-data/AUT/Design & Creative Technologies/School of Computing & Mathematical Science
pubs.organisational-data/AUT/PBRF Researchers
pubs.organisational-data/AUT/PBRF Researchers/Design & Creative Technologies PBRF Researchers
pubs.organisational-data/AUT/PBRF Researchers/Design & Creative Technologies PBRF Researchers/DCT C & M Management & Allied
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Pages 145ff from Vol136.pdf
Size:
644.42 KB
Format:
Adobe Portable Document Format
Description:
Download
License bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
licence.htm
Size:
29.98 KB
Format:
Unknown data format
Description:
Download
Collections
School of Engineering, Computer and Mathematical Sciences - Te Kura Mātai Pūhanga, Rorohiko, Pāngarau