• If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • You already know Dokkio is an AI-powered assistant to organize & manage your digital files & messages. Very soon, Dokkio will support Outlook as well as One Drive. Check it out today!


Session 4B Notes

Page history last edited by Roger Hadgraft 14 years, 8 months ago

Notes by Nicola Sochacka [n.sochacka@uq.edu.au]. (Thanks Nicola). Please add comments and queries in the Comment box. If you don't see the Comment box below, you will need to apply for write access.


Paper 1: A Knowledge Framework for Analysis of Engineering Mechanics Exams

·         Concerned with consistently high failure results in certain engineering courses, e.g. statics and dynamics

·         Investigated actual responses to exam questions in a number of institutions

·         Difficulty in achieving consistency across institutions which prompted need for underlying theory à Modified Romiszowski Mechanics Framework

·         Heavy bias towards ‘procedural’ knowledge, not other three areas, e.g. conceptual knowledge

·         Also depends on how question is asked, e.g. 1st law of thermodynamics asked in a ‘factual’ manner, students not required to explain

Question: What are the strengths and weaknesses of using exams to evaluate the effectiveness of teaching?


Were teachers asked what they were trying to test? Which concepts? No, not in this study. Perceived level of difficult was asked, but not direct relationships between question and theory.

Paper 2: Use of Item Response Theory to Facilitate Concept Inventory Development

·         Using IRT to evaluate altered questions, 2 parameter model (alpha and beta)

o   Discrimination

o   Item difficulty

·         Four areas are evaluated: graphical interpretation, inferential…+ 2 others

·         Downfalls: concept inventories needs large sample sizes, questions are ‘right or wrong’

Question: How can we use IRT to understand misconceptions better? For example by clearly tying misconceptions to these types of questions.


Feedback to teachers is given as topic within subcategory, not as relationships between subcategories.

Paper 3: Student awareness of conceptual variations in a key nanoscience concept: Conceptual change in an engineering course

·         How undergraduates understand key concepts

·         What are key concepts if students are to understand a particular subject area? E.g. in nanoscience, size and scale and surface area to volume ratio (see paper below), four part process:

a.      Identifying constructs

b.      Unpacking constructs

c.       What indicates that students actually understand these constructs?

d.      Developing pedagogical interventions to effectively assess concepts.

·         Study used phenomenology/ variation theory with aim to

a.      Maximise variability in student understandings to capture hierarchy of conceptions/ understandings, from least to most sophisticated understanding, therefore understandings are internally related

b.      Aim to locate/ distinguish students on continuum of understanding

Question: Is this a meaningful way of going about this subject area?

Paper 4: Unpacking Student Conceptions of Surface Area to Volume Ratio in the Nanoscience Context: An Empirical Application of the Construct-Centered Design Framework

·         Constructed a typology of student conceptions containing three understandings relating to surface area to volume in the nanoscience context

·         Surface area to volume and size and scale, designed 3 interventions to address variations in understandings

·         Results suggested that interventions were successful, also able to identify which particular aspect needed further attention

Question: How likely is this methodology to be accepted by instructors? i.e. To gauge what their students are understanding?


Interventions? Students working on transcripts to identify key concepts + another intervention (?).

Process used to identify key concepts? Referenced in paper (Shaun Stevens: the big ideas of nanoscience), workshops with people from industry and education to identify key concepts, done for K-12 and for undergraduate education.

General discussion

·         Different stages of development? IRT vs. different understandings of key concepts. IRT might be a shortcut to identifying conceptual misunderstandings and then designing interventions

·         CI hub (a library of conceptual inventories) will be put on-line by Purdue University in the near future

·         Some authors resist to have their instruments put up on line due to misuse, accessibility, confidentiality

·         Relationship between instruments is a fundamental question

·         Do different approaches to assessment between US and Australia drive research areas of interest? E.g. multi-choice vs. procedural

·         Is it convenient that conceptions in phenomenological studies fall into a hierarchy? Suggests that there is no such thing as an utter misconception à hierarchy does not exclude “wrong” answers, but rather measures depth of understanding

·         Students have difficulties in using concepts appropriately which are presented in a fragmented way by teachers, (who presumably have a holistic understanding of the interactions between the contexts) à e.g. “in the nano world”, this typical misconception might be identified by IRT (links between papers)

·         Discussion of the use of metaphors to explain concepts which are beyond our lived experience, problem lies when metaphor overshadows message/ key concept

Closing remarks/ questions

Discussion of what we are aiming to achieve in undergraduate education: transfer of knowledge or development of conceptual understanding?

Does our background as engineers (researchers with an engineering background) lead to a bias towards studies which focus on assessments and concept inventories (i.e. tangible and measurable tools/studies, papers 1 & 2)?

Such studies seem to suggest that more research is needed which focuses on in-class learning theories to assist student development of conceptual understandings (papers 3 & 4).

There may be promising collaborative opportunities between both areas of research.


Comments (0)

You don't have permission to comment on this page.