Love in a cold climate: Assessment of Digital Artefacts in an Irish University

Last week a group of us set out to write a piece  on assessment and evaluation of digital material in response to a call for contributions to the Journal of Digital Humanities. We began by writing fast stream of consciousness pieces which we later merged and edited into a single piece – that process is described in another post. This is the initial piece of fast writing which I did for our writing project, of which the final version appears online here

My engagement with assessing digital artefacts in academe is driven by the problem of grading the digital work being produced in our Phd and MA programmes in Digital Arts and Humanities

When the DAH Phd consortium developed as a funding proposal for a structured Phd, there was passing note that the outputs could include digital “somethings” but no detailed discussion on what those might be, what discount might be applied to the traditional  80,000 word thesis for the digital artefacts, and what criteria might be applied. Realistically, since the consortium involved 4 different universities and a collection of disciplines, it was probably just as well that we did not seek to engage with this in too much detail, or we would never have made the funding deadline.  Now, with our own 1 year MA course in UCC, we need to address this issue in a very immediate way – our students are currently working on digital products which we need to guide, and that guidance needs to be shaped by an clear awareness of what our expectations are, and how we will assess their digital work.

I actually have no real problem with this, but I know it is an insuperable conceptual issue for people outside our course team. Trying to talk about grading with colleagues, many of whom have never explicitly thought about grading because they rely on conventional essays or dissertations, and ‘everyone’ knows what is good and bad in those, is difficult. Colleagues who have never explicitly considered grade descriptors or grading rubrics cannot begin to conceive of how one might grade an as yet undefined assessed digital object.

Disciplines not only have signature pedagogies, they also have signature assessments, and the skill of grading those is handed down from generation to generation as an artisan craft, which is understood across the community of the discipline so external examiners have no problem validating the marks assigned in their discipline. I have no doubt but that grading is part art, and never wholly science, but in an interdisciplinary field like DH, where we must assess new types of student work, some frameworks are necessary.

In fact, in the dear old National University of Ireland, we have clear and well established guidelines in the NUI grade descriptors.  (insert ref, attach appendix)  The descriptors clearly lay out, in a general, non-discipline specific way, the sort of “evidence of a mind at work” (Joe Lee) we should expect at various grade brackets. They do not explicitly map to Blooms’ Taxonomy, so beloved of behaviourists as the gospel behind Learning Outcomes, but they do come fairly close.  So a Fail in the NUI descriptors manifests “a display of some knowledge of the material relevant to the question posed, but with very serious omissions/errors and/or major inaccuracies included in answer” which relates to the lowest level of Blooms’ Taxonomy. At the other end of the scale, a top first class answer includes “an outstanding ability to organise, analyse and express ideas and arguments in an original, sophisticated and discriminating manner” and while we must accept that words like “sophisticated” are subject to debate, this seems to me to clearly tick the boxes of the higher levels of Blooms’ Taxonomy.

 

Alongside these two “foundational” documents, and easily aligned to them, are any number of rubrics for grading different types of work. The NUI descriptors require not real modification for application to essays and other traditional work. Drawn from that traditional rubrics for “regular” student essays, people have produced countless rubrics for assessing blog posts and posting in discussion forums – these are, after all, written work and different from traditional academic writing mostly in extent, and sometimes in the formality of tone or voice. Equally, it isn’t actually an enormous stretch to apply the NUI descriptors to developing a rubric for grading twitter; and there are rubrics for those as well.

When we move into less conventional forms that matters become more complicated. How do you assess a database, a critical edition, a performance, a piece of multimedia or an ‘app’.  The optimal manner to build a relational database is, at one level, defined by a set of normalisation rules which are pretty clear and preclude, it seems, developing an argument, but at another level the choice of data to capture, the choice of datatypes, indices and relations are all driven by the questions which are informed the particular inquiry being pursued. It just so happens that the vast majority of databases in the real world are designed to track transactions and bill people for things, so they all look very similar. This is particularly true where they need to conform to accounting or legal standards imposed at national or international level. And yet it is still possible for different enterprises in the same field to design databases to perform the same tasks which can differ markedly from one another. It seems to me that databases designed by different students for different inquiries can and should differ, and while at the lowest level of Blooms’ Taxonomy, you would expect, for example that they would all comply with the SQL standard and be reduced to 3rd Normal Form, at the higher level you would expect to see individual choices about analysis and synthesis of the material to suit to particular analytic questions being asked of the original raw, pre-digital material.

I recently heard a passing comment on a film – the Phoenix Tapes – that “if it were a proper academic essay, it would of course have footnotes and so on” (Rod, but let’s not name him!) as if it was not a proper academic essay. (http://www.sixpackfilm.com/en/catalogue/show/1641 ) Now, to my mind, the Phoenix tapes are a very fine collection of 6 essays on Hitchcock – they chose themes, extract examples, arrange those examples in a structure which, to my eyes, includes a clear progression from introduction, through development to show the often horrific end result of the obsessions, highlighting along the way the manner in which Hitchcock visually expresses these themes. Another more professional description of them is “ The Phoenix Tapes show re-edited excerpts from 40 films by Alfred Hitchcock. The six chapters focus on a personal selection of various leitmotifx in Hitchcock’s work. The consequence of this is not only to highlight HItchcock’s obsessions with certain types of repetitive movements and highly loaded visual signifiers, but to suggest that these actions are part of a universal language of gesture that encompasses both cinematic and everyday modes of communication.Â’ (John Tozer, Camera Austria) (http://www.sixpackfilm.com/en/catalogue/show/1641) If we skate over the detail that the original footage was not digital, the Phoenix tapes are an artefact, an essay of sorts, albeit in a medium which doesn’t easily permit footnotes, but if a student submitted them, with a copy of the script, I can see how, under the NUI grade descriptors, they should be a clear first class mark.

Part of our problem with assessment of digital artefacts is that many academics, as I suggested previously, have never explicitly considered their instinctive grading rubrics. When challenged in discussion on a particular essay, most academics can explain why they gave it the mark which it got, but do not, as a matter of course, have a set of grading rubrics to hand, nor do they provide students with copies of grading rubrics at the start of courses.

This lack of explicit rubrics explains, I think, some of the resistance I see among colleagues to the adoption of structured feedback forms for student work, where they see themselves being driven to ticking boxes on a form rather than “proper” grading.  Equally, I know colleagues who sweat over the difference between a 67% and a 68%, when to me both marks are a high 2.1 (a B+ for those of you outside the Irish and UK grading system), an expression of a qualitative judgement which can never be precisely reduced to a number. I recall a common complaint when the NUI changed the grade break for a 2.1 from 62% down to 60% was that work which, at 61% used to be regarded as a 2.2 class honour would now become a 2.1. To me, it was clear that work which I used to mark as a 61% (not quite a 2.1) should now get a 59% or a 58%. However, the critical point here is that this issue was never really satisfactorily resolved for many people – and if you can’t be clear about the relationship between a certain quality of written work, and how that is expressed as a numeric value for the system, then of course you will not be able to engage with questions of how you assess a performance in a digital medium!

 

Another issue of the Computer Science principle “code doesn’t count”

How will we create marking schema to take account of the sort of creative digital project work students do in the new JC and LC?

 


Posted

in

by

Tags:

Comments

2 responses to “Love in a cold climate: Assessment of Digital Artefacts in an Irish University”

  1. missushopSiobhan Keane Hopcraft Avatar

    Presumably a marking schema similar to that applied to DAH will transfer down to LC only given that JC examinations are soon to be a thing of the past.
    Great article by the way!

  2. […] by writing a set of individual ‘stream of consciousness’ responses to the call – mine is online here. Apparently, some of the DAH MA class type perfect MLA references in their stream of consciousness […]

Leave a Reply

Your email address will not be published. Required fields are marked *

css.php