{"id":932,"date":"2012-12-04T19:56:09","date_gmt":"2012-12-04T19:56:09","guid":{"rendered":"http:\/\/www.mikecosgrave.com\/blog2006\/?p=932"},"modified":"2012-12-04T19:56:09","modified_gmt":"2012-12-04T19:56:09","slug":"love-in-a-cold-climate-assessment-of-digital-artefacts-in-an-irish-university","status":"publish","type":"post","link":"https:\/\/mikecosgrave.com\/blog2006\/random\/love-in-a-cold-climate-assessment-of-digital-artefacts-in-an-irish-university\/","title":{"rendered":"Love in a cold climate: Assessment of Digital Artefacts in an Irish University"},"content":{"rendered":"<p>Last week a group of us set out to write a\u00c2\u00a0piece\u00c2\u00a0\u00c2\u00a0on assessment and evaluation of digital material in response to a call for contributions to the Journal of Digital Humanities. We began by writing fast stream of\u00c2\u00a0consciousness\u00c2\u00a0pieces which we later merged and edited into a\u00c2\u00a0single\u00c2\u00a0piece &#8211; that process is described in another post. This is the\u00c2\u00a0initial\u00c2\u00a0piece of fast writing which I did for our\u00c2\u00a0writing\u00c2\u00a0project, of which the final version appears <a href=\"http:\/\/www.mikecosgrave.com\/blog2006\/?p=925\" target=\"_blank\">online here<\/a><\/p>\n<p>My engagement with assessing digital artefacts in academe is driven by the problem of grading the digital work being produced in our Phd and MA programmes in Digital Arts and Humanities<\/p>\n<p><!--more--><\/p>\n<p>When the DAH Phd consortium developed as a funding proposal for a structured Phd, there was passing note that the outputs could include digital \u00e2\u20ac\u0153somethings\u00e2\u20ac\u009d but no detailed discussion on what those might be, what discount might be applied to the traditional\u00c2\u00a0 80,000 word thesis for the digital artefacts, and what criteria might be applied. Realistically, since the consortium involved 4 different universities and a collection of disciplines, it was probably just as well that we did not seek to engage with this in too much detail, or we would never have made the funding deadline.\u00c2\u00a0 Now, with our own 1 year MA course in UCC, we need to address this issue in a very immediate way \u00e2\u20ac\u201c our students are currently working on digital products which we need to guide, and that guidance needs to be shaped by an clear awareness of what our expectations are, and how we will assess their digital work.<\/p>\n<p>I actually have no real problem with this, but I know it is an insuperable conceptual issue for people outside our course team. Trying to talk about grading with colleagues, many of whom have never explicitly thought about grading because they rely on conventional essays or dissertations, and \u00e2\u20ac\u02dceveryone\u00e2\u20ac\u2122 knows what is good and bad in those, is difficult. Colleagues who have never explicitly considered grade descriptors or grading rubrics cannot begin to conceive of how one might grade an as yet undefined assessed digital object.<\/p>\n<p>Disciplines not only have signature pedagogies, they also have signature assessments, and the skill of grading those is handed down from generation to generation as an artisan craft, which is understood across the community of the discipline so external examiners have no problem validating the marks assigned in their discipline. I have no doubt but that grading is part art, and never wholly science, but in an interdisciplinary field like DH, where we must assess new types of student work, some frameworks are necessary.<\/p>\n<p>In fact, in the dear old National University of Ireland, we have clear and well established guidelines in the NUI grade descriptors. \u00c2\u00a0(insert ref, attach appendix)\u00c2\u00a0 The descriptors clearly lay out, in a general, non-discipline specific way, the sort of \u00e2\u20ac\u0153evidence of a mind at work\u00e2\u20ac\u009d (Joe Lee) we should expect at various grade brackets. They do not explicitly map to Blooms\u00e2\u20ac\u2122 Taxonomy, so beloved of behaviourists as the gospel behind Learning Outcomes, but they do come fairly close. \u00c2\u00a0So a Fail in the NUI descriptors manifests \u00e2\u20ac\u0153a display of some knowledge of the material relevant to the question posed, but with very serious omissions\/errors and\/or major inaccuracies included in answer\u00e2\u20ac\u009d which relates to the lowest level of Blooms\u00e2\u20ac\u2122 Taxonomy. At the other end of the scale, a top first class answer includes \u00e2\u20ac\u0153an outstanding ability to organise, analyse and express ideas and arguments in an original, sophisticated and discriminating manner\u00e2\u20ac\u009d and while we must accept that words like \u00e2\u20ac\u0153sophisticated\u00e2\u20ac\u009d are subject to debate, this seems to me to clearly tick the boxes of the higher levels of Blooms\u00e2\u20ac\u2122 Taxonomy.<\/p>\n<p>&nbsp;<\/p>\n<p>Alongside these two \u00e2\u20ac\u0153foundational\u00e2\u20ac\u009d documents, and easily aligned to them, are any number of rubrics for grading different types of work. The NUI descriptors require not real modification for application to essays and other traditional work. Drawn from that traditional rubrics for \u00e2\u20ac\u0153regular\u00e2\u20ac\u009d student essays, people have produced countless rubrics for assessing blog posts and posting in discussion forums \u00e2\u20ac\u201c these are, after all, written work and different from traditional academic writing mostly in extent, and sometimes in the formality of tone or voice. Equally, it isn\u00e2\u20ac\u2122t actually an enormous stretch to apply the NUI descriptors to developing a rubric for grading twitter; and there are rubrics for those as well.<\/p>\n<p>When we move into less conventional forms that matters become more complicated. How do you assess a database, a critical edition, a performance, a piece of multimedia or an \u00e2\u20ac\u02dcapp\u00e2\u20ac\u2122. \u00c2\u00a0The optimal manner to build a relational database is, at one level, defined by a set of normalisation rules which are pretty clear and preclude, it seems, developing an argument, but at another level the choice of data to capture, the choice of datatypes, indices and relations are all driven by the questions which are informed the particular inquiry being pursued. It just so happens that the vast majority of databases in the real world are designed to track transactions and bill people for things, so they all look very similar. This is particularly true where they need to conform to accounting or legal standards imposed at national or international level. And yet it is still possible for different enterprises in the same field to design databases to perform the same tasks which can differ markedly from one another. It seems to me that databases designed by different students for different inquiries can and should differ, and while at the lowest level of Blooms\u00e2\u20ac\u2122 Taxonomy, you would expect, for example that they would all comply with the SQL standard and be reduced to 3<sup>rd<\/sup> Normal Form, at the higher level you would expect to see individual choices about analysis and synthesis of the material to suit to particular analytic questions being asked of the original raw, pre-digital material.<\/p>\n<p>I recently heard a passing comment on a film \u00e2\u20ac\u201c the Phoenix Tapes \u00e2\u20ac\u201c that \u00e2\u20ac\u0153if it were a proper academic essay, it would of course have footnotes and so on\u00e2\u20ac\u009d (Rod, but let\u00e2\u20ac\u2122s not name him!) as if it was not a proper academic essay. (<a href=\"http:\/\/www.sixpackfilm.com\/en\/catalogue\/show\/1641\">http:\/\/www.sixpackfilm.com\/en\/catalogue\/show\/1641<\/a> ) Now, to my mind, the Phoenix tapes are a very fine collection of 6 essays on Hitchcock \u00e2\u20ac\u201c they chose themes, extract examples, arrange those examples in a structure which, to my eyes, includes a clear progression from introduction, through development to show the often horrific end result of the obsessions, highlighting along the way the manner in which Hitchcock visually expresses these themes. Another more professional description of them is \u00e2\u20ac\u0153 The Phoenix Tapes show re-edited excerpts from 40 films by Alfred Hitchcock. The six chapters focus on a personal selection of various leitmotifx in Hitchcock&#8217;s work. The consequence of this is not only to highlight HItchcock&#8217;s obsessions with certain types of repetitive movements and highly loaded visual signifiers, but to suggest that these actions are part of a universal language of gesture that encompasses both cinematic and everyday modes of communication.\u00c2\u2019 (John Tozer, Camera Austria) (<a href=\"http:\/\/www.sixpackfilm.com\/en\/catalogue\/show\/1641\">http:\/\/www.sixpackfilm.com\/en\/catalogue\/show\/1641<\/a>) If we skate over the detail that the original footage was not digital, the Phoenix tapes are an artefact, an essay of sorts, albeit in a medium which doesn\u00e2\u20ac\u2122t easily permit footnotes, but if a student submitted them, with a copy of the script, I can see how, under the NUI grade descriptors, they should be a clear first class mark.<\/p>\n<p>Part of our problem with assessment of digital artefacts is that many academics, as I suggested previously, have never explicitly considered their instinctive grading rubrics. When challenged in discussion on a particular essay, most academics can explain why they gave it the mark which it got, but do not, as a matter of course, have a set of grading rubrics to hand, nor do they provide students with copies of grading rubrics at the start of courses.<\/p>\n<p>This lack of explicit rubrics explains, I think, some of the resistance I see among colleagues to the adoption of structured feedback forms for student work, where they see themselves being driven to ticking boxes on a form rather than \u00e2\u20ac\u0153proper\u00e2\u20ac\u009d grading.\u00c2\u00a0 Equally, I know colleagues who sweat over the difference between a 67% and a 68%, when to me both marks are a high 2.1 (a B+ for those of you outside the Irish and UK grading system), an expression of a qualitative judgement which can never be precisely reduced to a number. I recall a common complaint when the NUI changed the grade break for a 2.1 from 62% down to 60% was that work which, at 61% used to be regarded as a 2.2 class honour would now become a 2.1. To me, it was clear that work which I used to mark as a 61% (not quite a 2.1) should now get a 59% or a 58%. However, the critical point here is that this issue was never really satisfactorily resolved for many people \u00e2\u20ac\u201c and if you can\u00e2\u20ac\u2122t be clear about the relationship between a certain quality of written work, and how that is expressed as a numeric value for the system, then of course you will not be able to engage with questions of how you assess a performance in a digital medium!<\/p>\n<p>&nbsp;<\/p>\n<p>Another issue of the Computer Science principle \u00e2\u20ac\u0153code doesn\u00e2\u20ac\u2122t count\u00e2\u20ac\u009d<\/p>\n<p>How will we create marking schema to take account of the sort of creative digital project work students do in the new JC and LC?<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Last week a group of us set out to write a\u00c2\u00a0piece\u00c2\u00a0\u00c2\u00a0on assessment and evaluation of digital material in response to a call for contributions to the Journal of Digital Humanities. We began by writing fast stream of\u00c2\u00a0consciousness\u00c2\u00a0pieces which we later merged and edited into a\u00c2\u00a0single\u00c2\u00a0piece &#8211; that process is described in another post. This is [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-932","post","type-post","status-publish","format-standard","hentry","category-random"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/mikecosgrave.com\/blog2006\/wp-json\/wp\/v2\/posts\/932","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mikecosgrave.com\/blog2006\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mikecosgrave.com\/blog2006\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mikecosgrave.com\/blog2006\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mikecosgrave.com\/blog2006\/wp-json\/wp\/v2\/comments?post=932"}],"version-history":[{"count":0,"href":"https:\/\/mikecosgrave.com\/blog2006\/wp-json\/wp\/v2\/posts\/932\/revisions"}],"wp:attachment":[{"href":"https:\/\/mikecosgrave.com\/blog2006\/wp-json\/wp\/v2\/media?parent=932"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mikecosgrave.com\/blog2006\/wp-json\/wp\/v2\/categories?post=932"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mikecosgrave.com\/blog2006\/wp-json\/wp\/v2\/tags?post=932"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}