This report is the second annual report for academic staff on course and unit evaluations in the Arts Faculty. It is based on the latest Monash Experience Questionnaire, Course Experience Questionnaire and Monash Graduate Survey, as well as unit evaluation data from 2007. Our performance on all of these measurements is an increasingly significant issue, especially in light of the role they play in the allocation of the federal government’s Learning and Teaching Performance Fund; in the last three years, the LTPF has provided more than $2.7 million to our Faculty for teaching- and student-related projects.
For individual staff, this report should be read in light of the qualitative comments and quantitative outcomes of your 2007 units. It is intended to provide information on the broad student experience of our teaching approaches and practices. In Arts, for instance, unit evaluations consistently underscore the importance of enthusiasm, intellectual content and participation. They also highlight our most common weaknesses, which are largely to do with the timeliness and nature of feedback. In particular, we need to provide adequate, detailed, consistent, and constructive criticism on all assessment. Furthermore we should ensure that it is returned to students with sufficient time for them to use it to improve their work.
Everyone should be aware of the limitations of statistics obtained from course and unit evaluations. Indeed, there are results in which little confidence can be placed, due to a low response rate (we have set 35% as a reasonable benchmark in any unit) or because there were fewer than ten enrolments. We should also acknowledge that there are various reasons for some units receiving lower numerical evaluations than others. Core and compulsory units, for instance, may contain a higher proportion of students who would not have elected to study a particular topic or theme. Teaching staff may have tried out a particular approach, technique or assessment task that proved challenging and demanding, or was perhaps unsuccessful. There is also some research evidence suggesting that particular student cohorts—and particularly some international students—are very unlikely to give ‘5’ or ‘1’ scores and instead cluster around ‘3’ in their evaluations, while students in areas with a high degree of what might be called ‘practical’ instruction—music, languages and drama, for instance—are more likely than others to give very high or very low scores to those units.
In that light, this report is intended to provide broad indications and encourage staff to reflect upon and review their current and future teaching. It provides some explanations for higher and lower levels of student satisfaction, and encourages changes in the areas that are under our control.
The Monash Experience Questionnaire (MEQ) is administered at all Monash campuses every second year. MEQ examines the experiences of undergraduate and postgraduate coursework students at all campuses. The response rate for the Arts Faculty is generally high, with a rate of approximately 53% in previous MEQs. In 2007 however, the overall response rate for the Arts Faculty dropped to 40.8%. The response rate for the university was 29.7%.
In the majority of comparisons of the study experience between MEQ 2003, MEQ 2005 and MEQ 2007, the Arts Faculty demonstrated an improvement in general satisfaction in 2007. The top ten performing items in 2007 were:
Arts students gave low scores to the following ten items:
The bottom ten items in 2007 were very similar to those in 2003 and 2005, suggesting that there are persistent problems in specific areas. Some of these relate to issues over which the faculty has little control or, like numeracy, that have relatively little relevance to much of our teaching. Others relate to university-wide issues. But some—such as feedback—have more direct bearing on our teaching decisions, while the focus on professional engagement suggests that we need to consider ways of building internships and similar opportunities into our programs.
The areas which showed the greatest increases in general satisfaction from MEQ 2005 to MEQ 2007 were:
| Item | Difference |
|---|---|
| I am generally satisfied with the online classroom environment (eg MUSO, portal) | +0.24 |
| I am able to access information technology resources when I need them | +0.18 |
| I am generally satisfied with my physical classroom environment (eg lecture and tutorial rooms, laboratories) | +0.16 |
| There is sufficient access (e.g. email, online, face to face, telephone) to teaching staff | +0.15 |
| I am generally satisfied with the opportunities I have to develop my language and learning skills | +0.14 |
| Teaching resources (eg books, notes, etc) are appropriate for my needs | +0.14 |
| My course provides me with the opportunity to engage with professionals in my field of study and the community it serves | +0.13 |
| Class sizes at Monash do not impede my learning | +0.11 |
| Teaching facilities (equipment in laboratories, studios, etc) are appropriate for my needs | +0.09 |
| Information and communication technology (e.g. MUSO, online quizzes, podcasting, online discussion groups etc) is used effectively in my course to facilitate my learning | +0.09 |
| The teaching staff encourage me to participate in classroom and/or online discussion when required | +0.09 |
The areas which showed the greatest decrease in satisfaction from MEQ 2005 to MEQ 2007 were:
| Item | Difference |
|---|---|
| I believe that the skills I am learning at Monash will help me work anywhere in the world | -0.31 |
| In areas other than coursework, teaching staff make a genuine effort to assist me with any problems I might be having | -0.29 |
| My course provides me with opportunities to engage in enhancement activities (e.g., in work, research, extra study and internationally) | -0.15 |
| The course develops my ability to work with and interpret data (numeracy) | -0.14 |
| My course makes reference to research to enhance my learning | -0.14 |
| The course develops my ability to use information technology | -0.13 |
| The course improves my skills in oral communication | -0.1 |
| The course sharpens my analytical skills | -0.05 |
| The course develops my ability to conduct research | -0.05 |
| The course develops my confidence to investigate new ideas | -0.05 |
The Course Experience Questionnaire (CEQ) and Graduate Destination Survey (GDS) are administered by Australian tertiary institutions under the national coordination of Graduate Careers Australia. The surveys are intended to give an indication of where graduates go after graduation and how they felt about their course experience. New graduates are invited to respond to the surveys approximately four months after they complete their course of study. In 2006, 11,276 graduates, who graduated from Monash in 2005, completed the surveys. Of these, 54.6% responded to the GDS and 48.9% responded to the CEQ.
The CEQ contains core items which are the same for all universities, and some items which are specific to the institution. These core items are the Good Teaching Scale (GTS), the Generic Skills Scale (GSS), and the Overall Satisfaction Item (OSI). The GTS, GSS and OSI are calculated from items which asked graduates to rate aspects of teaching skills and generic skills, and present a measure of overall satisfaction.
| Year of Graduation | % Agreement | ||
|---|---|---|---|
| Generic Skills Scale | Good Teaching Scale | Overall Satisfaction | |
| 2001 | 71.2% | 54.8% | 72.2% |
| 2002 | 72.7% | 58.1% | 70.5% |
| 2003 | 72.9% | 59.2% | 71.9% |
| 2004 | 72.3% | 59.6% | 73.4% |
| 2005 | 77.2% | 65.2% | 77.7% |
Graduates from the Arts Faculty have reported an increase in their agreement with the core items of generic skills, good teaching and overall satisfaction during their course of study at Monash.
The GDS is a study of the activities of new university graduates, and is conducted at the same time as the CEQ. Graduates are asked about their major activities, including study, employment, or unavailability for study or work. The following table shows the percentage of Monash Arts Faculty students in further study or employment four months after course completion:
| Year of Graduation | Percentage in further study | Percentage in full-time employment (of those available for employment) |
|---|---|---|
| 2001 | 34.8% | 68.1% |
| 2002 | 41.0% | 73.0% |
| 2003 | 51.8% | 69.5% |
| 2004 | 41.0% | 71.4% |
| 2005 | 36.7% | 68.8% |
There was a significant increase in graduate employment between the 2001 and 2002 graduates. Since then however, numbers have fallen away and now remain relatively stable. The number of students going on to further study varies greatly from year to year, with at least a third of students continuing their studies. The 2006 data will be available mid year, and it will be particularly important to monitor the flow to further study.
Student response to units is measured by calculating the mean for individual units, and creating overall means for the faculty. The mean rating for the Arts Faculty over the past two years has been approximately 3.90, which indicates ‘above average’ satisfaction, according to interpretations devised by the university. The Faculty has introduced its own interpretations, based on its average result. The following table shows the interpretations of the mean ratings in our Faculty:
| Band | Number of units |
|---|---|
| Well below Arts average | 0-2.99 |
| Below Arts average | 3.00-3.59 |
| Arts average | 3.60-4.29 |
| Above Arts average | 4.30-4.69% |
| Well above Arts average | 4.70-5.00 |
The response rate for the Arts Faculty increased significantly in 2007. The overall response rate for 2007 was 42.8% (up from 36% in 2006). This is largely due to the trial of paper-based evaluations at Berwick, Caulfield and Clayton in semester 1, which resulted in an overall semester 1 response rate of 48.6%, while the online evaluations in semester 2 returned a response rate of 37.5%. In 2008, we will be conducting paper-based evaluations where possible in both semesters at all campuses, in the hope that the overall response rate will continue to increase.
In 2007, units recording the highest levels of satisfaction usually rated highly on the following questions:
Units recording lower levels of satisfaction usually received below average responses on the following questions:
It is notable that students perceived both a lack of clarity in 'instructions and criteria' and in feedback. This suggests a need for coordinated improvement in areas of assessment guidelines and responses.
The following table shows the number and percentage of units in each of the Arts bands in 2007:
All units:
| Band | Number of units | Percentage |
|---|---|---|
| Well above Arts average | 52 | 5.3% |
| Above Arts average | 161 | 16.3% |
| Arts average | 593 | 60.2% |
| Below Arts average | 155 | 15.7% |
| Well below Arts average | 24 | 2.4% |
| Total | 985 | 100% |
Units with at least 10 enrolments and 35% response rate:
| Band | Number of units | Percentage |
|---|---|---|
| Well above Arts average | 1 | 0.3% |
| Above Arts average | 55 | 14.3% |
| Arts average | 267 | 69.5% |
| Below Arts average | 57 | 14.8% |
| Well below Arts average | 4 | 1.0% |
| Total | 384 | 100% |
As is clear, higher response rates and larger enrolments reduce the proportion of very high and very low ratings, in part because those ratings are more likely to occur in small Honours and postgraduate coursework units, where one or two very happy or very unhappy students can have a major impact on the evaluation.
The surveys above provide an overview of data used to identify poorly performing units so that improvements can be suggested and monitored, as well as exemplary units that will be recognised in the form of Dean’s Commendation Awards. The information is also used for benchmarking in promotion applications (see the following table).
The first table shows the average score of Arts units in comparison to the University for the 8 university-wide questions in the 2007 unit evaluations:
| Area | The learning objectives of this unit were made clear to me | The unit enabled me to achieve the learning objectives | I found the unit to be intellectually stimulating | I found the resources provided for the unit to be helpful | I received constructive feedback on my work | The feedback I received was provided in time to help me improve | The overall amount of work required of me for this unit was appropriate | Overall I was satisfied with the quality of this unit |
|---|---|---|---|---|---|---|---|---|
| University | 3.91 | 3.81 | 3.81 | 3.74 | 3.55 | 3.53 | 3.77 | 3.74 |
| Arts | 4.05 | 3.97 | 4.10 | 3.93 | 3.87 | 3.80 | 4.00 | 3.94 |
The second table shows the average score for the Arts-specific questions in 2007:
| The organisation and progression of the topics in this unit made sense to me | The assessment tasks helped me develop relevant knowledge and skills | I found the instructions and criteria for assessment tasks were clear | In this unit I was encouraged to participate actively | The teaching staff were enthusiastic about teaching the unit | I had sufficient access to teaching staff out of class | This unit conveyed to me the dynamic nature of knowledge in this discipline | I would recommend this unit to other students | The unit was organised in ways that helped off-campus students overcome the difficulties that arise from not being able to attend on-campus classes |
|---|---|---|---|---|---|---|---|---|
| 3.97 | 4.03 | 3.84 | 4.08 | 4.26 | 4.00 | 4.04 | 3.94 | 3.55 |