This report is the first of what will now be annual reports for academic staff on course and unit evaluations in the Arts Faculty. It is based on the latest Monash Experience Questionnaire, Course Experience Questionnaire and Monash Graduate Survey, as well as unit evaluation data from 2006. Our performance on all of these measurements is an increasingly significant issue, especially in light of the role they play in the allocation of the federal government's Learning and Teaching Performance Fund; in the last two years, the LTPF has provided more than $1.5 million to our Faculty for teaching - and student-related projects.
For individual staff, this report should be read in light of the qualitative comments and quantitative outcomes of your 2006 units. It is intended to provide information on the broad student experience of our teaching approaches and practices. In Arts, for instance, unit evaluations consistently underscore the importance of enthusiasm, intellectual content and participation. They also highlight our most common weaknesses, which are largely to do with the timeliness and nature of feedback.
Everyone should be aware of the limitations of figures obtained from course and unit evaluations. Indeed, there are results in which little confidence can be placed, due to a low response rate (we have set 35% as a reasonable benchmark in any unit) or because there were fewer than ten enrolments. We should also acknowledge that there are various reasons for some units receiving lower numerical evaluations than others. Core and compulsory units, for instance, may contain a higher proportion of students who would not have elected to study a particular topic or theme. Teaching staff may have tried out a particular approach, technique or assessment task that proved challenging and demanding, or was perhaps unsuccessful. There is also some research evidence suggesting that particular student cohorts - and particularly some international students - are very unlikely to give '5' or '1' scores and instead cluster around '3' in their evaluations, while students in areas with a high degree of what might be called 'practical' instruction - music, languages and drama, for instance - are more likely than others to give very high and very low scores to those units.
In that light, this report is intended to provide broad indications and encourage staff to reflect upon and review their current and future teaching. It provides some explanations for higher and lower levels of student satisfaction, and encourages changes in the areas that are under our control.
The Monash Experience Questionnaire (MEQ) is administered at all Monash campuses every second year. MEQ looks at the experiences of undergraduate and postgraduate coursework students at all campuses. The response rate for the Arts Faculty is generally high, with a rate of approximately 53% in previous MEQs. In 2005, the overall response rate for the university was 34.1%; this gives Arts (52.6%) the second highest response rate across all faculties.
In all comparisons of the study experience between MEQ 2003 and MEQ 2005, the Arts Faculty demonstrated an improvement in general satisfaction in 2005. The top ten performing items in 2005 were:
Arts students gave low scores to the following ten items:
The bottom ten items in 2005 were very similar to those in 2003, suggesting that there are persistent problems in specific areas. Some of these relate to issues over which the faculty has little control or, like numeracy, that have relatively little relevance to most of our teaching. Others relate to university-wide issues. But some - such as feedback - have more direct bearing on our teaching decisions, while the focus on professional engagement suggests that we need to consider ways of building internships and similar opportunities into our programs.
Students were also able to comment on the best aspects of their course, and the areas most in need of improvement. Their comments were analysed and divided into five categories (assessment, course design, outcomes, staff, and support), each containing several sub-categories.
In MEQ 2005, the areas that were most in need of improvement were in the categories of support (41.7%) and course design (31.1%). Infrastructure (class size and facilities) and the sense of belonging were the areas of greatest concern in the support category. In the course design category, the areas most in need of improvement concerned the methods of teaching and learning, the flexibility of the course design, and the course structure and expectations. These were very similar to the concerns expressed by students in MEQ 2003.
Significantly, these comments about the areas for improvement were similar to the comments about the best aspects of the course. While this may seem to be contradictory, it also suggests that the experiences of students within the Arts Faculty are divergent, and that the faculty currently provides an uneven educational experience across different schools, majors, campuses and student cohorts.
The Monash Experience Questionnaire (MEQ) is administered at all Monash campuses every second year. MEQ looks at the experiences of undergraduate and postgraduate coursework students at all campuses. The response rate for the Arts Faculty is generally high, with a rate of approximately 53% in previous MEQs. In 2005, the overall response rate for the university was 34.1%; this gives Arts (52.6%) the second highest response rate across all faculties.
In all comparisons of the study experience between MEQ 2003 and MEQ 2005, the Arts Faculty demonstrated an improvement in general satisfaction in 2005. The top ten performing items in 2005 were:
Arts students gave low scores to the following ten items:
The bottom ten items in 2005 were very similar to those in 2003, suggesting that there are persistent problems in specific areas. Some of these relate to issues over which the faculty has little control or, like numeracy, that have relatively little relevance to most of our teaching. Others relate to university-wide issues. But some - such as feedback - have more direct bearing on our teaching decisions, while the focus on professional engagement suggests that we need to consider ways of building internships and similar opportunities into our programs.
Students were also able to comment on the best aspects of their course, and the areas most in need of improvement. Their comments were analysed and divided into five categories (assessment, course design, outcomes, staff, and support), each containing several sub-categories.
In MEQ 2005, the areas that were most in need of improvement were in the categories of support (41.7%) and course design (31.1%). Infrastructure (class size and facilities) and the sense of belonging were the areas of greatest concern in the support category. In the course design category, the areas most in need of improvement concerned the methods of teaching and learning, the flexibility of the course design, and the course structure and expectations. These were very similar to the concerns expressed by students in MEQ 2003.
Significantly, these comments about the areas for improvement were similar to the comments about the best aspects of the course. While this may seem to be contradictory, it also suggests that the experiences of students within the Arts Faculty are divergent, and that the faculty currently provides an uneven educational experience across different schools, majors, campuses and student cohorts.
The Course Experience Questionnaire (CEQ) and Graduate Destination Survey (GDS) are administered by Australian tertiary institutions under the national coordination of Graduate Careers Australia. The surveys are intended to give an indication of where graduates go after graduation and how they felt about their course experience. New graduates are invited to respond to the surveys approximately four months after they complete their course of study. In 2005, 11,450 graduates who graduated from Monash in 2004 completed the surveys. Of these, 36% responded to the GDS and 26.5% responded to the CEQ.
The CEQ contains core items which are the same for all universities, and some items which are specific to the institution. These core items are the Good Teaching Scale (GTS), the Generic Skills Scale (GSS), and the Overall Satisfaction Item (OSI). The GTS, GSS and OSI are calculated from items which asked graduates to rate aspects of teaching skills and generic skills, and measure the percentage of graduates that agree with each item.
The following table shows the percentage of agreement by graduates of the Arts Faculty over the past five years:
| Year | % Agreement | ||
|---|---|---|---|
| Generic Skills Scale | Good Teaching Scale | Overall Satisfaction | |
| 2002 | 72.8% | 54.6% | 72.0% |
| 2003 | 72.9% | 56.5% | 70.0% |
| 2004 | 74.2% | 59.4% | 73.1% |
| 2005 | 72.8% | 57.6% | 73.1% |
| 2006 | 79.2% | 65.9% | 80.3% |
As is clear, graduates from the Arts Faculty are increasingly likely to express satisfaction with the core items of generic skills and good teaching.
The GDS is a study of the activities of new university graduates, and is conducted at the same time as the CEQ. Graduates are asked about their major activities, including study, employment, or unavailability for study or work. The data are not Monash-specific however, and the following percentages are for graduates of all Australian tertiary institutions:
| Year of Graduation | Percentage in full-time study | Percentage in full-time employment 4 months after completion | ||
|---|---|---|---|---|
| Humanities | Languages | Visual/Performing Arts | ||
| 2001 | 23% | 74% | 78% | 60% |
| 2002 | 24% | 67% | 71% | 57% |
| 2003 | 22% | 67% | 73% | 54% |
| 2004 | 23% | 67% | 72% | 56% |
| 2005 | 23% | 71% | 75% | 60% |
There was a significant drop in graduate employment between the 2001 and 2002 graduates. Since then, however, the numbers have remained relatively stable, with an increase in the employment of Arts graduates in 2005. The number of students going on to further study has also remained consistent, with almost a quarter of students continuing their studies.
Student response to units is measured by calculating the mean for individual units, and creating overall means for the faculty. The mean rating for the Arts Faculty over the past two years has been approximately 3.90, which indicates 'above average' satisfaction, according to interpretations devised by the university. The Faculty has introduced its own interpretations, based on its average result. The following table shows the interpretations of the mean ratings in our Faculty:
| Mean Rating | Interpretation |
| 0-2.99 | Well below Arts average |
| 3.00-3.59 | Below Arts average |
| 3.60-4.29 | Arts average |
| 4.30-4.69 | Above Arts average |
| 4.70-5.00 | Well above Arts average |
The response rate for the Arts Faculty has been fairly low (approximately 35%) since the unit evaluations changed from paper-based to web-based. In semester 1, 2007, we are trialling a return to paper-based evaluations for some on-campus units, to ascertain whether this will improve the response rate.
In 2006, units recording the highest levels of satisfaction usually rated highly on the following questions:
Units recording lower levels of satisfaction usually received below average responses on the following questions:
The following table shows the number and percentage of units in each of the Arts bands in 2006:
All units:
| Band | Number of units | Percentage |
|---|---|---|
| Well above Arts average | 46 | 4.7% |
| Above Arts average | 150 | 15.3% |
| Arts average | 589 | 60.2% |
| Below Arts average | 158 | 16.1% |
| Well below Arts average | 36 | 3.7% |
| Total | 979 | 100% |
Units with at least 10 enrolments and 35% response rate:
| Band | Number of units | Percentage |
|---|---|---|
| Well above Arts average | 2 | 0.6% |
| Above Arts average | 28 | 8.7% |
| Arts average | 228 | 70.8% |
| Below Arts average | 59 | 18.3% |
| Well below Arts average | 5 | 1.6% |
| Total | 322 | 100% |
As is clear, higher response rates and larger enrolments reduce the proportion of very high and very low ratings, in part because those ratings are more likely to occur in small Honours and postgraduate coursework units, where one or two very happy or very unhappy students can have a major impact on the evaluation.
Staff applying for promotion will find the following information useful for those parts of the application that ask for benchmarking.
The first table shows the average score of Arts units in comparison to the University for the 8 university-wide questions in the 2006 unit evaluations:
| Area | The learning objectives of this unit were made clear to me | The unit enabled me to achieve the learning objectives | I found the unit to be intellectually stimulating | I found the resources provided for the unit to be helpful | I received constructive feedback on my work | The feedback I received was provided in time to help me improve | The overall amount of work required of me for this unit was appropriate | Overall I was satisfied with the quality of this unit |
|---|---|---|---|---|---|---|---|---|
| University | 3.88 | 3.78 | 3.79 | 3.71 | 3.54 | 3.50 | 3.74 | 3.70 |
| Arts | 3.99 | 3.93 | 4.11 | 3.91 | 3.86 | 3.79 | 3.96 | 3.91 |
The second table shows the average score for the Arts-specific questions in 2006:
| The organisation and progression of the topics in this unit made sense to me | The assessment tasks helped me develop relevant knowledge and skills | I found the instructions and criteria for assessment tasks were clear | In this unit I was encouraged to participate actively | The teaching staff were enthusiastic about teaching the unit | I had sufficient access to teaching staff out of class | This unit conveyed to me the dynamic nature of knowledge in this discipline | I would recommend this unit to other students | The unit was organised in ways that helped off-campus students overcome the difficulties that arise from not being able to attend on-campus classes |
|---|---|---|---|---|---|---|---|---|
| 3.93 | 4.01 | 3.80 | 4.08 | 4.25 | 3.98 | 4.02 | 3.93 | 3.55 |