TLCONF

This Forum will integrate three bodies of research in post-secondary education: practices in effective teaching evaluation; institutional teaching culture; the nature of change and leadership in post-secondary institutions.

An extensive literature review (Wright et al., 2014a) identified four recurring themes required for effective teaching evaluation: shared understandings of quality (Arreola, 2007; Hénard & Roseveare, 2012); multi-faceted data and evaluation (Arreola, 2007; Berk, 2009, 2014; Buller, 2012); robust feedback cycles (Gibbs & Coffey, 2004; Piccinin, 2003; Theall & Franklin, 2001); and sustained leadership for education, engagement, and change (Arreola, 2007; Gravestock, 2011; Hénard & Roseveare, 2012). Although national survey data are not available, the University of Windsor led a provincial survey of Ontario universities’ teaching evaluation practices which identified gaps and challenges across all of these themes (Wright et al., 2014a). Research based on successful institutional teaching evaluation initiatives – launched in part to mediate these gaps – will form a core element of the material presented at the Forum. Topics include: adaptation of empirically developed frameworks for promotion and tenure teaching criteria (Chalmers et al., 2014); documentation of instructor strategies for teaching evaluation data (Hativa, 2013); ethical data use and publication; instructor-customizable student ratings of instruction (Gravestock & Gregor-Greenleaf, 2008); visualization tools for the study of teaching evaluation data; annotation tools for teaching dossiers (Graniero & Hamilton, 2016); and the use of teaching dossiers for presenting evidence.

A post-secondary institution’s culture consists of its embedded patterns, behaviours, shared values, beliefs, and ideologies (Cox et al., 2011; Kustra et al., 2014), as well as numerous micro-cultures (Mårtensson & Roxå, 2016). For example, an institution’s teaching culture might involve a shared campus commitment to teaching excellence (Bergquist & Pawlak, 2008). Whether, and how, an institution values teaching can impact critical outcomes such as student learning (Cox et al., 2011), student engagement (Grayson & Grayson, 2003), and student retention (Berger & Braxton, 1998), as well as faculty motivation and behaviour (Feldman & Paulsen, 1999). Implementing effective, evidence- based, fair evaluation of teaching practices has been frequently identified as one of the critical elements influencing whether an institution values teaching, and is an indicator of a strong teaching culture. Teaching evaluation practices are designed, implemented, and employed through that culture and in support of that culture (Graniero & Hamilton, 2016). They are a key practice through which the culture’s values are articulated, reinforced, and replicated, and have a powerful though not always linear effect on how people operate within that culture.

Change initiatives intended to improve teaching evaluation are often supported in principle, but are challenging in reality (Hénard, 2010). The difficulty is not so much technical – identifying the right tools and techniques – as it is cultural. Making changes to teaching evaluation practices intervenes in an institution’s culture, often questioning and reshaping values, with implications for both patterns of behaviour and identities. As Arreola (2007) puts it: “The real problem lies in getting large numbers of intelligent, highly educated, and independent people to change their behaviour” (p. xxiv). Deeply held beliefs – many of them myths and misconceptions – regarding legitimacy of teaching evaluation, coupled with overreliance on student ratings of instruction as a single source of data, are challenging barriers to engagement (Hativa, 2013). Without systematic mediation, resulting methods and tools often do not reflect instructor experience of what is valuable in teaching, or how teaching works (Allen et al., 2015). More generally Sterman (2006) describes the ways in which people understand, interpret, react to, and evade imposed measures in systems similar to universities, resulting in highly unpredictable and often counterintuitive outcomes. For these reasons, it is becoming increasingly clear that approaches strongly grounded in systematic attention to teaching culture, institutional context, and the nature of effective change leadership in post-secondary institutions are critical to effective and sustained change in teaching evaluation.

Very little has been written about educational leadership in Canadian post-secondary contexts (Wright et al., 2014b). However extensive European and Australian research suggests that “distributed” leadership, which disperses the powers and responsibilities of leadership amongst multiple individuals and groups at multiple levels of the university (see Bolden et al., 2009; Roxå & Mårtensson, 2013; Southwell & Morgan, 2009), is most effective. The distributed leadership model reflects how hierarchies and knowledge networks interact in academic settings. Members of the university work through social and information networks to navigate and make meaning across the system. Over time, emergent leaders develop within these significant networks whether or not they occupy a role of formal authority. Because of the powerful influence of these significant networks, distributed leadership can be an effective way to bring about change in complex adaptive systems: however, these leaders operate most effectively in a context of constructive collaboration with the formal leadership of their institutions. Coordination of “top-down” and “bottom-up” perspectives and activities has been identified as a central challenge of institutional leadership (Bolden et al., 2009). This model matched the conditions and practices found through an environmental scan of educational leadership (Wright et al., 2014b). The model conceptualized in the study was strongly influenced by the work of Torgny Roxå, the keynote speaker for the Forum. This event is an opportunity to explore the potential of a distributed leadership change model while focusing on teaching evaluation as a specific element of institutional practice, providing opportunities for institutional change and collaboration, to enhance cultures that value teaching in order to improve teaching and learning across the Canadian and international post-secondary context.

References

Allen, A., Caron, M., Forrest, A., Liddle, M., Sabourin, B. & Wright, A. (2015, November). Evaluation Vignettes. Invited panel discussion at Weighed in the balance: Evaluating teaching in higher education. University of Windsor: Windsor, ON.

Arreola, R. (2007). Developing a comprehensive faculty evaluation system: A guide to designing, building and operating large-scale faculty evaluation systems. San Francisco: Anker Publishing

Berger, J.B. & Braxton, J.M. (1998). Revising Tinto’s interactionalist theory of student departure through theory elaboration: Examining the role of organizational attributes in the persistence process. Research in Higher Education, 39(2), 103-119.

Bergquist, W.H. & Pawlak, K. (2008). Engaging the six cultures of the academy: Revised and expanded edition of the four cultures of the academy (2nd ed.). San Francisco: Jossey-Bass.

Berk, R.A. (2009). Using the 360° multisource feedback model to evaluate teaching and professionalism. Medical Teacher, 31(12), 1073-1080.

Berk, R.A. (2014). Should student outcomes be used to evaluate teaching? Journal of Faculty Development, 28(2), 87-96.

Bolden, R., Petrov, G., & Gosling, J. (2009). Distributed leadership in higher education: Rhetoric and reality. Educational Management Association and Leadership, 37(20), 257-277.

Buller, J. (2012). Best practices in faculty evaluation: A practical guide for academic leaders. Jossey-Bass: San Francisco.

Chalmers, D., Cumming, R., Elliot, S., Stoney, S., Tucker, B., Wicking, R., & Jorre de St Jorre, T. (2014). Australian University Teaching Criteria and Standards Project Final Report. Retrieved from http://uniteachingcriteria.edu.au/wp-content/uploads/2013/11/Draft-SP12-2335-Project-Final-Report-21-July-2014.pdf

Cox, B.E., McIntosh, K.L., Reason, R.D., & Terenzini, P.T. (2011). A culture of teaching: Policy, perception, and practice in higher education. Research in Higher Education, 52(8), 808-829.

Feldman, K.A., & Paulsen, M.B. (1999). Faculty motivation: The role of a supportive teaching culture. New Directions for Teaching and Learning, 1999(78), 69-78.

Gibbs, G. & Coffey, M. (2004). The impact of training of university teachers on their teaching skills, their approach to teaching and the approach to learning of their students. Active Learning in Higher Education, 5(1), 87-100.

Graniero, P. & Hamilton, B. (2016). Making Sense of Teaching Evaluations: the Value of Noise. Paper presented at the annual conference of the Society for Teaching and Learning in HigherEducation. Western University: London (ON)

Gravestock, P. (2011). Does teaching matter? The role of teaching evaluation in tenure policies at selected Canadian universities. (Unpublished doctoral dissertation). University of Toronto, Toronto, Ontario.

Gravestock, P. & Gregor-Greenleaf, E. (2008). Student course evaluations: Research, models and trends. Toronto: Higher Education Quality Council of Ontario.

Grayson, J.P. & Grayson, K. (2003). Research on retention and attrition (No. 6). Montreal: The Canada Millennium Scholarship Foundation.

Hativa, N. (2013). Student ratings of instruction: Recognizing effective teaching. USA: Oron Publications.

Hénard, F. (2010). Learning our lesson: Review of quality teaching in higher education. Organization for Economic Cooperation and Development. Retrieved from: https://www.oecd.org/edu/imhe/44058352.pdf

Hénard, F. & Roseveare, D. (2012). Fostering quality teaching in higher education: Policies and practices. An Institutional Management in Higher Education (IMHE) guide for higher education Institutions. Retrieved from http://www.oecd.org/edu/imhe/QT%20 policies%20and%20practices.pdf

Kustra, E., Doci, F., Meadows, K., N., Dishke Honzel, C., Goff, L., Gabay, D., Wolf, P., Ellis, D., Grose, J., Borin, P., Hughes, S. (2014). Teaching culture indicators: Enhancing quality teaching. Report to the Ministry of Training, Colleges and Universities Productivity and Innovation Fund Program, University of Windsor, Ontario. Retrieved from http://www1.uwindsor.ca/ctl/system/files/Teaching_Culture_Indicators.pdf

Mårtensson, K., & Roxå, T. (2016). Working with networks, microcultures and communities. Advancing practice in academic development, 174-187.

Piccinin, S.J. (2003). Green Guide 4: Feedback key to learning. Society for Teaching and Learning in Higher Education. London, ON.

Roxå, T. & Mårtensson, K. (2013). Significant networks for educational development. Retrieved at http:// www.konferenslund.se/pp/CEQ_Roxa_Martensson.pdf

Southwell, D. & Morgan, W. (2009). Leadership and the impact of academic staff development and leadership development on student learning outcomes in higher education: A review of the literature. Australian Learning and Teaching Council (ALTC), Queensland University ofTechnology.

Sterman, J.D. (2006). Learning from evidence in a complex world. American Journal of Public Health, 96(3), 505-514. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1470513/

Theall, M. & Franklin, J. (2001). Looking for bias in all the wrong places: A search for truth or a witch hunt in student ratings of instruction? In P. Theall, L. Abrami, & L. Mets (Eds.) The student ratings debate: Are they valid? How can we best use them? New Directions in Educational Research, 109. San Francisco: Jossey-Bass.

Wright, A., Hamilton, B., Mighty, J., Muirhead, B., & Scott, J. (2014a). The Ontario universities’ teaching evaluation toolkit: A feasibility study. Report to the Ministry of Training, Colleges andUniversities – Productivity and Innovation Fund Program. University of Windsor: Windsor, ON.

Wright, A., Hamilton, B., Raffoul, J. & Marval, P. (2014b). Embedded educational leadership initiatives at the University of Windsor: A Ministry of Training, Colleges, and Universities Productivity and Innovation Fund Initiative (July 2014). Contributors: T. Ackerson, D.Andrews, J. Bornais, J. Dixon, L. Gil, E. Kustra, S. McMurphy, M. Potter and N. Timperio.University of Windsor: Windsor, ON.