|
Report of the Task Force on Assessing and Improving Teaching and Learning at Indiana State University October 1998
Executive Summary |
Most faculty and administrators readily acknowledge that the current systems for evaluating teaching are unsatisfactory. Given that teaching is a major part of the mission of Indiana State University, it is clear that a comprehensive, systematic, campus-wide model is needed to provide faculty adequate feedback on their pedagogy and to allow the campus community to examine its teaching strengths and weaknesses. In response to this need, the Provost appointed a university task force to develop a model designed to assess and improve teaching at our university. This report is the result of the work of this task force.
The task force's proposed model is grounded on a framework of Guiding Principles. It uses the scholarship on college teaching to identify six major areas where faculty expertise will enhance the learning success of our students and uses them as the components around which a review of teaching is constructed. It acknowledges that there are multiple sources of data that contribute to making fair and consistent evaluations regarding these components and offers a summary of how each of these sources can contribute to the evaluation process. Finally, using a common rating system, the model presents a matrix which allows enough flexibility for each unit to discuss and select the weights of components and data sources that match departmental values.
This report concludes with the recommendation that, after some campus discussions, select units first adopt this model as a pilot program before the model is applied on a university-wide scale. Doing so would ensure a careful and well-thought out implementation of the proposed model. This purposefully deliberate implementation should be closely monitored by both the faculty and the administration to assure that the resultant teaching evaluation system becomes a valuable part of a university culture committed to student success.
Unmet Needs
Academics tend to regard teaching as an unaccountable art. You either have the gift or you don't. If you possess the ability to inspire students, to organize knowledge into teachable moments and semesters of instruction, the assumption is made that this admirable ability will be recognized intuitively by peers and probably reflected in student answers to standardized evaluations. Moreover, conceived of as a talent expressed in many fashions, teaching also enjoys tremendous protection in the name of academic freedom, which provides cover from quantifiable and externally validated measurements, even to the point of obscuring the need to assess and evaluate teaching with the same rigor as scholarship. Many tacitly agree that publishing demonstrates academic competence, whereas teaching is approached so variably and conceptualized so vaguely that no one can agree on how to gather information about it ("assessment") and how, reliably, to interpret the resulting data ("evaluation").
A review of our assumptions about assessing and evaluating teaching is warranted by new evidence of increased focus on pedagogy and decreased satisfaction with the balance of teaching and research in rewards systems (Dey et al. 1993; Boyer 1990, 23-41). These national doubts call attention to issues at home, such as pay-for-performance. Our campus has observed warning signs about perceived inequities between recognition for teaching achievement versus scholarly production.
Because pedagogy is considered complex and unknowable, the perfect time to re-examine teaching always seems to be "next year" or "never." Such delays are no longer tolerable, given improved tools for assessing and more thoughtful, systematic methods of evaluating teaching. A reasonable starting point is a survey of the teaching culture on our campus.
Local History
Indiana State University, despite having advanced its goals in the 1970s beyond the "Teacher's College" level, still pursues the idea that the classroom counts. Recruiters point to comparatively low faculty-to-student ratios and market the University by touting an unintimidating 1950s' mood of "college the way it is supposed to be." Traditional teaching styles, like the lecture method or the master/apprentice relationship, seem to earn high marks from parents, though their 1990s children have been struggling to keep up grades and graduate, or even to persist in college. Whether ISU's troubling retention level is influenced more by teaching methods or by inadequate academic preparation coupled with distracting financial and social factors is the focus of close scrutiny by the Lilly Project (Chesebro, 1997). Added to the traditional-aged students who are at risk of "slipping out" of academic life, the student population includes a cohort of displaced workers who attend ISU to upgrade job skills, plus another cadre of mothers and fathers who never had a chance to go to college.
Change is a fact of university life, on both sides of the desk. Professors teach different students than they ever met in graduate school. Information-intensive teaching methods aren't working as well as they apparently did during the "golden years" of the G.I. Bill, an era of post-WWII educational enthusiasm. Renewed attention to pedagogy is one response to the search for equilibrium between attracting the students whom professors wish for and serving the ones their universities can retain. Some teachers, who are reluctant to blame the students for underpreparedness, seek changes in teaching approach, learning communities, and classroom assessment; other teachers urge different recruiting strategies and admissions policies. Within the instructional realm--the area classroom teachers can most readily affect--new methods and measures are needed to discover which approaches should be kept and which can be improved.
The lack of a carefully examined and adapted system of evaluating teaching prevents intelligent analysis of rich opportunities for self-improvement and enhancement of the learning environment. No longer trustworthy are the instruments and practices inherited from a time when failed students would be replaced by other diploma seekers. And in the context of that anxiety-producing pressure induced by linking salary adjustments to demonstrable performance, teaching effectiveness is hard enough to document within one's own unit and harder still to compare among departments and across schools.
The Three Wise Professors of Gotham
Consider this accumulated worry from the viewpoint of three typical university teachers. Full Professor Oldcastle is a well-respected, capable lecturer. She seriously doubts whether high schools and community colleges give enough of a head start to students generally, and particularly to the ones who can't write, don't think, and won't come to office hours for enlightenment. She is frustrated that her teaching has stagnated. Associate Professor Henryson approaches his tenure year with some apprehension because his modest publications aren't balanced by any indicators more substantial than Student Instructional Reports, whose averages for "overall quality of instruction" his department's personnel committee had not found very impressive. He wonders whether he can sneak through the screening process on the basis of minimal evidence and the hope that rumors of his teaching skill will be heard by the right ears. Assistant Professor Hotspur is having second thoughts about her new employers at Indiana State. Will they reward the cutting- edge pedagogy she began to use as a T.A. in grad school? Without any endorsed indicator of teaching effectiveness, and lacking a teaching support service beyond a drop-in Learning/Teaching center, she wonders whether she should have taken that job at Mega Junior College. Once, the lure of teaching and doing scholarship at a medium-sized university had overcome most of her doubts. Now worries remain.
The hypothetical cases of each of these professors in early, middle, and later stages of their careers will convey the practical meaning of high-level choices about teaching evaluation. Teachers' needs differ depending on their place in the span between first paycheck and the retirement tea.
Call and Response
In January 1998, Provost Wells solicited sixteen faculty members for a task force to look again at teaching's effects on learning, within the context of assessable goals and appropriate rewards. He charged the group 1) to describe a conceptual model for the improvement of teaching assessment and evaluation,
2) to propose a set of steps for implementing the recommended model, and 3) to put forward incentives that would reward validated achievement.
Alan Lacy chaired this group of independently-minded, vocal, and dedicated teacher-colleagues. We met regularly during the spring semester, to debate guiding principles and investigate particular practices and constituencies. During the summer, a quintet drafted sub-committee reports and composed a working manuscript. Then, at the start of the fall semester, the entire group criticized and revised the document, always mindful that other universities' best practices might not fit our needs. Abstract schemes would not necessarily work here, either.
The following survey of options and recommended approaches to teaching evaluation was conducted with a realistic appreciation of effects a new evaluative tool will have on teachers who have the honor of working at the Indiana State University in this stressful yet stimulating stage of its history.
A program for evaluating teaching must be designed to inform and guide faculty and administrators in making judgments affecting the quality of the university's teaching and learning environment. This assessment system should reflect best principles and practices. It should be grounded in the scholarship of effective teaching and learning. It should apply fairly and reasonably to all segments of the teaching faculty.
A. An effective teaching evaluation program must be comprehensive and systematic.
Because teaching is the central aspect of the university, it is essential that everyone engaged in teaching be evaluated annually. While evaluation procedures may necessarily be adapted according to teaching areas, the overall procedures for conducting teaching evaluations should be integrated into a campus-wide evaluation system so that results fairly represent the quality of teaching throughout the campus.
B. A sound teaching evaluation program must be designed according to the scholarship on teaching and learning.
A teaching evaluation program should measure qualities that are known to affect the success of learners. Despite the fact that there is lively debate among pedagogical scholars, there are also broad areas of consensus about teaching practices and teacher qualities. These areas of agreement should be reflected in the design of a teaching evaluation program.
A teaching evaluation system must integrate two distinct, but connected, aspects: formative and summative evaluations. To do one without the other produces a less effective evaluation system.
A. One purpose of a teaching evaluation system must be to provide valid, summative information regarding the quality of teaching.
An evaluation program must provide a report that offers valid indicators of the quality of teaching. These measures must provide comparable ratings for faculty at all levels and across all teaching areas. These ratings should be the primary summative documentation used for promotion, tenure, and performance evaluations.
B. The second purpose of a teaching evaluation program should be to provide formative direction for faculty improvement efforts.
Evaluation of teaching should not exist separately from a faculty development program. The design of faculty evaluation procedures and reports should provide information about areas of strength and those that could use improvement. Campus faculty development programs should be related to this process.
An effective teaching evaluation model relies on procedures that assure the collection of reliable, valid, and adequate information. It also ensures the generation of reports reflecting fair and responsible interpretations of that information
A. Evaluations should include multiple sources of data collected from students, peers, administrators, and self-evaluation.
Because teaching involves many dimensions, it is best to gather information from a number of sources. Each different source should provide that information about the instructor's teaching with which it is most familiar.
B. Every course must be evaluated using a standardized university procedure.
To assure continual improvement of instructional quality, it is essential to collect information about teaching in an ongoing, systematic way. It is invaluable to generate consistent, comparable results over time and across courses, so that patterns may be discerned and acted on. While faculty or departments may also choose to administer more specific evaluations, a standardized form administered by the university should be used in each course.
An evaluation system must be organized, managed, and adjusted to assure its integrity as well as its ability to enhance the teaching and learning environment at the university.
A. The primary authority to interpret the data from teaching evaluations lies with individual faculty members and his or her department.
Because there are multiple and complex factors involved in the interpretation of information, particularly in relation to the educational character and mission of each department, the evaluation system must be responsive to the individual's and the department's priorities, norms, and
positions--provided this responsiveness exists within the general parameters of the overall evaluation system itself.
B. Faculty are entitled to a fair and consistent evaluation of data regarding their teaching.
In order to reduce over-reliance on subjective representations of individuals' teaching performance, the evaluation system must aim to utilize reliable data collection methods and clear criteria for making interpretations and judgments. It is the primary role of the administration to ensure the integrity of this process when summative evaluations are made.
Scholarship on the Components of Effective College Teaching
Teaching evaluation systems should be organized around characteristics crucial to the success of the teaching endeavor. As John Murray asserts, "If colleges and universities intend to include standards for good teaching when making personnel decisions, they need to be able to recognize good teaching" (1995: 61). He claims, further, that along with general research of learning and teaching, cognitive psychology is helping to reveal demonstrable characteristics of good teaching. This section aims to categorize some of the qualities scholars have identified as fundamental to student success. The sources used in this section are drawn from studies that summarize the relevant literature. What follows is an effort to consolidate this information into clusters of specific areas known to have an impact on the quality of teaching and learning.
I. Teacher's Content Expertise
A teacher's knowledge base in a subject area is fundamental to the creation and enhancement of students' opportunities to learn well.
A. Effective teachers understand their academic field well.
It is obvious that teachers must know the material well in order to instruct students accurately. More importantly, a teacher's expertise assures that content is current and taught in adequate depth. Expertise in a subject can provide a teacher with the flexibility to design and adapt material for the learner. The instructor can use this background to respond to a range of learners' needs.
Supporting claims:
-- An instructor has a good background in the course concepts, principles, and paradigms. (Feldman, 1988)-- Excellent teachers have a strong command and organization of their subject. (DuBois, 1993)
-- Competencies for good teachers include demonstrated mastery of the subject. (Smith and Simpson, 1995)
-- Good teachers are knowledgeable about their work areas and disciplines; they integrate current subject matter into their work. (McCabe and Jenrette, 1990)
A teaching assessment program must include some way to measure and rate a faculty member's expertise in the content area. Competence includes not only content knowledge but the ability to organize, integrate, adjust, and adapt this content in ways that make it accessible and thought-provoking to the learner.
The organization of course materials and activities has a profound effect on a student's possibility to succeed in a course. Effective organization goes beyond the orderly arrangement of content to involve the design of experiences that assist students in their learning. A number of course design principles that need attention.
A. Effective teachers have a clear purpose that organizes course elements.
A teacher needs to provide an organizing framework that orients students to the course's ideas, materials, and activities.
Some research that supports the need for planned organization:
-- An instructor prepares and organizes the course. (Feldman, 1988)-- Good courses are crafted. (CTL, 1997)
-- Excellent teachers spend considerable amount of time preparing course presentations. (DuBois, 1993)
-- Excellent teachers do their work in a well-prepared and well-organized manner. (McCabe and Jenrette, 1990)
The assessment program would search for evidence of careful planning. The quality of this planning would be one indicator of the likelihood of successful learning.
B. Effective teachers communicate high but realistic expectations.
The goals of a course must be challenging enough to motivate students, yet not so demanding as to overwhelm them. The teacher must identify what Lev Vygotsky called the "zone of proximal development." The research on college teaching recognizes this balance between high expectations and adaptation to students' ability level.
The studies claim:
-- The teacher sets and maintains high but realistic expectations & goals. (Angelo, 1993)-- Good practice communicates high expectations. (Chickering and Gamson, 1987)
-- Excellent teachers have the ability to motivate students to form goals and succeed academically. (DuBois, 1993)
-- Good teachers communicate and manage appropriate expectations for achievement in the course. (Smith and Simpson, 1995)
-- Excellent teachers project a positive attitude about students' ability to learn. They set challenging individual and collective performance goals for themselves. (McCabe and Jenrette, 1990)
-- Good teachers set challenging performance goals for students. (McCabe and Jenrette, 1990)
Realistic challenges are those deemed appropriate for students. Effective teachers are aware of their students.
-- The teacher shows a sensitivity to and concern with students' level and learning progress. (Feldman, 1988)-- The teacher selects course material suited to students' backgrounds, abilities, and interests. (Smith and Simpson, 1995)
This balance allows for intellectual and personal growth.
-- Students feel good courses are those where they learn a lot. (CTL, 1997)-- Build confidence in students by helping them to successfully meet learning objectives. (Smith and Simpson, 1995)
-- Students reach beyond course content and grow as human beings. (CTL, 1997)
-- Good teachers enhance student motivation by demonstrating the subject's relevance to their future needs and goals. (Smith and Simpson, 1995)
Accordingly, an assessment program must be designed to determine how well suited the course design is to students' abilities and interests. A well-constructed course (or set of activities) that is too simplistic or too challenging will not enhance students' success. It is necessary to gather and rate the appropriateness of the course's teaching materials.
C. Effective teachers match the instruction to students' learning needs and interests.
The design of a course must include deliberate connections between the subject matter and students' needs and interests, which engage them in the learning process. The various research statements below describe a number of points at which the course material should be designed to work with students' experiences.
Research on teaching suggests:
-- Good courses are those where students can connect new information to prior knowledge. (Angelo, 1993)-- Good courses are those that are relevant to students' needs and interests. (CTL, 1997)
-- Students become aware of their values, beliefs, preconceptions, and prior learning - and are willing to unlearn when necessary. (Angelo, 1993)
-- The teacher focuses student attention on what matters most. (Angelo, 1993)
-- Students seek and find real-world applications of what they are learning. (Angelo, 1993)
The teaching assessment system needs a method for determining to what degree a course provides opportunities for students to develop valuable personal connections to the course content. A straightforward organization of content may ignore the need to establish bridges to learners' experiences. Good courses should demonstrate deliberate strategies for determining student needs and interests regarding content. They should be designed to help students extract main points and they should incorporate activities that connect learning to applications.
Student learning is also affected by academic interactions between teachers and learners as they engage the subject matter. The various strategies and techniques that teachers use to implement the course design are critical to the learner's success. There are several key points that help teachers strengthen student learning.
A. Effective teachers use good communication skills.
It is necessary for teacher to communicate ideas clearly.
Research indicates:
-- The [effective] instructor is clear and understandable (Feldman, 1988)-- Excellent teachers communicate effectively in both written and oral form. (Smith and Simpson, 1995)
-- Effective teachers have the ability to present material interestingly and clearly. (Bernoff, 1992)
-- Excellent teachers present ideas clearly. (McCabe and Jenrette, 1995)
-- Excellent faculty are talented in clarifying difficult subject matter. (DuBois, 1993)
An effective teaching assessment process should determine that the ideas and directions presented in a class are clear and understandable. Miscommunication, confusion, or failure to convey key information leaves students at a distinct disadvantage. Good communicators go beyond clear information delivery to create environments that encourage comfortable, two-way communication between students and teacher.
B. Effective teachers design learning environments that encourage time on task.
Simply because time is spent in a classroom chair does not mean that time is being spent learning the subject. Successful classrooms spend time on tasks that aid learning. An effective teacher uses what is known about how people learn to design productive learning time.
Reports state:
-- Learning is enhanced when students invest as much time and high-quality effort as possible. (Angelo, 1993)-- Good practice emphasizes time on task. (Chickering and Gamson, 1987)
-- Good teachers manage the learning environment so that maximum learning will result. They lead class discussions that stimulate learning and enhance the goals of the course. (Smith and Simpson, 1995)
The assessment of effective teaching should be able to characterize the productiveness or lack of productiveness of a class period. There should be some way to determine how teaching and learning efforts are committed to the most important information.
C. Effective teachers engage students to actively use knowledge.
Learning is enhanced when students are engaged in active cognitive processes. Teachers need to design learning activities that require thoughtful information processing.
On this point, researchers say:
-- Students are (cognitively) actively engaged in their academic work. (Angelo, 1993)-- Good practice encourages active learning. (Chickering and Gamson, 1987)
--Excellent faculty encourage student participation and see themselves as student-centered teachers. (DuBois, 1993)
-- Good teachers promote students' individual involvement through learner-centered teaching methods. (Smith and Simpson, 1995)
-- Excellent teachers encourage independent thinking, analytic listening. (McCabe and Jenrette, 1990)
-- Good new teachers lecture in a relaxed style that provides opportunities for students' comprehension and involvement. (Boice, 1991)
Using course materials, an assessment program should be able to identify activities that involve students in active engagement with course material. Straightforward presentations of materials are not, in themselves, supportive of solid learning. Excellent teachers can describe specific ways in which their understanding of cognitive processes shapes the design of instructional activities to ensure that students encounter the subject thoroughly. An assessment process should be able to discriminate among teachers at either end of this continuum.
D. Effective teachers use an appropriate array of methods.
Because of the semester has its own rhythms and because students' needs change, teachers must be prepared to alter instructional methods to suit immediate goals. Methods must also be varied to accommodate students' different learning styles or developmental levels. However, it is obviously the aptness of methods that is important to learning, not simply the presence of different teaching techniques.
Researchers report the following:
-- Excellent teachers match varying teaching methods with specific instructional objectives. (Smith and Simpson, 1995).
-- Excellent teachers provide students with alternative ways of learning. (McCabe and Jenrette, 1990)
-- Good courses contain varied instructional methods. (CTL, 1997)
-- Students encounter a balance of intellectual challenge and academic social support. (Angelo, 1993)
-- Good teachers present material that is sequenced and paced appropriately for learners. (Smith and Simpson, 1995)
Teaching assessment procedures should indicate the degree to which the teacher makes deliberate use of varied techniques to engage students. The mere fact that there are varied techniques is not as promising as deliberate use of variation. It is important that a teacher respond to student needs and to the changing focus of the curriculum. The review process should develop methods to identify and acknowledge teachers' success in adapting teaching methods to meet student needs.
E. Effective teachers encourage students to work together to learn.
Peers are one of the most powerful learning aids. Effective teachers structure activities that use peer relationships to assist in the learning enterprise.
Research studies indicate:
-- Students work regularly and productively with other students. (Angelo, 1993)-- Good practice encourages cooperation among students. (Chickering and Gamson, 1987)
-- Excellent teachers provide cooperative learning opportunities for students. (McCabe and Jenrette, 1990)
-- Good teachers encourage cooperation and collaboration among students. (Smith and Simpson, 1995)
A complete teaching assessment program should thus be able to indicate how well a teacher creates conditions that enable students to work together productively, to improve their understanding of the material. A course should not be composed simply on teacher talk. Classroom activities that teach students how to work together effectively should yield high rates of learning success. Protocols for determining whether adequate levels of cooperation exist should be established within the assessment rubric.
F. Effective teachers give regular, helpful evaluations of learning.
To improve the quality of their work, students need continual, immediate, and helpful feedback. The teacher must incorporate an effective system for giving feedback to students, if they are to reach their potential.
The research states:
-- In good courses, the quality of examinations is good. (Feldman, 1988)-- Effective learning occurs when students are assessed and evaluated on what matters most (main points) and understand the criteria and standards against which they will be evaluated. (Angelo,1993)
-- Students become aware of their own ways of learning, so they can better monitor and direct their energies and efforts. (Angelo, 1993)
-- Good practice gives prompt feedback. (Chickering and Gamson, 1987)
-- Good courses have timely, informative feedback/evaluations of academic work. (CTL, 1997)
-- Good teachers provide helpful feedback to students in a variety of ways. They construct valid and reliable tests and administer other evaluation measures fairly. (Smith and Simpson, 1995).
-- Excellent teachers evaluate their students frequently and always let them know where they stand with regard to academic performance. (DuBois, 1993)
-- Excellent teachers give constructive feedback promptly to students. They are fair in their evaluations. (McCabe and Jenrette, 1990)
In order to assure that students have the information necessary to make progress in the subject matter, the teacher must have a thoughtful plan for assessing students' knowledge and abilities. In contrast to a generic letter grade, this evaluation system must provide specific information that both confirms knowledge gains and highlights the next steps for improvement. A teaching assessment system should be able to determine how well course evaluation methods contribute to enhancing the students' learning.
A special dimension of academic support integral to high student performance is the relationship between the teacher and the students. Teachers who can develop relationships that foster and encourage student engagement will enhance learning.
The researchers claim:
-- Excellent teachers are open, have respect for opinions of others, and encourage questions and discussions. (Bernoff, 1992)-- Excellent teachers never embarrassed or berated students (DuBois, 1993)
-- Excellent teachers create a climate that is conducive to learning. They treat individuals with respect. (McCabe and Jenrette, 1990)
-- Good new teachers have positive attitudes about students. (Boice, 1991)
A. Effective teachers and students are enthusiastic.
Clearly, motivation is a vital source of energy in learning activities. The effective teacher is energetic and enthusiastic and finds ways to create student interest and commitment to the learning task.
Reports indicate that:
-- The effective instructor shows an enthusiasm for the subject and for teaching. (Feldman, 1988).-- In good courses students and teachers are enthusiastic about learning. (CTL, 1997)
-- Effective teachers have enthusiasm about/interest in the subject matter, they are dynamic, energetic, and stimulate interest. (Bernoff, 1992)
-- Excellent teachers exhibit enthusiasm about their discipline and class presentations. They are able to convey a strong sense of presence in the classroom to elicit students' attention and stimulate their emotions. (DuBois, 1993)
-- Good teachers enhance students' motivation through personal enthusiasm for the subject. (Smith and Simpson, 1995)
-- Excellent teachers are enthusiastic about their work. (McCabe and Jenrette, 1990)
Classrooms with enthusiastic learners are not hyped-up circuses but places where dedicated, sincere efforts are being made to learn the subject. While strategies for creating this enthusiasm may vary, the degree to which a classroom generates this dedication should be evident and should be assessed as an important quality in promoting learning.
B. Effective teachers and students need to know and respect each other.
Students are unlikely to learn from people they do not know or respect. This does not mean that teachers are required to reveal their personal lives to students (or vice versa), but it does suggest that students learn best when they see how scholarship is connected to dimensions more personal than the mere repetition of content.
Researchers claim:
-- The relationships between students and teacher matter. (CTL, 1997) -- Learning is enhanced when students work regularly and productively with instructors. (Angelo, 1993) -- Good practice encourages student-faculty contact. (Chickering and Gamson, 1987) -- The instructor is available and helpful. (Feldman, 1988) -- Effective teachers are interested in, have concern and respect for students. (Bernoff, 1992) -- Excellent teachers have an approachable and friendly style with students ( DuBois, 1993) -- Excellent teachers are responsive to students' needs. (McCabe and Jenrette, 1990)
Teaching assessments should determine the degree to which a constructive relationship was established between students and teacher. At the very least, it is important that students feel welcomed to talk to the instructor. It is even better when students understand and respect the teacher because he or she models a commitment to scholarship (Smith and Simpson, 1995). Teachers' ability to connect with students constitutes a significant factor in learning success.
C. Effective teachers acknowledge and adjust to student differences.
Because students come from a variety of class, race, gender, ethnic and lifestyle backgrounds, because they manifest a variety of learning styles, and because they are often at various developmental learning stages, it is imperative that teachers be responsive to student differences.
Research demonstrates:
-- Good practice respects diverse talents and ways of learning. (Chickering and Gamson, 1987)-- Good teachers accommodate students' different learning styles by using a variety of teaching methods. They deal appropriately with issues that relate to various aspects of diversity. (Smith and Simpson, 1995).
-- Excellent teachers respect diverse talents. They regard students as individuals operating in a broader perspective beyond the classroom. They provide perspectives that include a respect for diverse views. (McCabe and Jenrette, 1990)
-- Excellent teachers understand [students' backgrounds]. (DuBois, 1993)
The degree to which the classroom provides ready access to multiple world views in its material, activities, and relationships represents an important marker of students (equal) opportunity to succeed. Teachers who address students' different needs for support are generally perceived as fair. Assessments of teaching should recognize teachers' efforts to assure that all students feel welcome and can be successful in learning the subject matter. The assessment should incorporate a rating of the degree to which they reach these goals.
D. Effective teachers are fair and impartial in dealings with students.
For students to succeed, they must trust that their efforts will be treated with the same respect as those of all other students. Perceived unfairness can destroy student interest and motivation.
The researchers state:
-- The teacher is impartial when grading students. (Feldman, 1988)-- The teacher shows overall fairness to students. (Feldman, 1988)
-- Effective teachers are fair. (Bernoff, 1992)
The quality of fairness is also mentioned in McCabe and Jenrette (1990) and Smith and Simpson (1995) as cited in section III F (above).
Students' perception that the instructor treats all equally is fundamental to the integrity of a class. An assessment program must therefore collect data that indicates how fairly the teacher treats students, if it is to assure this dimension of effective teaching.
E. Effective teachers are open to receiving feedback and adjusting courses appropriately.
When there is a clear and open channel for communication, teachers and students can modify and adjust to circumstances that might otherwise obstruct learning. Without such channels, confusion and misunderstanding may lead students and teachers to work in a too restrictive environment.
The reports indicate:
-- The teacher provides, receives, and makes use of regular, timely, specific feedback about course procedures (Angelo, 1993)-- Two-way communication is evident in good courses. (CTL, 1997)
-- Excellent teachers give consideration to feedback from students and others. They listen attentively to what students say. (McCabe and Jenrette, 1990)
-- Good teachers develop a reflective approach to teaching by collecting feedback and using it to continually modify the approach to teaching. (Smith and Simpson, 1995)
Therefore, teacher assessments should gauge how well teachers and students communicate on matters dealing with the classroom environment. The assessment process should be able to determine whether there are mechanisms in place to collect feedback and how well students and teachers use them to maximize learning.
Obviously, logistical and administrative components of a course must be handled efficiently and effectively. Even though the research on college teaching and learning largely overlooks course management, the smooth functioning of a course is an important factor in creating an environment conducive to learning. Course management encompasses a range of issues such as: regular meetings with classes, timely assignment of tasks and return of feedback, etc. The way in which the classroom is organized as an environment determines how effectively it will support learning.
The reports do suggest:
-- Good teachers manage the process of planning, teaching, and evaluating in a timely manner. They deal appropriately with matters of discipline, academic honesty, and legal information. (Smith and Simpson, 1995)
Nearly all the researchers recognize the importance of being available to help students (McCabe and Jenrette, 1990; Bernoff, 1992; DuBois, 1993; Smith and Simpson, 1995).
No single arrangement of teaching and learning components will be successful in all circumstances. Rather, good teaching requires that educators have a rich understanding of pedagogical theory and practice and are able to engage in the on-going development and adaptation of their teaching approach. Stephen Brookfield (1995) describes the importance of critical reflection to college teaching. Mary Ellen Weimer (1990) and Anthony Grasha (1996) have detailed the processes that help faculty improve their teaching practices. Moreover, Raoul Arreola (1995, pp. 78-79) has documented how faculty evaluation scores improve when they participate in faculty development initiatives. Faculty members who hold themselves to high professional standards in their role as teacher exhibit qualities which attest to the dynamic nature of effective teaching.
Researchers claim:
-- Excellent teachers recognize and accept teaching as a fundamental and challenging dimension of scholarship. They use research in teaching as it applies to instruction in their field. They use a reflective process. (Smith and Simpson, 1995)--Excellent teachers have a strong sense of commitment and dedication to college teaching. (DuBois, 1993)
-- Excellent teachers are committed to education as a profession. They display behavior consistent with professional ethics. They work collaboratively with colleagues. They provide clear and substantial evidence that students have learned. (McCabe and Jenrette, 1990)
-- Good new teachers show evidence of actively seeking advice about teaching from a colleague. They demonstrate a great readiness to become involved in campus faculty development programs (Boice, 1991)
-- Excellent teachers have a distinct identity as a teacher, have been inspired by past teachers, and need their students as much as students need them. (DuBois, 1993)
It is reasonable to expect that faculty be able to document their scholarly efforts to enhance pedagogical knowledge and to develop practices that enrich their teaching and learning environments. Portfolios (Seldin, 1997) or other reports can be used to demonstrate such scholarly efforts in the area of good college teaching. An assessment program should incorporate some tool to encourage professional development, to measure progress, and to reflect areas for further growth.
Conclusion
Researchers argue that there are multiple components involved in effective college teaching. The major research has been summarized in six categories to better focus our efforts to collect and rate performance levels. Figure 1 presents the six categories and the components that make up each. These categories form one of the underlying structures from which a sufficiently complex evaluation system can emerge.
The Components of Effective College Teaching
|
I. Teacher's Content Expertise
|
|
II. Course Design
|
|
III. Instructional Delivery
|
|
IV. Instructional Relationships
|
|
V. Course Management
|
|
VI. Professional Development
|
|
|
Using Multiple Sources for the Assessment and Improvement of Teaching
True objectivity in a faculty evaluation system is an impossible goal; in a process that, by necessity, relies on the subjective judgement of peers, administrators, students, and self-evaluation to gather its data, subjectivity is unavoidable. Informed, personal assessment is indeed required to examine the complex human interactions of education. Yet the majority of faculty must believe that the system used is fair and impartial. Unfairness arises when evaluations are based on biased, capricious, hear-say comments about teaching. Thus, the goal of the evaluation system should be to reduce and control the negative aspects of subjectivity. By using multiple sources in appropriate ways, subjectivity can be controlled, and a fairer and more equitable system can be developed.
Current practices for evaluation of teaching undoubtedly vary from department to department on our campus. However, in all cases, students, peers, and administrators are the most common sources of information for assessing teaching, whether data are systematically gathered or randomly acquired. Depending on the situation, the credibility of these sources varies. The credibility of a source depends on the opportunity that the source has (had) to be a firsthand observer of the teaching component in question, as well as on the willingness of the person being evaluated to accept and believe what that source has to say (Arreola, 1995).
As stated earlier in the Guiding Principles, teaching evaluations should include multiple sources of data including students, peers, administrators, and self-evaluation. This type of evaluation must provide valid, summative information regarding the quality of teaching and formative direction for faculty improvement efforts. Finally, faculty are entitled to a fair and consistent evaluation of their teaching. With these principles in mind, this section will discuss issues related to the various sources and their roles in the assessment and improvement of teaching.
Students play an essential role in the evaluation of teaching. They are, after all, the group most directly affected by the quality of teaching. As firsthand observers in classes, students are in the best position to evaluate specific and critical aspects of classroom teaching. Students are the only individuals who regularly observe us in action, and it is for their benefit that we teach. Murray (1995, p. 50) states, "This symbiotic relationship between professors and students means that it is not only in our best interests to respect what they can tell us about our teaching, but also in their best interests to assist us to improve our teaching."
Techniques for formative evaluation can range from very formal to more informal. At the formal end of the spectrum are instruments that look very similar to student evaluation instruments used for summative purposes at most U.S. institutions of higher learning. Informal techniques using such strategies as student journals, open-ended questions, writing an unsigned letter to a fictitious friend who is thinking of taking your class, etc. can be valuable formative tools (Murray, 1995). Many of the formative evaluation techniques should be implemented after students have had enough time to become familiar with a teacher's style, but no later than midterm to allow for mid-semester modifications. Several studies (McKeachie et al., 1980; Overall and Marsh, 1979; Stevens and Aleamoni, 1985) have reported that midterm student feedback helps faculty improve teaching. Brinko (1993) suggests that this is particularly true if such feedback is combined with advice from a faculty development specialist. By providing this type of early formative evaluation, the same students who provide the feedback will gain some of the benefits from any course modifications the faculty member makes.
Formal student evaluations given at the end of the semester are typically used for summative purposes, but can also be used in formative ways as well. Student ratings are one of the most common features of faculty evaluation systems. Seldin (1993) reports that more than 85% of all faculty evaluation systems make regular use of student ratings. Unfortunately, these ratings are often the only component used to evaluate teaching. Given that this information is used for personnel decisions that are not always positive for the faculty member, it is not surprising that student rating forms are not met with universal acclaim by faculty! This misuse of limited perspectives has led to many myths and misconceptions about student ratings.
It is critical that the instrument used be both reliable and valid, and provide meaningful feedback that can be used for making improvement. Because locally developed rating forms may not possess the necessary psychometric qualities of reliability and validity, it is generally a good idea to adopt an existing form rather than developing one from scratch (Arreola, 1995). Many believe that student rating forms are unreliable and invalid. Since many forms are "home-made," this perception is often reality. However, well-developed, professional student rating forms have been shown to be valid and reliable (Aleamoni, 1978; Aleamoni and Hexner, 1980; Costin, Greenough, and Menges, 1971; Marsh, 1984; Burdsal and Bardo, 1986.)
According to Arreola (1995), there have been more than 70 years of research on student ratings, and it continues to be one of the most heavily studied topics in faculty development literature. Space does not allow for a complete review of this research, however Cashin (1995) and Arreola (1995) both provide excellent summaries of the research literature that examines variables related to student ratings. In brief, they report:
Variables not consistently related to student ratings include:
Variables consistently related to student ratings:
Given these types of variables, it is critical to use a well-constructed student evaluation capable of taking these variables into account. In order to reduce subjectivity and to make this process fair and equitable, it is strongly recommended that institutional procedures for administering the end of course survey be adopted and carefully followed by all faculty members. The Course/Instructor Evaluation Questionnaire (CIEQ) and the Instructional Development and Effectiveness Assessment System (IDEA) instruments have been shown to be reliable, include national norms, and provide flexibility to faculty in the type of questions included. Self-developed instruments often lack some or all of these qualities (Arreola, 1995, p. 63).
As major participants in the educational process, students represent a valuable source of information for teaching assessment and improvement efforts. They are firsthand observers of course design, instructional delivery, class management, and teacher-student relationships -- key elements of effective teaching. Well-designed student rating forms can provide valid information concerning these critical components of teaching. A wide variety of other formative assessment techniques can also provide important feedback as well. When a teaching assessment system is implemented, it is essential that students understand how the process works and how the information they provide will be used.
Peer evaluation typically takes the form of classroom observations, review of instructional materials, and critiques of course design. These types of review have traditionally been used for both formative and summative assessment. Using colleagues as a source of assessment has both advantages and disadvantages. It is certainly advantageous that colleagues, who are usually from the same discipline, are familiar with departmental goals, programs, and priorities. However, such familiarity can also have negative consequences resulting from bias due to previous evaluations, personal relationships, and peer pressure (Arreola, 1995).
While commonly used, classroom visitations are not recommended for summative assessment unless certain guidelines are followed that result in satisfactory inter-rater reliability. The peer should be a part of an observation team whose members have been trained in the use of a standardized observation tool. Each peer must make multiple observations produce adequate inter-rater reliability. Centra (1975) reported that visitation ratings had less inter-rater reliability and a greater bias toward leniency than student ratings. He concludes that colleague visitation ratings are not typically reliable enough to use for personnel decisions. He suggests that reliability of raters can be improved by increasing the number of observers, increasing the number of visits per observer, and by providing special training in classroom observation technique. Each of these factors, however, can be prohibitively expensive in both faculty time and effort.
Other research findings have supported Centra's work. Braskamp, Brandenbury, and Ory (1984) reported that almost all peers rated their colleagues as excellent or good instructors. Furthermore, they reported that the relationship between observed instructor behavior and student learning was weak. Findings like these make the validity of peer ratings highly debatable, and researchers suggest that colleague visitation is more appropriate for faculty development activities than for summative evaluation. Finally, it has been suggested that colleagues can provide valuable, reliable, and valid assessments of course materials such as examinations, syllabi, course assignments, and instructional media (Kulik and McKeachie, 1975).
Based on a review of literature, Cohen and McKeachie (1980) concluded that peers can reliably evaluate:
Departments will need to select which areas (from the list above) peers should evaluate. Further, decisions on appropriate methods will need to be made. Recommendations about methods for effective peer review are included later in this document. Thus, it is suggested that peer evaluation can be used effectively in the assessment of content expertise of the instructor and course design. Instructional delivery should be assessed by peers for formative assessment linked to professional development activities, but not for summative evaluations used for personnel decisions.
It should be noted that peer evaluations may also involve professionals from other universities, industry, government or private agencies. While this method may need to be used selectively because of the costs and logistical demands involved, they provide an way to assure that the university's internal reviews are in line with professional standards outside the department.
According to the American Association for Higher Education (AAHE) Joint Task Force on Student Learning, the "whole college" of teachers, administrators, students, and career professionals must collaborate on the assessment and improvement of teaching and learning. Their draft position paper states that "It will take the whole college, and those affiliated with our institutions as members of the larger community, to ensure that we achieve our mission of increased higher learning" (1998).
Whetten and Whetten (1985, p. 35) declared that "...the most powerful predictor of organizational effectiveness in colleges and universities is administrative behavior. Results from research show that administrators are more important than environment, structure, institution type, and control in accounting for performance." However, the AAHE (1998) position paper contends that "Administrators frequently fail to focus resources on those functions at the college and university that are most effective in producing learning. Indeed, they may not even know what those functions are, having little information about results on the basis of which to plan and make decisions."
Thus, administrators must play a key role in creating and nurturing a "culture of teaching and learning" (Hutchings, 1996). There must be a long-term commitment on the part of upper administration to plan, support, and properly implement any proposed assessment plan if it is to be effective. The President, Provost, Deans, and Chairs all have essential roles in the assessment of teaching and learning.
The Role of the President and Provost
As Whetten and Whetten (1985) point out, administrative behavior is likely to result in improvement of teaching and learning if that behavior creates a culture that values teaching and learning. The founding and continued support of the Center for Teaching and Learning, as well as the Lilly Project are steps toward establishing such a culture at ISU. The continued role of the President and Provost in providing leadership for faculty evaluation of teaching should include enforcement of standards and criteria used by deans and chairs, as well as the review of deans and chairs with regard to their appropriate use of both developmental (formative) evaluation and personnel decisions (summative evaluations).
The Role of the Dean
The dean should ensure that faculty development and faculty evaluation are carried out in a fair, impartial, and systematic way at the departmental and school/college level within guidelines endorsed generally by the AAUP policy on academic freedom (ISU Handbook 3-1) and specifically by the AAUP's statement on teaching evaluation (1990). Because the dean is seldom a "firsthand observer" of teaching, s/he is not directly involved with the assessment of teaching. S/he ensures that teaching assessments follow the stated school mission and evaluation criteria and that evaluations follow explicit procedures. The dean should encourage a culture within the respective school/college that values teaching and learning.
The Role of the Chair
If an organization's effectiveness largely depends on administrative behavior, it follows that chairs, if for no other reason than sheer numbers, must play a crucial role in the assessment of teaching and learning (Gmelch and Miskin, 1993). It is estimated that chairs outnumber all other administrators by five to one (Murray, 1995). Since most of the choices within higher education occur at the departmental level, it is reasonable to ascertain that chairs are key players in evaluating and providing formative direction to faculty.
A primary role of the chair is to oversee departmental procedures for both summative and formative evaluation. The chair must work to build trust and integrity into these ongoing processes. As a source for evaluating teaching, the chair shares many of the same advantages and disadvantages of peers in the department. Generally, it is logistically impossible for chairs to visit classes of all faculty members frequently enough to achieve reliability. As with colleague visitations, occasional observations by the chair can be biased by personal relationships, common values, favored teaching methods, etc. Class visitations for professional development reasons may be appropriate, but they are problematic for summative purposes. In most cases, chairs should be able to adequately evaluate course design and content expertise. The chair, moreover, is in the best position to evaluate the teaching component of course management.
The principal characteristics for chairs who oversee teaching evaluations are impartiality, vigilant protection of faculty rights, and active support of teaching improvement. The chair should work with departmental personnel committees to ensure proper procedures are being followed in summative evaluations. Providing leadership and direction in this process is critical. The chair should try to facilitate the work of the committee. While the chair can provide data input into these evaluations, s/he should be limited to that which is observed "firsthand," The most important role a chair can play is providing leadership for collegial professional development activities within the department. This can prove to be a very challenging task if faculty perceive all evaluation activities to be linked with personnel decisions. Ideally, both summative and formative evaluations should facilitate professional development activities.
Self-evaluation of instruction is based on subjective evaluation of classroom performance by the individual professor. Self-evaluation that results in improvement in the classroom requires both reflections (self assessment) and input from other sources. There is a fine line between self-assessment and reflective practice. Self-assessment usually indicates that one is applying some measure to one's own work and passing judgment. However, engaging in self-assessment does not necessarily imply that one has fully reflected on one's work. Neither does reflective practice necessarily imply that one will perform an assessment.
College professors are primarily motivated by the intrinsic rewards of academic work, including intrinsic rewards gained from teaching...(they) want to feel competent and have a sense of self-determination...The need for self-determined competence also prompts individuals to scan the environment for feedback that informs them of the results of their performance -- producing a critical psychological state that an intrinsically motivated person continually seeks through work. (Paulsen and Feldman, 1995)
This continual feedback is what most faculty consider as a first (and sometimes only) step in self-evaluation. Without deeper reflection, it tends to confirm the faculty member's theories of instruction, which are often anecdotal and faulty (Paulsen and Feldman, 1995). This somewhat sustained reflection requires more structure, such as record keeping, and may lead to systematic change. There are a variety of techniques that are recommended for complete self-evaluation -- videotapes, journals, teaching portfolios (which can include such things as tapes and journals, collegial evaluations to name a few -- that are fully documented in the literature (Murray, 1995; Travis, 1995).
The strength of self-evaluation is that faculty are most likely to act on data collected by themselves. The various means of gathering information to assess performance relative to personal needs, goals, and objectives may be used as an ongoing program of continuous formative assessment. The weakness of the process is that most faculty tend to rate themselves higher than students do (Arreola, 1995). Centra (1993) reported that several studies have found low correlations of self-evaluations with those collected from other sources. Despite this research, Centra contended that self-ratings could be very useful for instructional improvement efforts and that self and student ratings tended to identify the same relative strengths and weaknesses. After a thorough review of literature, Brinko (1993, p. 577) pointed out that "feedback is more effective when information is gathered from oneself as well as from others."
Arreola (1995) identifies several conditions for effective use of self-evaluation. First, instructors need to understand precisely how the information is going to be used in order to feel confident and secure in the process. The exact form and substance of the self-evaluation would likely differ for faculty at different stages of their career. Faculty need to have skills in identifying goals and collecting appropriate data. Chairs should help facilitate this process, and the CTL should be a valuable resource in providing this type of training.
Effective self-evaluation requires self-reflection and self-assessment. Further, it should be used in conjunction with administrator/colleague observations and student feedback to determine if the instructor's self-perceptions of classroom performance are consistent with more objective observers. Differences in perceptions can be used to target weaknesses and strengths in teaching, the recognition of which is integral to the improvement of instructional effectiveness. In the best instances, faculty will reflect thoughtfully upon their performance and their students' outcomes, compare this self-reflection with more objective perceptions of colleagues and students, and modify their teaching as needed. A teacher should continually consider both the quality of the knowledge, values, and skills being conveyed as well as the pedagogical background and skill they bring to the delivery of this content.
Summary
Because teaching is a complex, multifaceted endeavor, it must be assessed from many viewpoints. An evaluation system that fails to gather data from qualified, first-hand sources would necessarily be compromised. Likewise, a system that restricts its view to only those components fairly measured by one data source would neglect vital information on teaching effectiveness. It is necessary to choose from multiple sources in order to receive the most appropriate data on each of the components of quality college teaching. Doing so eliminates obvious mismatches (where students would interpret a teacher's content expertise, for example), reduces the use of methods that are complicated, inefficient, or potentially unfair (such as summative use of classroom observations by peers), and emphasizes reliable procedures that can help reduce the arbitrary and confusing dimensions of teacher evaluation.
The summary of data sources outlined above describes the sources essential to a comprehensive evaluation system, while outlining some of the qualifications that are needed to keep the system consistent and fair. Clearly, different teaching conditions require some flexibility in selecting what combination of sources and what methods are most suited to particular disciplines or even courses. However, the general parameters described above offer a sound framework for assuring a flexible but consistent evaluation system for the campus. Figure 2 illustrates how these components can be viewed in reference to the multiple dimensions of effective teaching, allowing a complex system for measuring the various components of effective teaching appropriately.
Assessing the Components of Effective College Teaching
COMPONENTS / SOURCES Students Peers Admin Self
|
I. Teacher's Content Expertise
|
|
II. Course Design
|
|
III. Instructional Delivery
|
|
IV. Instructional Relationships
|
|
V. Course Management
|
|
VI. Professional Development
|
|
|
Introduction
An evaluation system must acknowledge the six components of effective college teaching described in part 2 as well as the various sources from which suitable data can be gathered summarized in part 3. The challenge is to develop a way in which this array of information can be brought together in a report that permits comparisons to specific standards. We recommend the following assessment process in order to combine the teaching components and multiple data sources described in the previous two sections. The process allows departments a range of flexibility in determining the value weighting of components and sources. At the same time, it asks departments to take some steps that permit consistent assessment reports for all its faculty. In general, this process involves four steps:
1. Discussing the academic unit's relative valuing of components that the pedagogical literature associates with effective teaching
2. Distributing these choices on a grid that incorporates the possible assessment sources
3. Deciding how much weight to give to each assessment source, in relation to a given component.
4. Determining what instruments should be applied.
These phases are explained in more detail in the remainder of this section.
Departments / units discuss their professional approaches and traditional obligations in the six areas:
1) the content knowledge teachers need to carry out their various assignments; 2) course designs appropriate to disciplinary integrity; 3) various approaches to delivery, from among the best practices in the field; 4) the different manners and degrees of accommodation by which teachers relate to students; 5) reasonable expectations for a well-managed course; and 6) professional teaching development
Although some of these attributes may be recorded in departmental by-laws and informal traditions, it is crucial for departments to review the full scope of faculty members' practices in tutorials, classrooms, labs, lecture halls, seminars, and anywhere else that directed learning occurs. CTL staff could help facilitate departmental exchanges, so that colleagues in a unit would debate and decide for themselves, within their discipline, the relative significance of the six components in their discipline.
The participants must distribute their agreed-upon component interpretations into an array of appropriately matched sources. This relationship has been called an Assessment Matrix (Arreola, 1995). A blank grid is shown in Figure 3.
Each row of the grid represents a component of effective instruction. Each column of the grid shows an assessment source. The formatted matrix displays cells for reporting standardized scores determined by interpreted data that have been collected about that teaching component. Since not all sources are appropriate for assessing all of the teaching components, blanks will appear in the final grid. Also, the deliberative process allows departments, as well as individuals (if the unit so decides), to weight the value of cells according to their own commitments. However, no unit should be able to ignore any of the six components nor shall their weights fall outside the range adopted by the university.
|
Components |
|
|
|
|
|
|
I. |
Content Knowledge |
||||
|
II. |
Course Design |
||||
|
III. |
Instructional Delivery |
||||
|
IV. |
Instructional Relationships |
||||
|
V. |
Course Management |
||||
|
VI. |
Professional Development |
Phase 3
Deciding Source Weights
The matrix also allows each teaching component to be rated according to some or all of the data sources. Each source may be weighted differently to reflect the degree to which that data source should be utilized in the evaluation of that teaching component. Figure 5 provides a list of recommended weight ranges for each source.
|
Components |
|
|
|
|
|
Content Knowledge |
||||
|
Course Design |
||||
|
Instructional Delivery |
||||
|
Instructional Relationships |
||||
|
Course Management |
||||
|
Professional Development |
||||
|
TOTAL WEIGHTS |
|
|
|
|
With these two parameters set by general debate within the unit, it is possible too use the matrix to give a value for each cell. The percentage that each source contributes to the evaluation of the teaching component appears in the small boxes and add up to 100% for that row. In the sample provided in Figure 6 the crossed-out cells represent sources that are not deemed appropriate in evaluating that teaching component.
The total value allocated to the sources of evaluation can be seen by multiplying the percentages and summing down the columns. For example, the student source accounts for 35% of the evaluation [.15(.0) + .20(.20) + .20(.70) + .20(.85) + .05(0) + .20(0) = .35]. With these weights, the sources have the percentage impact listed in Figure 6. These final weights should fall within the range indicated in Figure 5.
SAMPLE APPLICATION OF SOURCE WEIGHTS
|
Components |
|
|
|
|
|
||||||||||||||||
|
Content Knowledge |
|
70% |
|
|
|
||||||||||||||||
|
Course Design |
|
|
|
|
|
||||||||||||||||
|
Instructional Delivery |
|
|
|
|
|
||||||||||||||||
|
Instructional Relationships |
|
|
|
|
|
||||||||||||||||
|
Course Management |
|
|
|
|
|
||||||||||||||||
|
Professional Development |
|
|
|
|
|
||||||||||||||||
|
Total Source Weights |
|
|
|
|
|
Figure 6: The Assessment Matrix with Source Component Weights
Adding Scores to the Matrix
With values adjusted according to the individual's and the unit's commitments, the matrix can serve as a tool for developing an overall measure of a faculty member's teaching performance. An evaluation process can place a rating in each cell to represent an interpretation of how well a teacher has performed in this component, according to that source. When the ratings in each cell are based on the same scale they can be multiplied by the cell's value factor to produce an overall score.
Consider, for example, that each cell is rated using a 6-point system. In this scale, a score of 1 or 2 would reflect below average performance; 3 or 4 would represent average performance; and 5 or 6 would indicate above average performance. Student responses on the standard instrument will yield a set of scores where predetermined thresholds would translate student rankings into this scale. Peers, administrators, and self assessments would be established to translate the data gathered into this rating system. After a score from 1 to 6 is determined for each cell, the score is multiplied by the cell percentage and by the row percentage. These numbers are then summed to give an overall score for teaching. As an example, consider the professor whose scores were as recorded in
Figure 7.
|
Components |
|
|
|
|
|
|
||||||||||||||||
|
Content Knowledge |
|
70% |
|
|
|
|
||||||||||||||||
|
Course Design |
|
|
|
|
|
|
||||||||||||||||
|
Instructional Delivery |
|
|
|
|
|
|
||||||||||||||||
|
Instructional Relationships |
|
|
|
|
|
|
||||||||||||||||
|
Course Management |
|
|
|
|
|
|
||||||||||||||||
|
Professional Development |
|
|
|
|
|
|
||||||||||||||||
|
OVERALL SCORE |
|
|
|
|
|
|
The results for this professor would show the component scores:
Content: .70(6) + .30(5) = 5.7
Course Design: .20(5) + .40(3) + .30(4) + .10(5) = 3.9
Instructional Delivery .70(6) + .10(5) + .10(5) + .10(5) = 5.7
Instructional Relationships .85(5) + .15(5) = 5.0
Course Management: 1.00(5) = 5.0
Professional Development .55(4) + .25(3) + .20(4) = 3.75
These scores indicate that this professor's strengths are in his/her content and delivery, while relationships with students and professional development activities would be the areas that might be the focus for improvement.
The overall score would be found by multiplying the score for each component by the percentage assigned to that component:
.15(5.7) + .20(3.99) + .20(5.7) + .20(5) + .05(5) + .20(3.75) = 4.775
This score could be used to compare with 1) his/her overall ratings in previous years, 2) pre-established thresholds, or 3) ratings received by other faculty.
The use of a numerical scoring system balanced within the assessment matrix helps address several guiding principles: it is comprehensive; it is systematic; and it provides for flexibility. Academic units will retain the ability to interpret and evaluate instruction in their areas, and individual faculty will be able to submit for evaluation whatever materials they and their peers deem appropriate.
Assessment Instruments
A number of things remain unaddressed by this report: what assessment instruments would be used by each of these sources; what an acceptable assessment protocol in the use of these instruments might be; and what reasonable assessment rubrics are appropriate. These issues are critical to the establishment of an effective assessment program. They must be considered within the research on the best practices. Our recommendation is that the university, through the Center for Teaching and Learning, provide an outline of best practices found in the scholarship, and that departments engage in careful dialogue about the instruments, protocols, and rubrics they will adopt. In accordance with our guiding principles, we also recommend that information from students be derived from a university-wide course evaluation instrument, one that would be recommended by a committee and adopted by the university (see implementation section).
Conclusion
The process described here aims to accumulate data from appropriate sources, convert interpretations to a common scale, and weigh factors according to clear value commitments. It calls on academic units to work through a complex series of steps to adapt the assessment matrix, so that it is suited to their unique teaching conditions and requirements. Following the described procedures to place numeric values in the assessment matrix, departments can develop an invaluable tool for mapping their teaching quality in reference to the university. The effect of this system on the individual is outlined in the following section.
Potential Applications of the Assessment Process
Because teaching evaluation should be both formative (assist the teacher in making improvements) and summative (assist others in making judgments at professional decision points), the assessment matrix should provide a framework that facilitates both. The results of the assessment process should provide a gauge of each faculty member's overall efforts in reference to others, while permitting a range which accommodates individual commitment. The aggregate results should also inform the design of faculty development, by providing direction and assistance for personal improvement in terms of those factors that matter most in teaching evaluations.
Formative Uses of the Assessment Process
The principal value of a faculty evaluation system to both the faculty and the institution depends on how well it is integrated with a professional development program (Arreola, 1995: 78). When evaluation and development are organized around the same guiding principles, they become mutually enhancing. Research suggests that the most efficient route to improvement is to establish a process in which data from the evaluation process guides choices about professional development (Aleamoni, 1987; Theall and Franklin, 1991).
The development and coordination of a reputable series of support services for faculty (Murray, 1995:61; Travis, 1997:11) should include:
CTL Workshops. The Center for Teaching and Learning can provide regular workshops and institutes around each of the components. These opportunities can provide an information, ideas, and practice regarding the most current knowledge and skills in these areas. Faculty can attend to refresh and enhance their abilities.
Faculty Orientation. The CTL can use this same format to introduce new faculty (including Graduate Teaching Assistants and Adjunct Faculty) to these components of teaching effectiveness, assuring that they understand the framework for the university's and the department's expectations regarding teaching (Boice, 1992).
Department Assistance. The department is the critical site for formative faculty development. The assessment process can assist chairs and faculty review committees as they work with each member of the department in establishing targeted professional development projects (Gmelch and Miskin, 1993:29; Lucas, 1994; Higgerson, 1996). The CTL can also act as a resource on the best practices for observing, reviewing, and interpreting pedagogical data collection
Consulting. One of the most widely endorsed approaches to faculty development acknowledges that individuals proceed through a series of steps that are best initiated by reliable data gathering and thoughtful analysis of strengths and weaknesses (Weimer, 1990; Theall and Franklin, 1991:53-96; Arreola, 1995:110; Travis, 1997). These steps guide faculty through a process of reflection, selection, and experimentation that can be documented in a variety of formats, the most popular of which is the teaching portfolio (Murray, 1995; Seldin, 1997).
Faculty Research. Classroom research (Angelo and Cross, 1993) or other research projects may provide significant areas for faculty improvement and enhancement of their pedagogy.
Because the assessment process brings together an array of information, it is a valuable tool for identifying discrepancies and points of concern. Yet, it has been long established in the research that "in the absence of intervention, faculty members show little change in the ratings they receive from students about their teaching effectiveness from one time to another" (Wilson, 1986). It is essential that mechanisms be created to provide recommendations about directions for improvement. Formative discussions should be designed that use this framework to target specific areas for personal development -- areas targeted to enhance the results of future summative reports. On-going programs in each of the teaching component areas can readily provide information and techniques that can enhance faculty members' chance for re-appointment (as in the case of the wise professor Hotspur), reward their efforts within the tenure system (the mid-career case of Henryson), and increase their satisfaction as senior members of the university (witness, the case of the venerable Oldcastle).
Dedicated teachers want to know whether they are providing the best instruction possible. Developing a formative evaluation system assists in gathering, interpreting, and acting on this information in a systematic way. It provides the level of support for such teachers that can help them become highly effective in their teaching.
Summative Uses of the Assessment Process
Teaching has long been considered a major factor in summative faculty evaluation at Indiana State University. The assessment matrix will enable this evaluation to be more consistent than it currently is in practice. All faculty will be working within the same protocol. At the same time, the process will permit departments and faculty to emphasize values that reflect their particular commitments.
Assistant Professor Hotspur would receive a description of the six areas for teaching evaluation when she attended New Faculty Orientation. (Even before that point, during final interviews her dean would have explained the process and introduced the data-collecting instrument.) Hotspur's first semester of teaching would not be assessed summatively, but rather, the results would form a set of baseline scores. She would discover how well the instrument fit her teaching style and pedagogical planning, and her general results would indicate how she was doing in ISU's classrooms relative to her colleagues' achievements. If the data suggested problems of adjustment, she would know whom to consult; if her previous teaching was adapting well, she would sense what to enhance or where to concentrate in the other areas, scholarship and service. In successive years of contract renewal, an array of scores and commentary would accumulate until the summative judgment of tenure was made. Not all the formative information Hotspur gathered would be forwarded, only the most significant scores and necessary contextual explanations. Each year's results would provide a score which, placed on a chart with prior years' ratings, would yield a profile that would offer a valuable insight into the trajectory of her teaching ability. The cumulative efforts at improvement and the ability to sustain high performance levels should lead to confident recognition of her teaching performance.
Associate Professor Henryson would teach his new semester's courses, having reflected on several teaching assessments and collegial evaluations from the recent past. He would add to his teaching portfolio some focused reflective writing on how he was adjusting his design, delivery, instructional relationships, management. His opinions on what constituted high, middle, or sufficient success in these areas would have been discussed in his department and agreed upon as guidelines for the personnel committee. Threshold scores may have been set or a rank ordering system put in place that allows the overall score to determine overall evaluation of teaching performance. In December, he and everyone else would submit documents for review. By early spring, he would receive a score that would be comparable within and among departments. His performance adjustments would be based on more substantial information than guesswork, and he would be assured that appropriate sources were weighed--the negative effects of administrative whim or collegial back-stabbing would worry him less than it used to.
Full Professor Oldcastle, no longer worried about tenure and promotion, has more reliable feedback on her traditional teaching than she received formerly. The matrix of scores over time allows her to see trends in the match between her style and the learning capabilities of recent students. Attendance at an occasional workshop or faculty development series has increased her confidence that minor changes in her lecture methods seem to yield considerable improvements in student learning. She's still skeptical of newfangled pedagogy but has to admit that her teaching is more stimulating. With an organized and consistent set of information, she is more confident that teaching evaluations have moved away from an "old boy network" or a blame-the-student attitude and gained scholarly respectability.
Gathering reliable information about the quality of instruction found at the university is critical to its ability to create and effective learning community. A comprehensive evaluation system that allows for fair comparisons allows faculty, students, and administrators the opportunity to make decisions that shape the institution into a quality educational institution. When tied to the formative process, the university has in place opportunities to respond quickly and efficiently.
Although it is premature to guess how these proposals would fare during the process of faculty and administrative review, the following six-step process would help to systematically introduce the essential values of a consistent and fair program to assess and improve teaching. Adjustments naturally occur when innovations meet old practices. However, turf protection and narrow self-interest should not kill the spirit of these changes, because the ISU teaching community must realize that even a complex system is preferable to a loose, disorganized one, when we consider the gains in student learning and faculty comprehension. A perhaps greater threat to the implementation of these suggestions is divided attention. National accreditation may draw energy away from improving teaching evaluation, as may the perception that this effort is just one more administrative initiative. Faculty must participate in the debate and re-structuring, and a period of wrangling and adjustment must be anticipated. The best assurance that the University's leadership cares about better evaluation of teaching is to keep asking how the process is proceeding, to share in the debate, to participate in any training, and to remember constantly that formative concerns about our lives as faculty members are likely to move us to action sooner than perceived threats about summative judgments.
Phase 1 - Fall, 1998 and Spring, 1999
Campus discussions of this Task Force's report should begin. The conceptual framework and assessment procedures should be shared with Deans' Council, Faculty Senate, Chairs, faculty focus groups, etc. Self-selected departments from several disciplines should be chosen to pilot this new system. These departments should understand they will need to accelerate the discussions and decisions included in Phase 3 in order to pilot the system. Departments willing to participate in this pilot should receive compensation in the form of extra travel or equipment one-time monies. They should be expected to share the results of their efforts with others on campus.
Phase 2 - Late Spring and Summer, 1999
A committee should be formed to make final recommendations concerning the choice of a student evaluation form. This Task Force favors the IDEA or CIEQ mentioned in the "Role of Students" section of "Using Multiple Sources" chapter of this document. Another possible alternative is a locally developed instrument. This choice would entail a large commitment on the part of the university in staff hours and expertise in the development of such an instrument. The use of consultants should be considered in the study of this issue. Summer stipends should be paid to faculty members working on this project.
The CTL should begin to specifically design workshops to address the role of those teaching components identified in the model. An overview of the model should be presented at new faculty orientation. These workshops should be scheduled throughout the year and be required of all new faculty and academic administrators. All faculty should be encouraged to attend. The required workshops for new personnel should continue each year.
Phase 3 - Fall, 1999 and Spring, 2000
Departments should begin the process of discussing what percentage weightings they would assign within the parameters suggested in this document. Tools for colleague, administrative, and peer evaluation should be chosen by each department. Guest speakers should also be brought to campus to provide insight to these issues. Final decisions should be made the end of the academic year. Departments should work closely with the CTL to facilitate these discussions. A committee should be formed to review and respond to these plans over the summer. Faculty involved with this work should be paid summer stipends.
A final decision on a student evaluation form should have been made at this point.
Phase 4 - Spring, 2000
Departmental representatives with the greatest curiosity to improve teaching evaluation should be paid to attend workshops on the process. During the year, faculty experts should be brought to campus to consult with these departmental experimenters.
Phase 5 - Fall, 2000 and Spring, 2001
Reports from the "pilot" departments should be used to guide the implementation of the model into all campus departments. This should be considered the induction year. The reliability of procedures should be verified as all departments join the effort.
Phase 6 - Fall, 2001 and Spring, 2002
The model should be fully implemented by all departments.
The procedures outlined in this report inherently affect the reward system. Beyond the financial support of careful departmental and/or school experimentation toward an improved system, the increased quality of pedagogical information and faculty discussion will be felt in personnel deliberations about on-going salary adjustments. Increased staff for data gathering may be necessary, and faculty development efforts must be continued. These investments in people who assist in the assessment and support of teaching should not be considered one-time rewards but baseline budget necessities.
If teaching well--evaluated by a systematic, fair process-- becomes the natural ally of scholarly excellence at Indiana State, the University's role will become more distinct in Indiana. The greatest reward will accrue when students choose ISU because they can trust that our professors care about learning, which has always been true though under-appreciated, and we hold ourselves accountable for the many ways we spread wisdom.
Aleamoni, L. (1978). Development and factorial validation of the Arizona course/instructor evaluation questionnaire. Educational and Psychological Measurement, 38, 1063-1067.
Aleamoni, L. (1987). Techniques for evaluating and improving instruction. New directions for teaching and learning, 45. San Francisco: Jossey Bass.
Aleamoni L. and Hexner, P. (1980). A review of the research of student evaluation and a report on the effect of different sets of instructions on student course and instructor evaluation. Instructional Science, 9, 67-84.
American Association for Higher Education. (1998, March). AAHE joint task force on student learning: draft position paper. Available: http://www.aahe.org/assessment/joint.html.
American Association of University Professors (1990). Statement on teaching evaluation. AAUP Policy Documents and Reports. Washington, D.C.: AAUP, 167-70.
Angelo, T. (1993). A teacher's dozen: Fourteen general, research-based principle for improving higher learning in our classrooms. AAHE Bulletin 45, 8, April, 3-13.
Angelo, T. and Cross, P. (1993) Classroom assessment techniques. San Francisco: Jossey Bass.
Arreola, R. (1995). Developing a comprehensive faculty evaluation system. Bolton, MA: Anker Publishing Company, Inc.
Bernoff R. (1992). Effective teaching techniques. Paper presented at the National Conference on Successful Teaching and Administration, Orlando, Florida.
Boice, R. (1992). The new faculty member. San Francisco: Jossey Bass.
Boyer, Ernest L. (1990). Scholarship reconsidered: Priorities of the professorate. Princeton, New Jersey: Carnegie Foundation for the Advancement of Teaching.
Braskamp, L., Brandenbury, D., and Ory, J. (1984). Evaluating teaching effectiveness: A practical guide. Beverly Hills, CA: Sage Publications, Inc.
Brinko, K. (1993). The practice of giving feedback to improve teaching. Journal of Higher Education, 64, 574-93.
Brookfield, S. (1995). Becoming a critically reflective teacher. San Francisco: Jossey Bass.
Burdsal C. and Bardo J. (1986). Measuring students' perception of teaching: Dimensions of evaluation. Educational and Psychological Measurement, 46, 63-79.
Cashin, W. (1995). Student ratings of teaching; The research revisited. IDEA paper No. 32. Kansas State University, Center for Faculty Evaluation and Development.
Center for Teaching and Learning , ISU (1997). Characteristics of favorite/least favorite courses: A survey of first year students at Indiana State University. (draft manuscript). -- this study incorporated prior studies from Marcia Baxter Magolda on students growth and from Kenneth Bain on qualities of model teachers.
Centra, J. (1975). Colleagues as raters of classroom instruction. Journal of Higher Education, 46, 327-337.
Centra, J. (1993). Reflective faculty evaluation. San Francisco: Jossey-Bass.
Chesebro, James W. Validating the predictive model and results of open ended student interviews. Lilly Endowment Project, Sept. 29, 1997. Available: http://web.indstate.edu/lilly/aa1.htm
CIEQ. Lawrence M. Aleamoni. Comprehensive Data Evaluation Services, Inc. 6730 N. Camino Padre Isidoro. Tucson, AZ 85718.
Cohen, P. and McKeachie, W. (1980). The role of colleagues in the evaluation of college teaching. Improving College and University Teaching, 28, 147-54.
Costin, F., Greenough, W., and Menges, R. (1971). Student ratings of college teaching: Reliability, validity, and usefulness. Review of Educational Research, 41, 511-535.
Dey, Eric L., Claudia Ramirez, William S. Korn, and Alexander W. Astin. (1993). The American college teacher: National norms for the 1992-93 HERI survey. Los Angeles: UCLA, Higher Education Research Institute.
DuBois G. (1993). Hidden characteristics of effective community college teachers, Community College Journal of Research and Practice, 17, 459-71.
Feldman, K. (1988). Effective college teaching from the students' and faculty's view: Matched or mis-matched priorities? Research in Higher Education, 28, 291-344.
Gmelch, W. and Miskin, V. (1993). Leadership skills for department chairs. Bolton, MA: Anker Publishing Co.
Grasha, A. (1996). Teaching with style. Pittsburgh: Alliance.
Higgerson, M.L. (1996). Communication skills for department chairs. Bolton, MA: Anker Publishing.
Hutchings, P. (1996). Making teaching community property: A menu for peer collaboration and review. Washington, D.C.: American Association for Higher Education.
IDEA. William E. Cashin. Center for Faculty Evaluation and Development. Kansas State University. 1627 Anderson Avenue, Box 3000, Manhattan, Kansas 66502.
Kulik, J. and McKeachie, W. (1975). The evaluation of teachers in higher education. In F.N. Kerlinger (Ed.), Review of Research in Education, 3. Itasca, IL: Peacock Publishers.
Lucas, A. 1994) Strengthening departmental leadership. San Francisco: Jossey Bass.
Marsh, H. (1984). Students' evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and utility. Journal of Educational Psychology, 76, 707-754.
McCabe R. and Jenrette M. (1990). Leadership in action: A campuswide effort to strengthen teaching. In P. Seldin and associates (Eds.), How administrators can improve teaching. San Francisco: Jossey Bass.
McKeachie, W., Lin, Y., Daugherty, M., Moffett, C., Neigler, Nork, J., Walz, M., and Baldwin, R. (1980). Using student ratings consultation to improve instruction. British Journal of Educational Psychology, 50, 168-174.
Murray, J. (1995). Successful faculty development and evaluation: The complete teaching portfolio. ASHE-ERIC Higher Education Report No. 8. Washington, D.C.: George Washington University, Graduate School of Education and Human Development.
Overall J. and Marsh H. (1979). Midterm feedback from students: Its relationship to instructional improvement and students' cognitive affective outcomes. Journal of Educational Psychology, 71, 856-65.
Paulsen, M. and Feldman, K. (1995). The teacher as reflective practitioner. Taking teaching seriously: Meeting the challenge of instructional improvement. ASHE-ERIC Higher Education Report No. 2. Washington, D.C.: George Washington University, Graduate School of Education and Human Development, 39-52.
Seldin, P. (1993). How colleges evaluate professors 1983 v. 1993. AAHE Bulletin, October 1993, 6-12.
Seldin, P. (1997). The teaching portfolio, 2nd Edition. Bolton, MA: Anker.
Smith K. and Simpson R. (1995). Validating teaching competencies for faculty members in higher education: A national study using the Delphi Method. Innovative Higher Education, 19, 223-34.
Stevens J. and Aleamoni L. (1985). The use of evaluative feedback for instructional improvement: A longitudinal perspective. Instructional Science, 13, 285-304.
Theall M. and Franklin, J, eds. (1991). Effective practices for improving teaching. New Directions for Teaching and Learning, No. 48. San Francisco: Jossey Bass.
Travis, J. (1995). Models for improving college teaching: A faculty resource. ASHE-ERIC Higher Education Report No. 6. Washington, D.C.: George Washington University, Graduate School of Education and Human Development.
Whetten, D. and Whetten, K. (1985). Administrative effectiveness in higher education. Review of Higher Education, 9, 35-49.
Weimer, M. E. (1990). Improving college teaching. San Francisco: Jossey Bass.
Wilson, R. (1986). Improving faculty teaching. Journal of Higher Education, 57, 196-211.