You are here

Assessing Teacher Competency
Share

Assessing Teacher Competency
Five Standards-Based Steps to Valid Measurement Using the CAATS Model



April 2007 | 408 pages | Corwin
"I have not seen anything quite as systematic as this material in guiding the reader through a process for developing a valid and reliable assessment plan. Covers all the areas one would want in designing a system for accreditation or for other purposes."
—Martha Gage, Director, Teacher Education & Licensure
Kansas State Department of Education

"Realistically reveals the extent of the task of teacher certification and provides us with a structured learning experience that should improve our abilities with this task."
—Pearl Solomon, Associate Professor
St. Thomas Aquinas College

A complete, step-by-step guide to teacher assessments that meet national accreditation and accountability standards.

Written in a reader-friendly style for busy faculty members and school administrators with little or no prior knowledge of statistics, this comprehensive model is designed to create fair, valid, and reliable assessments of teacher knowledge and skills.

Evaluation experts Judy Wilkerson and Steve Lang provide detailed guidance for the complete five-step assessment process, making this an ideal resource both for preservice and inservice settings, including accreditation reviews and teacher induction programs. Offering worksheets and activities to illustrate every step of the process, this all-inclusive handbook covers:

  • Definitions, contextual factors, and sampling
  • Aligning performance tasks with standards defined by NCLB, NCATE, INTASC, and other groups
  • Designing and implementing data tracking and management systems
  • Ensuring psychometric integrity

Valid and reliable decisions about teacher competency are based on fair, valid, and reliable assessment systems. Assessing Teacher Competency is the book all teacher educators, supervisors, and mentors have been waiting for.


 
A Step-by-Step Guide to the CAATS Model
 
Foreword by Richard Kunkel
 
Preface
 
Acknowledgments
 
About the Authors
 
1. Expectations and Options for Accountability and Teacher Assessment
What This Chapter Is About

 
The Challenge from the National Commission on Teaching and America’s Future

 
Title II of the Higher Education Act Amendments of 1998

 
No Child Left Behind Legislation

 
National Research Council (2001) -- the Committee on Assessment and Teacher Quality

 
What A Few Others Have Said: A Brief Review of the Literature on Testing and Licensure

 
Standards: The Roadmap to Accountability and Scientifically-Based Performance Assessment

 
The Principal Sets of Standards Governing Our Work

 
National and State Pedagogical and Content Standards

 
Unit Accreditation and Operational Standards

 
Technical Standards for Measurement of Teacher Competency

 
Some Major Threats to Validity in Most Current Assessment Systems

 
Conceptual Frameworks: Pulling It All Together

 
NCATE Standards

 
AERA, APA, & NCME

 
INTASC Principles: Where NCATE and AERA, APA, & NCME Standards Converge

 
Making Sense of Conceptual Frameworks

 
Our Conceptual Framework: What We Value

 
Assessment Options

 
Records of Training Completed

 
Tests and Exam Scores

 
Observations of Performance

 
Portfolios of Assessable Artifacts

 
Job-Related Tasks and Work Sample Products

 
K-12 Student Work Samples

 
Wrap-Up

 
Activity #1: What’s Happening in Your State and School?

 
Activity #2: Questionnaire for Faculty Views on Competency Assessment

 
Activity #3: Assessment Belief Scale

 
Activity #4: Assessment Options

 
 
2. Portfolios – To Be or Not To Be?? That IS the Question!
What This Chapter Is About

 
The Portfolio: Panacea or Pandora’s Box

 
Portfolios as Certification “Tests”: Lessons from Standards and History

 
Assessment Illiteracy, Paradigms Shifts, and Conflicting Purposes

 
The Conflict of Formative vs. Summative Evaluation

 
The Conflict of Program Approval vs. Accreditation

 
The Conflict of Business or Legislative Personal/Professional Perspectives vs. Accountability Through Title II Requirements

 
The Conflict of Academic Freedom vs. Accountability

 
The Conflict of Constructivism vs. Positivism

 
Recommendations for Use of Portfolios in Accountability Contexts

 
Ten Recommendations for Assessment System Design

 
A Recommended, Standards-Based Model

 
Overview of “Competency Assessment Aligned with Teacher Standards” (CAATS) Model

 
CAATS Step 1: Define content, purpose, use, and other contextual

 
CAATS Step 2: Develop a valid sampling plan

 
CAATS Step 3: Create or update tasks aligned with standards and

 
CAATS Step 4: Design and implement data aggregation tracking and management systems

 
CAATS Step 5: Ensure credibility and utility of data

 
Wrap-Up

 
A Note about Chapters 8 and 9

 
Activities

 
Activity #1: Review Your Feelings

 
Activity #2: Thinking about Conflicting Paradigms

 
Activity #3: Getting Your Action Plan Started

 
 
3. CAATS Step 1: Assessment Design Inputs
Where We Have Been So Far

 
What This Chapter Is About

 
CAATS Step 1A: Define the Purposes and Uses of the System

 
The Importance of Purpose and Use

 
Different Strokes for Different Folks for Different Purposes

 
Purpose and Use in the Accountability Context

 
More on Accountability Based Systems: State Program Approval

 
CAATS Step 1B: Define the Propositions or Principles that Guide the System

 
CAATS Step 1C: Define the Conceptual Framework or Contents of the System

 
So What Is Assessment Content?

 
Standards as the Link between Purpose, Use, and Content

 
CAATS Step 1D: Review Local Factors That Impact the System

 
Wrap Up

 
Worksheets

 
Worksheet #1: Purpose, Use, Propositions, Content, & Context Checksheet

 
Worksheet #2: Purpose, Use, Content, Draft

 
Worksheet #3: Propositions

 
Worksheet #4: Contextual Analysis

 
 
4. CAATS Step 2: Planning with a Continuing Eye on Valid Assessment Decisions
Where We Have Been So Far

 
What This Chapter Is About

 
CAATS Step 2A: Organize Standards into Content Domains

 
All Those Standards Sets

 
Organizing for Alignment

 
“A Rose Is a Rose” Or More of the Same

 
Why Bother?

 
Crosswalks and Standards

 
CAATS Step 2B: Visualize the Competent Teacher Based on the Standards

 
CAATS Step 2C: Brainstorm a Set of Summative Tasks (Sampling Plan)

 
CAATS Step 2D: Sort Tasks into Formative and Summative Assessments

 
CAATS Step 2E: Build Assessment Frameworks

 
Framework Options

 
A Special Case: Aligning Tasks with NCATE Requirements

 
Wrap-Up

 
Worksheets

 
Worksheet #1: Organizing for Alignment (Version 1)

 
Worksheet #1: Organizing for Alignment (Version 2)

 
Worksheet #2: Our Critical Skills

 
Worksheet #3: Visualizing the Competent Teacher

 
Worksheet #4: Critical Task List

 
Worksheet #5: Sorting Formative and Summative Tasks

 
Worksheet #6: List of Summative Assessments by Competency Type

 
Worksheet #7: List of Summative Assessments by Levels of Inference

 
Worksheet #8: List of Summative Assessments by Points in Time

 
Worksheet #9: Matrix of Standard by Competency Type

 
Worksheet #10: Matrix of Critical Tasks by Competency Type and Benchmark

 
Worksheet #11: Aligning Tasks with NCATE Thematic Portfolios

 
 
5. CAATS Step 3 -- Writing Tasks Designed to Maximize Validity and Reliability
Where We Have Been So Far

 
What This Chapter Is About

 
CAATS Step 3A: Determine the Task Format for Data Aggregation

 
What Happens When There Is No Format?

 
Can I Use Percents and Total Points? No, They Don’t Cut It!

 
Elements of a Common Task Format

 
CAATS Step 3B: Create New Tasks or Modify Existing Tasks

 
Basic Concepts about Tasks

 
Hints and Advice about Writing Tasks

 
Rubric Examples: A Rose is a Rose

 
CAATS Step 3C: Conduct First Validity Study

 
CAATS Step 3D: Align Tasks with Instruction

 
Wrap-Up

 
Worksheets

 
Worksheet #1: Proficiency Level Descriptions

 
Worksheet #2: Task Design

 
Worksheet #3: Standards and Indicators Coverage Report

 
Worksheet #4: Individual Task Review for Job-Relatedness

 
Worksheet #5: Checklist for Reviewing Individual Tasks

 
Worksheet #6: Instructional Alignment

 
 
6. CAATS Step 4: Decision-Making and Data Management
Where We Have Been So Far

 
What This Chapter Is About

 
CAATS Step 4A: Determine How Data Will Be Aggregated

 
So What Is Data Aggregation?

 
The Relationship of Data Aggregation (Step 4A), Cut Scores, (Step 4B), and

 
An Approach to Decision-Making without Using Points and Percents

 
CAATS Step 4B: Set Standards for Minimal Competency

 
Different Strokes for Different Folks – Déjà Vu or Purpose Revisited

 
Is 100% Really Reasonable?

 
Criticality Yardstick Approach (CYA) to Cut Score Setting in Complex

 
Performance Assessments

 
First Cut

 
Second Cut

 
Working with Judges

 
An Example of What to Say to the Judges

 
CAATS Step 4C: Select and Develop a Tracking System

 
Sharing Information for Decision-Making: The Big Challenge

 
Data Storage Option #1: Course Grades or Records of Participation/Attendance

 
Data Storage Option #2: Teacher Folders

 
Data Storage Option #3: Portfolios

 
Data Storage Option #4: Electronic Data Management System

 
Reporting Aggregated Data

 
CAATS Step 4D: Develop Management System

 
Advising and Due Process

 
Scoring Procedures

 
Implementation

 
Wrap-Up

 
Worksheets

 
Worksheet #1: Cut Score Decisions

 
Worksheet #2: Sample Format for Candidate/Teacher Tracking Form

 
Worksheet #3: Format for Data Aggregation

 
Worksheet #4: Management Plan

 
Worksheet #5: Rater Monitoring Record

 
 
7. CAATS Step 5: Credible Data
Where We Have Been So Far

 
What This Chapter Is About

 
What Is Psychometric Integrity and Why Do We Have to Worry About It?

 
CAATS Step 5A: Create a Plan to Provide Evidence of Validity, Reliability, Fairness, and Utility

 
Elements of a Plan

 
Element #1: Purpose and Use

 
Element #2: Construct Measured

 
Element #3: Interpretation and Reporting of Scores

 
Element #4: Assessment Specifications and Content Map

 
Element #5: Assessor/Rater Selection and Training Procedures

 
Element #6: Analysis Methodology

 
Element #7: External Review Personnel and Methodology

 
Element #8: Evidence of Validity, Reliability, and Fairness (VRF)

 
CAATS Step 5B: Implement the Plan Conscientiously

 
Wrap Up

 
Worksheets and Examples

 
Worksheet #1: Assessment Specifications

 
Worksheet #2: Analysis of Appropriateness of Decisions for Teacher Failures

 
Worksheet #3: Program Improvement Record

 
Worksheet #4: Expert Rescoring

 
Worksheet #5: Fairness Review

 
Worksheet #6: Analysis of Remediation Efforts and EO Impact

 
Worksheet #7: Psychometric Plan Format

 
Example #1 (Empirical Data): Logistic Ruler for Content Validity

 
Example #2 (Empirical Data): Computation of the Lawshe (1975) Content

 
Example #3 (Empirical Data): Disparate Impact Analysis

 
Example #4 (Empirical Data): Computation of Cohen’s Kappa (1960) for

 
Example #5 (Empirical Data): Spearman Correlation Coefficient and Scatterplot:

 
Example #6 (Empirical Data): Pearson Correlation Coefficient and Scatterplot:

 
Example #7 (Empirical Data): Correlation Matrix and Scatterplots Knowledge,

 
Example #8 (Empirical Data): T-Test Comparing Mathematics and Science

 
Example #9 (Empirical Data): Differential Item and Person Functioning

 
 
8. The Trouble with Tribbles: Standard Setting for Professional Certification
What This Chapter Is About

 
An Overview of Cut Score Setting

 
From Norm-Referenced Objective Tests to Criterion-Referenced Subjective Tasks

 
Controlling Human Judgment

 
Difficulty vs. Importance

 
Social Consequences

 
Tidbits to Remember

 
Cut Scores Setting in Traditional, Objective Tests

 
Holistic Impressions: Method One

 
Item Content: Method Two

 
Performance of Examinees: Method Three

 
Combination Approach: Method Four

 
Standard or Cut Score Setting in Performance Tests for Professionals: Is it the Same as Multiple Choice?

 
Quotes to Remember

 
The Extended Angoff Procedure for Performance Assessment

 
The Judgmental Policy Capturing Approach for Performance Assessment

 
The Dominant Profile Judgment for Performance Assessment

 
Standard Setting Using Item Response Theory

 
What’s Really Wrong with Current Approaches to Cut Score Setting

 
Berk’s Suggestions and Commentaries Over the Years

 
The Criticality Yardstick Approach

 
Wrap Up

 
Activities

 
Activity #1: Change the Difficulty of an Item

 
Activity #2: Change the Criticality of a Task

 
Activity #3: Replicate a Cut Score

 
Activity #4: Take Your Choice

 
 
9. Using Teacher Scores for Continuous Improvement
What This Chapter is About

 
Reasons Why We Use the Rasch Model

 
The Classical Approach

 
A Quick Overview of Where Rasch Fits Into the Grand Scheme of IRT Models

 
Rasch: The Basics

 
Getting Started

 
Differences that Item Writers Make

 
Guttman Scaling

 
A Sample Rasch Ruler

 
From Pictures to Numbers

 
The Fit Statistic

 
Gain Scores – Real or Imagined?

 
Ratings and Raters

 
Learning More about Rasch

 
Wrap Up

 
Activities

 
Activity #1: Decision Making Tool for Measurement

 
 
10. Legal Integrity
What IF?? A Legal Scenario

 
Psychometric Issues and Legal Challenges

 
Legal Issues and Precedents

 
End Note

 
 
Appendix: Tasks Developed for Florida Alternative Certification Program
 
Glossary
 
References
 
Index

"Provides possible solutions for the problems faced in the assessment of future teachers, and realistically reveals the extent of the task of teacher certification. It provides those responsible for teacher certification with a structured learning experience that should improve our abilities with this task."

Pearl Solomon, Associate Professor
St. Thomas Aquinas College

"I have not seen anything quite as systematic as this material, which guides a reader through a process for developing a valid and reliable assessment plan.  It covers all the areas that one would want to see covered in designing a system for accreditation or other purposes."

Martha Gage, Director
Teacher Education & Licensure, Kansas State Dept. of Education

"Structurally accurate, complete, and readable. The activities at the end of each chapter are among the best I have ever seen."

Elaine L. Wilmore, Professor of Educational Leadership
Dallas Baptist University

"Wilkerson and Lang provide readers a solid foundation and strong design to help them think through the use of data collected on teacher preparation program graduates that impact the national mosaic."

SirReadaLot.org, June 2007

"The authors show how to track teacher performance in a way that ensures credibility through the use of collected data and scoring methodologies for differentiating between effective and ineffective instructors."

The School Administrator, December 2007
American Association of School Administrators
Key features
  • A comprehensive guide to designing and implementing accurate, reliable, and useful teacher assessments
  • Assessments are aligned with the standards and benchmarks defined by NCLB, NCATE, INTASC, and other key programs and groups in education
  • Provides background, context, guidelines for designing and implementing data tracking and management systems, and ensuring psychometric integrity 
  • Includes worksheets and activities to illustrate every step of the process
  • includes a Glossary, Index, and a Step-by-Step Guide to the CAATS Model 
  • Appropriate both for pre-service and in-service faculty, administrative staff, and supervisors

.

Preview this book

For instructors

Select a Purchasing Option

ISBN: 9781412941204
$44.95
ISBN: 9781412941198
$90.95