Eva Baker

9-12 Monday GSE&IS Room 325

Office Hours 12-1; 4-5 Mondays GSE&IS 301

Eva@ucla.edu

Office Phone: 310-206-1530

Assistant: David Westhoff

Educational reform policies continue to focus on the presumed importance of the use of information about student and institutional performance. Such findings are intended to guide instruction, determine progress, suggest revisions in programs, and serve as core accountability mechanisms. But this sector of education lags far behind the rest of society in its intelligent use of technology.

This seminar will focus on the links between technological innovation, demands for the evaluation of school and other reform interventions, and emerging practices testing the achievement of students.

The goals of the seminar are that:

• Students will develop expertise in an aspect of technology and its use to assess educational quality. They will illustrate their competence by preparing an analytic literature review of a key target or component of technology-based individual or institutional evaluation. They will prepare a high-quality project that proposes an application of technology to one or more critical purposes of testing and/or evaluation.
• Students will be able to apply technical standards to the development of new technological designs for testing and evaluation. They will document their expertise by preparing detailed critiques of colleagues’ projects.

Core readings in the course will be the Revision of the Standards for Educational and Psychological Testing, to frame the technical quality and use requirements for tests of various purposes, and overviews of technology options. Much of the reading will be acquired by the student in the process of preparing assignments. Tentative assignments may be modified as a result of the class composition.

The first meeting will be organizational, in part focused on the interests and competencies of participants. Almost every class will consist of discussion of readings, technological topics, and demonstrations of sophisticated uses of technology for understanding the quality of student performance, many of which will be shown by GSE&IS students. Most readings will be distributed because of the rapid development in the field. The course outline may be modified and additional readings will be provided.

Course Schedule

Note: there will be no class April 19 (AERA) and May 31 (Holiday)

Class Topics and Preliminary Assignments

April 5

•  
• Self-assessment/pretest
• Why Technology ?
• Model-Based Assessment
• Background of Technical Standards

Next Readings: Bennett; Baker, & O’Neil; Chapters 1-4 Revision of the Standards

April 12

•  
• Validity, Reliability, Test Development
• Introduction to Mental Models
• Demonstration of Concept Mapper

Next Readings: Mental Models handouts; Herl, et al.; Chapters 7, 9, 10 Revision of the Standards

April 17^th

No class

April 26

•  
• Analytic Literature Reviews due
• Directly Assessing Mental Models
• Discussion of Accommodations and Validity for Different Test Takers
• Demonstration of Accommodations Software and Strategies

Next Readings: Chapters 12, 13 Revision of the Standards; Stevens, et al; O’Neil; Baker and Mayer; Mislevy; Goldman, et al.

May 3

•  
• Educational and Psychological Testing
• Problem Solving
• Search Strategies

Next Readings: Frase; Landauer, et al.; Burstein and Kukich

May 10

•  
• Assessing Complex Behaviors including Writing

Next Readings: Fletcher Paper; O’Neil, Brown, and Chung

May 17^th

•  
• Project Drafts Due
• Colleague Papers distributed
• Evaluation
• QSP Demo

Next Readings: Remainder of Revision of the Standards

May 24

•  
• Colleague Reviews Due
• Class time includes providing feedback to classmates on papers
• Public engagement and technology: indicators, report cards, access

May 31 (Revision of papers-no class)

June 6

•  
• Rejoinders and Project Papers Due
• So what is the future?

Course Grading

It is expected that papers will be constructed in APA style and that the majority of references will be within the last five years.

Detailed specifications for all projects will be distributed in advance of due dates, but here are short overviews of expectations

Synthesis Project: 20% of course grade

Due Date: April 26^th

Choose a topic from the list below (or negotiate one with the instructor) and prepare a well-structured, analytic literature review intended to inform colleagues on an important element of technology-based assessment or evaluation. Topics that might be considered include:

•  
• Technology to assess one of the following

  • Problem solving
  • Writing
  • Literacy
  • Collaboration or team work
  • Scientific applications
  • Workforce skills

• Automated scoring
• Automated task or item generation
• Authoring systems
• Simulations as a means to test individual or group competence
• Evidence on the effectiveness of technology

Proposal Project and Rejoinder: 60% of course grade

Due Date: First draft&emdash;May 17^th; Final draft and rejoinder&emdash;June 6

A proposal that provides rationale, literature review, design, implementation, and impact of a technology-based implementation. Among topics that might be considered for project design are the following suggestions:

•  
• The design and use of technology-based assessments with very young children to assess literacy, problem-solving or collaboration
• The use of technology to assess key areas of academic performance
• The use of technology to assess workforce competencies
• The use of technology to accommodate students with linguistic or special needs
• The use of technology-based assessment to assess teacher knowledge
• The design of strategies to evaluate the impact of after-school programs, whole-school reforms, targeted programs, using technology
• The design of technology-based training strategies to assist teachers and parents to interpret information
• The use of technology to mobilize public engagement around the quality of student achievement.

Colleague Reviews: 20% of course grade

Due Date: May 24

Analyses of colleagues proposals using a set of prescribed standards. Suggestions for improvement, in a constructive style.

updated