Facts About The
UCLA Center for the Study of Evaluation
&
National Center for Research on Evaluation,
Standards, and Student Testing
Welcome
For more than 35 years, the UCLA Center for the Study of Evaluation (CSE) and, more recently, the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) have been on the forefront of efforts to improve the quality of education and learning in America. Located within UCLA’s Graduate School of Education & Information Studies, CSE/CRESST has pioneered the development of scientifically based evaluation and testing techniques, vigorously promoting the accurate use of data, test scores, and technology for improved accountability and decision making.
In the past year, CSE/CRESST has grown to meet expanded needs resulting from changes in federal and state laws and has substantially broadened its research well beyond the K-12 educator audience. Through the addition of several new projects, CRESST research and development now extends significantly into pre-school and after-school programs and also includes studies of adult learning using advanced technology. The 2002 passage of the No Child Left Behind Act provides additional opportunities for research and development related to state, district, and local school accountability needs.
CSE/CRESST History
The UCLA Center for the Study of Evaluation was established in 1966 when, as a result of a national competition, it was designated the national center for research in educational evaluation. In 1985, CSE won a competition for the National Center for Research on Evaluation, Standards, and Student Testing (CRESST), which has been continuously funded through the U.S. Department of Education’s Office of Educational Research and Improvement (OERI). In 2001, after a favorable federal review of its research quality and products, CRESST received a 5-year extension to continue its research and development programs. CRESST was one of only four centers to receive a full 5-year extension.
CSE/CRESST produces rigorous evaluation and accountability methods, as well as new designs for assessing and supporting learning, educational policy guidance, guidelines for improvements in practice, new information technologies, and explores opportunities for all children.
1. Rigorous Evaluation and Accountability Methods
CSE/CRESST research has produced some of the most respected research and evaluation methods by distinguished scholars, including Marvin Alkin, Eva Baker, Benjamin Bloom, Hilda Borko, Lee J. Cronbach, Robert Glaser, Joan Herman, Ernest House, Daniel Koretz, Robert Linn, W. James Popham, Lauren Resnick, Richard Shavelson, Lorrie Shepard, Noreen Webb, and dozens of others. Through innovative designs and new analytic techniques, CSE/CRESST research methods help states, districts, and schools improve the accuracy and decision making in their programs and accountability systems.
2. New Designs for Assessing and Supporting Student Learning
CSE/CRESST pioneered research in the conception, design, and analysis of new forms of student assessment. These include criterion- and domain-referenced tests, performance assessments and portfolios, and most recently, standards-based assessments. CSE/CRESST assessment designs have been used in numerous large-scale accountability systems in states and districts, including California, Hawaii, Illinois, Los Angeles, and Philadelphia.
3. Educational Policy Guidance
CSE/CRESST influences virtually every major assessment enterprise in the nation. CRESST Co-director Eva Baker was a member of the influential National Council on Education Standards and Testing, which has had tremendous impact on state and national assessment systems during the past decade. Our technical research team, headed by CRESST Co-director Robert Linn, conducted invaluable research supporting the National Assessment of Educational Progress, America’s Report Card.
CRESST research supported the implementation of the accountability requirements of the 1994 Improving America’s School Act, providing regional conferences for all states and school districts, with important standards and assessment guidance. CRESST has recently released the Standards for Educational Accountability Systems, a targeted synthesis of research findings to guide policymakers as they implement assessment and accountability systems. Current research has studied important issues related to the adequate yearly progress requirements of the 2002 No Child Left Behind Act. Policymakers continue to use CRESST research to inform their decision making and view CRESST as a source of accurate and useful information.
4. Improvements in Practice
CRESST Co-director Eva Baker was co-chair of the joint committee conducting the 1999 revision of the Standards for Educational and Psychological Testing, a document that influences virtually every state and district test in the nation. Research on assessment accommodation issues for English language learners conducted by CRESST researcher Jamal Abedi has had substantial impact on state testing practices across the nation. CRESST research into the effects of accountability systems on schools and teaching has had substantial impact on assessment designs and information use.
CSE/CRESST also produces valuable tools for use by practitioners including School Report Cards and the Quality School Portfolio (QSP). QSP is data analysis software that is used in 49 states to help schools measure school progress and improvement. The highly regarded CSE Program Evaluation Kit, now in its second edition, has sold more than 150,000 volumes. A Practical Guide to Alternative Assessment has reached more than 100,000 educators, making it one of the most widely utilized assessment books in the country.
5. New Information Technologies
CSE/CRESST projects have explored technology’s potential in the design, administration, and interpretation of assessments; in the dissemination of the center’s products; and in stimulating communication with constituencies. Studies by CRESST Co-director Joan Herman and other CRESST researchers have evaluated and analyzed technology innovations in dozens of schools, providing key recommendations for improvement. The Quality School Portfolio software is now moving to a Web-based format and will provide teachers and parents with comprehensive information about student performance, including recommendations for improvement.
CRESST has created a partnership with the Alexandria (VA) Co-Lab of the Advanced Distributed Learning (ADL) initiative to provide assessment support intended to significantly improve the training and readiness of U.S. armed forces personnel.
6. Opportunities for All Children
CSE/CRESST has had a long-term goal to promote evaluation and test use that provides children with improved opportunities to achieve high standards. Our research for the U.S. Department of Education, for example, has promoted the use of assessment accommodations not only in assessment but also for classroom instruction. Recent work by CRESST researchers and partners Jamal Abedi, Frances Butler, Alison Bailey, Lorrie Shepard, and Dan Koretz has provided invaluable insight into methods for assuring that thousands of at-risk children are included in accountability systems and assessed equitably.
CSE/CRESST goals will be further enhanced by the addition of several new long-term projects. They include
• long-term support from the Office of Naval Research for the development of the next generation of distance learning, based on an integrated set of distance learning and instruction guidelines and assessment models;
• funding from the Interagency Educational Research Initiative (IERI) to design the Assessment Design and Delivery System (ADDS), which supports teachers’ design and use of classroom assessments in science;
• 3 years of funding from the U.S. Department of Education to create an online version of the Quality School Portfolio; and
• a partnership with funding from the National Science Foundation, supporting the training of scholars in assessment, in collaboration with UC Berkeley, Stanford University, and WestEd.
Current CSE/CRESST Initiatives
CRESST Initiatives
CSE’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is a partnership of UCLA, the University of Colorado, Stanford University, Harvard University, The RAND Corporation, the University of Pittsburgh, the University of Southern California, Educational Testing Service, and the University of Maryland. The CRESST mission focuses on the assessment of educational quality, addressing persistent problems in the design and use of assessment systems to serve multiple purposes. Together, the institutions comprising CRESST are committed to
1. exerting intellectual leadership in the purpose, design, and interpretation of assessment systems for the research, practice, and policy communities;
2. conducting high-quality research to create new knowledge and advance theories that will have long-term impact on conceptions and interpretations of educational quality;
3. developing tools, procedures, and systems to improve the quality of assessment as practiced in schools, interpreted by policymakers, and understood by the public;
4. creating and evaluating approaches and accommodations that promote equity of assessment; and
5. engaging teachers, researchers, policymakers, and the public in reflection and action to improve assessment and its link to educational quality through direct connections with CRESST products and staff.
CRESST research is organized into four sustained R&D programs and one integrated dissemination program as described below.
Program One is at the heart of CRESST’s largest interests: the validity of inferences drawn from information that is part of accountability systems. CRESST researchers explore the impact on instruction of different approaches to assessment and accountability and the impact of variations in system design on the validity of inferences, with a special emphasis on the challenges and requirements of current No Child Left Behind legislation. The program also includes a forward-looking design component focusing on "ideal" and improved assessment, indicator, and accountability systems.
Program Two features research on assessment tools, including new approaches to studying classroom practice and classroom assessments that can support student learning and accountability purposes. This program uses cognitive models in its approach and explores tools and strategies for potential use in bridging the gap between large-scale assessment and classroom teaching and learning.
Program Three focuses on methodology, continuing our research inquiry into classification accuracy, indicators development, growth modeling, new approaches to alignment methodology, standard setting, and study group topics such as validity and multiple measures. Program Three has direct impact on all operational programs.
Program Four addresses the overarching issue of the validity of inferences drawn from assessment and accountability systems but focuses on historically lower performing students, particularly those who are English language learners or who are in big-city school systems. Projects address issues of capacity and instructional improvement, as well as special topics associated with linguistic and other special needs of students.
Program Five is our dissemination effort, including our Web-based resources, conference activities, video and Web professional development, media and legislative programs, and school report cards.
CSE/CRESST Special Studies
CSE/CRESST has been a leader in international, national, state, and local testing and evaluation projects. CSE/CRESST has created a diverse and broad talent range of expert project leaders who frequently collaborate on challenging educational research questions. Current or recent special and field service projects include the following.
Assessment Research and Development
• Assessments to Support the Transition to Complex Learning in Science is a project that integrates science education and assessment research in the construction of Web-based assessment tools for middle school science teachers. In the course of the 4-year project, funded by IERI (Interagency Educational Research Initiative), CSE/CRESST is developing an Assessment Design and Delivery System (ADDS) that assists teachers in capturing relevant content knowledge and facets of student understanding so that instruction may be better adapted to student needs. ADDS will also provide online delivery of assessments, reporting and advice systems, on-the-fly scoring of some assessments, and instructional guidance. Contact Eva Baker or David Niemi for more information.
• In Technical Studies on New Kinds of Postsecondary Admissions Tests, CSE/CRESST researchers created a compendium of relevant research and completed an analysis of test content, linguistic demands, and cognitive tasks. The researchers also made evaluation recommendations in support of pending decisions to change the UC admissions testing requirements. Contact Eva Baker, Bob Linn, Joan Herman, or Dan Koretz for more information.
• CSE/CRESST provides assistance in the development of curriculum-embedded assessments, technical oversight, and internal research studies for the Inglewood Unified School District (CA). An integral part of this work involves reviews of the quality of the assessments based on their alignment with standards, their impact on classroom practice, and on the technical characteristics of the assessments given the intended purposes of test results. Contact Zenaida Aguirre-Muñoz for more information.
• Linked to the California State Standards, the Los Angeles Unified School District’s Comprehensive Assessment System consists of performance-based reading and writing assessments. CSE/CRESST designed these assessments to target student understanding of key concepts and skills that develop incrementally in complexity and demand across grade levels. Results from the assessment system serve the District’s accountability and school improvement goals. Extensive validation and score reliability evidence has been collected over the 6 years of the project, and instructional sensitivity studies are being conducted this year to augment that evidence.
The project includes teacher capacity-building and an online rater training system. Goals of the Performance Assignment Scoring Training Web Site include standardization of teacher training for scoring performance assessments, a system for scoring performance assessment measures, and real-time reporting. Teachers and other scorers are given access to all released performance assessments, participate in online training sessions, score sets of actual student responses, and receive feedback on score agreement with other groups of scorers. The Training Web Site is being funded by the Los Angeles Unified School District. Contact Roxanne Sylvester or David Niemi for more information.
Accountability Indicators and Reporting
• In the Teacher Assignment Indicators Project, CRESST and the Los Angeles Unified School District are using collections of teachers’ assignments and student work as part of the district’s proposed accountability system (the Local District Performance Measures project). The goal of this new accountability system, piloted during the 2000-2001 academic year, is to measure both direct outcomes of student performance and the school processes, including the quality of classroom practice, expected to increase student performance. For further information about this project, please contact Lindsay Clare Matsumura.
• Comprehensive Assessment System. CSE/CRESST recently completed a collaboration with the state of Wyoming on the refinement of its indicator system for assessing the quality of education at state, district, and local school levels. This work produced state, district, and uniform school-level reports for practitioners, policymakers, parents, and the public. Contact Joan Herman or Greg Leazer for more information.
• CSE/CRESST is working in collaboration with the National School Boards Foundation (NSBF) and the American Association of School Administrators (AASA) to study the Data Use and Reporting Practices of districts across the country. Drawing on surveys and case studies involving school board members, district administrators, principals, and teachers, CSE/CRESST researchers are developing the knowledge and design principles necessary to develop effective data systems and build capacity to meet school, district, and school board needs. The objective is to increase district capacity to ask good questions and use data to support local school improvement efforts. Contact Joan Herman for more information.
• In the School Report Project, CSE/CRESST worked closely with the Los Angeles Mayor’s Office and the Los Angeles Unified School District, Oakland Unified Schools, and other districts to develop a reporting system to improve public access to, and the understanding of, information about school performance. Researchers developed new approaches to displaying and reporting information on existing school accountability reports. Research questions addressed the usability of the information and the clarity of the reporting strategy. Contact Ron Dietel for more information about this completed project.
Evaluation of Educational Programs
• CSE/CRESST researchers are conducting a comprehensive evaluation of the implementation and impact of the Early Steps to Reading Success Initiative. The purpose of Early Steps is to improve the quality of children’s earliest learning experiences so that they enter school "ready to learn." To achieve this goal, Early Steps seeks to provide high-quality, developmentally and culturally appropriate professional development in early literacy to early childhood educators who serve at-risk preschool children. Contact Lindsay Clare Matsumura or Christy Kim Boscardin for further information about this project.
• Funded by the University of California Office of the President, CSE/CRESST researchers are conducting an evaluation of the California Professional Development Institutes (CPDI) in reading for pre-Kindergarten—Grade 3. These Institutes seek to provide intensive and ongoing professional development in reading instruction for pre-K—Grade 3 teachers through collaborations among the University of California, the California State University, private colleges, and California school districts. The 4-year evaluation project will provide information about the impact of the Institutes on participating teachers, their classroom practices, and their students. Contact Joan Herman or Noelle Griffin for more information.
• Since 1999, CSE/CRESST has been evaluating the Walt Disney Company’s Creative Learning Communities (CLC). The CLC grant program provides funding to public elementary schools to design and implement creative teaching strategies to improve student performance. Contact Joan Herman or Ann Mastergeorge for more information.
• CSE/CRESST is conducting a comprehensive, 74-site evaluation study of LA’s BEST, an afterschool program that includes academic tutoring, enrichment, recreation activities, nutrition, and self-esteem development. The CSE/CRESST study is designed to assess the impact of LA’s BEST on students’ social and academic development. Contact Denise Huang for more information.
• LA’s BEST, in cooperation with the Action Learning System, is pioneering a cross-age tutoring program called Literacy Loop in an effort to enhance the reading comprehension and writing skills of LA’s BEST students. CSE/CRESST is conducting an evaluation study to measure program impact. Contact Denise Huang for more information.
• The LA’s BEST Long-Term Effect Study is designed to assess the long-term impact of LA’s BEST on participating students’ academic attainments and citizenship. The results of this study will provide a preliminary overview on its long-term effects, such as educational aspirations and community involvement of the students, and document the long-term impression of the program on the participants. Contact Denise Huang for more information.
• CSE/CRESST is conducting research to evaluate the impact of the Kidzlit program, which has been implemented in several sites of the LA’s BEST afterschool program. Kidzlit, created by the Developmental Studies Center in Oakland, California, is a reading program that combines the relaxed and playful atmosphere of an afterschool setting with a focus on learning that supports in-school work. Contact Eva Chen for more information.
• The arts education component of the Grammy Awards organization, The Grammy Foundation, contracted with CSE/CRESST to conduct an evaluation of the Foundation’s Artful Learning Program. The Artful Learning Program is a school reform model that uses masterworks of art (music, dance, visual art) as centerpieces for the development and implementation of an integrated, schoolwide curriculum. This model, based on the arts education work of the late composer Leonard Bernstein, is currently being used by over 20 schools nationwide. CSE/CRESST is evaluating the quality of implementation of the program at the participating schools, as well as the impact of the program on the schools, teachers, and students. Contact Noelle Griffin for more information.
• CSE/CRESST researchers recently completed an evaluation of the Focused Monitoring Technical Assistance (FMTA) program, funded by the California Department of Education/Special Education Division. The FMTA program is a quality assurance initiative designed to transform special education programs across the state. The FMTA provides funding to school districts across California to focus on their students’ needs and to assure district accountability for progress on specific goals for student performance and compliance with IDEA ’97. To date, this evaluation has examined how districts reorganized and made programmatic and curricular changes in order to address the needs of their special education students. Contact Joan Herman or Valeria Chow for more information.
•
Learning Through Innovative Technology
• Knowledge, Models and Tools to Improve the Effectiveness of Naval Distance Learning. An interdisciplinary team led by CSE/CRESST is conducting research on assessment models and assessment-guided instructional strategies applied to advanced distributed learning (ADL) for Naval training. The project will study new approaches to online assessment and its use in designing and delivering distance learning courses for higher education and corporate training, as well as for military environments. The research will result in a science-based decision-support system aiding course specifiers, designers, and developers in providing distance learning on a variety of tasks to a variety of learners in a variety of environments. Funded by the Office of Naval Research, the project team includes CSE/CRESST’s assessment and technology staff and subcontractors providing simulation authoring software, learner support strategies, and ethnographic analysis of the learner environment. Contact Bill Bewley or Eva Baker for more information.
• CSE/CRESST is evaluating the Advanced Educational Technology Program. Teachers in the program create and implement innovative learning and assessment methods in their classrooms using the computer-based IMMEX program, developed by Dr. Ron Stevens at UCLA. CSE/CRESST is also assisting with psychometric studies and assessment of students’ problem-solving strategies. Contact Eva Chen or Greg Chung for more information.
• In Assessing Problem Solving Through Model-Based Simulations and Behavioral Processes, CSE/CRESST studied advanced graphics-based simulator technology to create improved assessments. CSE/CRESST believes that, compared to paper-based formats, simulation tasks can be more authentic, measure a range of cognitive demands, and require students to demonstrate understanding. This recently completed project was funded by UCLA’s Center for Digital Innovation, based on an award from the GTE Foundation. Contact Greg Chung for more information.
• In Technologies for Sensor-Based Wireless Networks, UCLA researchers are developing wireless networking, middleware, and data management technologies in problem-solving environments. The system will enhance the early childhood education process by providing a learning environment individualized to each child, adaptive to the context, and with unobtrusive evaluation of the learning process by the teacher. Funded by the National Science Foundation, the project team is interdisciplinary, with researchers from multiple UCLA departments. Contact Greg Chung or Eva Baker for more information.
• Funded by the National Science Foundation, CRESST is a partner in the Center for the Assessment and Evaluation of Student Learning (CAESL). CAESL is a 5-year project with the goal of improving student learning and understanding in science through focusing on effective assessment. Other partners include the School of Education at Stanford University, the Graduate School of Education at the University of California-Berkeley, the Lawrence Hall of Science (LHS), the Concord Consortium, and WestEd.
Tools to Link Accountability With School and Classroom Practices
• The U.S. Department of Education is funding CRESST $2.7 million during 3 years to create the Quality School Portfolio Web Version. Already used in a desktop version in 49 states, the new version of QSP will be Web-based and will help schools use data to inform curriculum and instruction, plan for school improvement and communicate effectively with stakeholders. CRESST researchers have recently completed the Quality School Portfolio Training Initiative, funded by the Stuart Foundation and the Joyce Foundation. The Initiative provided schools engaged in local reform efforts with the capacity to use and report on student information through a versatile software product developed by CSE/CRESST. Contact Margaret Heritage, John Lee, or Eva Baker for further information.
• Funded by the Stuart Foundation, CSE/CRESST recently concluded the School and Classroom Practices in LAAMP Schools study to investigate classroom assignments as indicators of effective teacher practice. The study also focused on the potential utility of this method for teachers’ professional development. Contact Lindsay Clare Matsumura for more information.
Validity of Assessments for English Language Learners
• CSE/CRESST is conducting studies to examine the Validity and Consequences of Performance Assessment for English Language Learners (ELLs). Researchers are examining how implementation of performance assessment in English language arts and related instructional changes influence the achievement of ELLs. Key areas of research focus are measurement of ELL opportunity to learn and teacher capacity-building strategies to increase the alignment of instruction and assessment and improve ELL learning. Current efforts are focused at Grade 6. Contact Zenaida Aguirre-Muñoz or Christy Kim Boscardin for more information.
• Supported by the Office of Bilingual Education and Minority Languages Affairs (OBEMLA), CSE/CRESST has recently completed research to examine the validity of Assessing English Language Learners (ELLs). Researchers studied the impact of reading proficiency on ELLs’ content-based performance. CSE/CRESST evaluated test accommodations for the inclusion of ELLs in large-scale assessment programs, with efforts focusing on Grades 4 and 8. Contact Jamal Abedi for more information.
A second part of this project was an observational study to help characterize the type of language used in the classroom for academic tasks and necessary for performance on standardized content tests. Research efforts focused on Grades 4-6 in science classrooms for this recently completed project. Contact Frances Butler or Alison Bailey for more information.
• Supported by the National Center for Education Statistics, CSE/CRESST recently completed studies examining the Validity of Accommodations for ELL Students. Several new accommodation strategies were studied for the first time by the CRESST researchers, including computer testing, linguistically modified versions of content-based test items, and use of a customized dictionary in which content-based terms can be excluded. Contact Jamal Abedi for more information.
Dissemination
CSE/CRESST produces a variety of dissemination materials. A quarterly newsletter, CRESST Line, contains up-to-date news on current CRESST assessment research, and annual CRESST Policy Briefs provide information and answers on urgent accountability topics. The CRESST Policy Briefs and CRESST Line are distributed free to a mailing list of approximately 7,500 education policymakers, researchers, and practitioners in the United States and overseas.
Other Center dissemination activities include an active public relations program, an annual conference, a technical report series, edited books, and videotape production. Technical reports and other educational products are produced annually, and more than 12,000 products are distributed at no cost each month, frequently to state, local, or national education agencies and also to many research organizations. CSE/CRESST maintains an active relationship with dozens of education service providers, including the OERI regional laboratories and research centers, plus hundreds of professional education organizations. A World Wide Web server provides Internet users with up-to-date information on Center research results.
Computer, Video, and
Technical Capacity
CSE/CRESST houses more than 150 Macintosh and PC systems on an Ethernet network and provides modem access to CSE/CRESST’s computer services for off-site staff. Information is shared internally through server-based databases and an internal Web server. CSE/CRESST has a state-of-the-art video conference system with dual-mode Internet connectivity and multiple location support. Both internal and external researchers are immediately accessible to clients and customers with adequate Internet speed.
The CRESST World Wide Web server is used to disseminate CSE/CRESST research information to Internet users. The server contains more than 230 downloadable technical reports, plus newsletters and assessment products, all available in PDF format. CSE/CRESST was the first OERI educational research center to offer technical reports formatted with full text, graphics, and tables.
CSE/CRESST has in-house video editing equipment and staff trained in the operational use of digital video technology. Final productions are satellite broadcast and downloaded by many PBS stations and school districts across the country.
CSE/CRESST employs approximately 170 people at its UCLA location, including more than 40 graduate or undergraduate students. Many project directors and staff are among the most widely recognized authorities in their respective fields.
For further information, please contact Ronald Dietel, CRESST Director of Communications, at 310-794-9168 or ron@ucla.edu, or visit our Web site at www.cse.ucla.edu.
Updated June 20, 2002