What Works Literacy Partnership: Making Data Work For You

Diane Rosenthal
Literacy Partners, New York, NY

Adult basic education programs collect large amounts of data. In many instances programs tend to collect more data than they know what to do with. Most often this data is used for reporting purposes and has limited impact on a program's day-to-day operations. However, as adult educators we recognize that accurate, complete data is essential to remaining a viable and credible organization. We also know that it takes time, financial support, committed personnel and patience to create a data system that informs and is fully integrated into an agency's day-to-day operations.

The What Works Literacy Partnership (WWLP), led by Literacy Partners in New York City, was founded in 1996 with a grant from the Wallace-Readers Digest Funds. It brought together 12 exemplary adult literacy programs from across the country who were interested in building their capacity to collect, manage and analyze data, before results and accountability became the driving forces behind educational policy in the United States. WWLP represents a discerning and proactive response by national leaders in adult education, the Wallace-Reader's Digest Funds and local programs to launch a state-of-the-art effort to dramatically improve the ability of adult education programs to collect data and report on student achievements. Over the past five years these programs have worked diligently to identify effective practices that lead to using data for program improvement and decision-making. These programs believe in the fundamental power of quality data.

The programs that comprise the Partnership represent the diversity of adult education providers. They are urban, rural and somewhere in between. Budgets range from $250,000 to $4 million. Together the partners engage the services of 1,837 volunteers and employ 270 paid part-time and full-time teachers. The programs include eight that are community-based, two school district-operated, and two community college-based. They are located in Illinois, Virginia, Pennsylvania, North Carolina, Massachusetts, New York, Wisconsin, Arizona, Ohio, Vermont and Montana. The total number of students served by Partnership agencies exceeds 28,000.

For the past five years the WWLP Partners have been on a journey, sometimes frustrating sometimes exhilarating. When Partners joined WWLP they were at various stages in their data collection practices. Some had been collecting data for years as part of centralized urban networks and had sophisticated management information systems in place. Others lacked even rudimentary databases and many had no uniform assessment practices. All were committed, however, to making substantial changes in their programs and to improving their approaches to collecting, using and analyzing data. They were eager to find out how their students were doing by employing more uniform assessment practices; they were willing to administer standardized tests as well as performancebased measures. Furthermore, they were convinced that, if they began asking better questions about their programs and collected appropriate data, they would uncover new and vital information that could lead to improved instructional and administrative practices. The challenges that each program has faced have been uniquely its own. The "lessons learned" as a result of this collaborative effort, however, have begun to create a remarkably consistent picture of just what it takes to make data "work" for an agency.

As WWLP enters its final year, the Partners are writing case studies to document and share their "lessons learned" with the field. One Partner writes,

"Prior to joining WWLP we had been providing educational services to adults seeking to increase their basic academic skills, increase their English language proficiency and rind sustainable employment. We were providing these services largely without a system of student assessment, which resulted in inaccurate data, no reliable system to asses the cost of specific outcomes, and few opportunities to reliably promote and advocate for the organization in terms of its efficiency and effectiveness.... Today our program has worked quite diligently to create a system of assessment that meets external requirements yet is flexible enough to accommodote the needs of staff and students. From the identification of assessment tools, to training instructors, to revamping the database system entirely, to creating all new forms that capture essential information, we believe we are now better equipped to meet the challenges that adult education faces in the 21st century."

The What Works Literacy Partnership represents a unique model of professional development that merits examination and replication. Partner programs have had opportunities to receive training from expert researchers and educational evaluators; they have been able to experiment with the development of their assessment systems and have had the benefit of sustained interaction with colleagues around shared goals. Another Partner states,

"Our early association with the What Works Literacy Partnership brought the issues of assessment and evaluation to the forefront. By interacting with other agencies throughout the country, we were able to see the benefits that involving teachers and learners in creating a formal assessment process would have. The importance of systematizing and standardizing assessment processes was revealed through our interaction with the partners and from the expert training that we received. We learned from WWLP the importance of asking the right questions and analyzing the correct data to present a rich and detailed picture of our agency, its programs and accomplishments to funders, trustees, staff, and learners."

Top

WWLP has identified key findings for developing assessment and data collections systems, including:

• Understanding the multiple purposes for assessment -- including documenting program impact, finding ways to improve programs and monitoring individual student progress -- will help in designing an effective system;

• Involving staff in every stage of assessment and achieving staff buy-in are key ingredients to the success of any evaluation and assessment plan;

• Investing in staff development is essential;

• Administering standardized assessment measures correctly yields valid and reliable data;

• Program managers, students, tutors, funders and policymakers share the responsibility to provide high-quality adult literacy programs and to gather the evidence necessary to demonstrate that these programs acutally work;

• Students need to be involved in and understand each phase of the assessment process; and

• Asking good questions and gathering good data enable a program to analyze successes and to make improvements when necessary and desired.

Out of the WWLP effort will come project materials that can help others design effective and efficient assessment and evaluation plans. The products include:

• Self-Assessment Survey of Agency Resources and Skills
  This instrument is designed to assist programs in identifying current data collection procedures and areas that need improvement.

• Indicators of Data Proficiency: Three Stages of Growth
  This model identifies three levels of program proficiency with corresponding descriptors assigned to each level. Programs can use this document to assess what systems and practices are currently in place that support the collection and to determine what needs to be done to move the organization to the next level.

• Data Bytes Guide Sheets
  This series of information sheets answers the most frequently-asked questions about data collections, management, and analysis. Sample sheets respond to questions such as: How do I build a data collection system? How can I train teachers and tutors to collect data? How do I involve students in data collection and assessment?

• Model Data Reports
  Sample reports from WWLP agancies will provide models for effectively using data to tell an organization's story.

• Case Studies
  Each WWLP agency tells the story of how it resolved an issue related to data collection and assessment. They represent the "lessons learned" from the field.

Conclusion

The past five years have been an exciting time of discovery and challenges for the WWLP initiative. Through a process of support, training, and experimentation, the partners have developed a broad body of knowledge about what it takes to build data collection systems and develop effective assessment practices. Agencies have been able to identify the skills they need and to focus on honing those skills. They have learned what they can do with the new knowledge they have gained and are sharing that knowledge with funders, policymakers, and adult educators across the country.

The case study that follows describes how one WWLP partner -- Literacy Volunteers of America-Chippewa Valley -- identified a challenge and worked together to find solutions. For more information on the What Works Literacy Partnership or on its upcoming publications, contact 212-802-1113 or go to its newly-designed Web site at: www.wwlp.org

Originally published in Adventures in Assessment, Volume 13 (Spring 2001),
SABES/World Education, Boston, MA, Copyright 2001.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.