Learning and Evaluation Activities at the Latin American Youth Center

The Latin American Youth Center’s (LAYC) Learning and Evaluation Division has the primary responsibility for the following:

  • Ensuring a culture of learning and reflection exists through all of LAYC’s programming and locations;
  • Measuring the effectiveness of each LAYC program through internal and external evaluation activities;
  • Collecting data and information across LAYC’s programming to determine progress towards the organization’s outcomes and indicators; and
  • Helping staff use demographic, output, and outcome information to make programmatic and strategic decisions.

In February 2005, LAYC hired Isaac D. Castillo, M.S., as the Director of Learning and Evaluation to oversee all learning and evaluation efforts for the organization. In September 2006, LAYC hired Hedda McLendon, M.P.H., to further support learning and evaluation activities. LAYC hired Andrea Del Pinal, M.P.P., in April 2007 as the third member of the Learning and Evaluation Division.

Currently, LAYC uses a variety of outcome measurement methods to determine the effectiveness of approximately 50 different programs. Since July 2005, the Learning and Evaluation Specialists have trained more than 200 staff in effective data collection and entry techniques unique to their program goals.

Each of LAYC’s programs seek to measure their effects on participants (primarily youth) using a variety of methods appropriate for the service populations in question. This participant-centered approach to outcome measurement and evaluation results in the use of data collection methods and analyses that are sensitive to the needs of our populations and the challenges they face in day-to-day life. For example, LAYC works to ensure that all data collection methods are sensitive to, and appropriate for, the cultural, linguistic, gender, and educational characteristics of our service populations.

Each program at LAYC provides different services to different participants leading to different results and outcomes. Therefore, all of the programs have specific evaluation plans to collect the most appropriate data and information to accurately determine the level of program effectiveness and participant change. In addition, since many of the programs at LAYC vary in length and complexity, many of the associated evaluation methods vary in scope and sophistication in an effort to collect as many outcomes as possible without overburdening staff or participants.

With each program evaluation at LAYC, the learning and evaluation staff seek to answer three basic questions:

  • Who does the program serve?
  • What does the program do and what is the outcome(s)?
  • Why did the outcome(s) happen?

To answer these questions, each program is expected to collect three types of information to help answer the “Who” question and the “What” questions, ultimately allowing us to then answer the “Why” question. These types of data/information are depicted in the graphic below:

As shown above, Demographic Information (which includes information such as age, race/ethnicity, country of birth) allows LAYC to answer the question of “who we serve”. Process Information and Data (which documents the types of services provided, such as hours of service) allows us to document “what we did”, while Outcome Information and Data (which documents the changes participants experience, such as improved grades, job attainment, or decreased substance use) allows us to determine “what are the outcomes”. Taken collectively, all of this data and information will ultimately allow LAYC to answer perhaps the most important question of all: “why did the program outcomes happen.”

To track demographic, process, and outcome data, LAYC uses the Efforts-To-Outcomes (ETO) software package, designed by Social Solutions, Inc. The ETO software allows an agency to customize the software based on program or organizational needs; an attractive feature for LAYC and its variety of social service programs. Using the software, LAYC can analyze data and information by program, by individual participant, or across programs; enabling staff to ask and answer a range of evaluation questions.

LAYC began using ETO in June 2005, and since that time has dramatically improved its ability to collect and report demographic information on participants. This includes the ability to generate reports that show duplicated and unduplicated participant counts by date ranges for each LAYC program and the entire organization.

Through the use of ETO, LAYC’s programs have also begun in-depth collection of process data such as hours of service, as well as attendance information. Each program also uses ETO in some fashion to track outcome data. These outcome tracking methods include pre/post-test results, report card grades, test scores, structured observation tools, weekly and monthly progress reports, and case notes.

Several years ago, LAYC underwent a strategic planning process which led to the creation of a set of six outcome objectives to direct program evaluation and data collection efforts. These outcome objectives serve to generally describe the types of outcomes LAYC hopes to achieve through programming. Each program is then required to identify and develop program outcomes that support the six LAYC outcome objectives. These six LAYC outcome objectives are:

LAYC’s Outcome Objectives

  • Increased academic achievement and commitment to education, for both in-school and out-of-school youth;
  • Acquisition of career skills and knowledge for immediate employment/financial purposes and long term career development;
  • Increased awareness of the community and engagement in improving it;
  • Improved ability to initiate and sustain positive relationships;
  • Adoption of healthy habits and avoidance of risky behaviors; and
  • Increased ability to find positive solutions to the immediate needs of daily life.

LAYC’s Learning and Evaluation Specialists assist each program in the use of indicators to measure progress towards these six outcome objectives. In addition, these individuals support each program with all data collection and program evaluation activities as needed. The Learning and Evaluation Specialists also serve as the primary resource for programs when they have questions relating to evaluation and outcome measurement. When necessary, these individuals also design (and modify existing) evaluation and data collection tools to measure program effectiveness.

LAYC utilizes all outcome information and evaluation results to foster an environment conducive to continuous learning and improvement. For example, senior staff and program directors at LAYC use outcomes derived from program evaluations to recognize staff for their positive progress. LAYC also uses these successes by working to identify the specific components of a successful program in the hopes of expanding the program to new populations or service areas.

For programs or activities that result in less positive results, LAYC staff work to determine what went wrong and how the services can be improved. Program directors and senior staff view these as opportunities to refine and address struggling program components and identify areas for improvement. Each unsuccessful outcome or evaluation result is treated as a learning opportunity where staff can rethink their approaches to service provision and use existing outcome measures to determine a more effective approach to serving youth.

As each program improves its data collection and outcome reporting techniques, LAYC’s Learning and Evaluation Specialists will aggregate similar data in an attempt to identify organizational level changes and outcomes. With this step, LAYC hopes to become a leader in the youth development field by measuring positive changes experienced by youth across programs at the same organization.