CRR - An Overview | Vision 20/20

CRR – An Overview

The Vision 20/20 Project supports the application of standard evaluation measures to Community Risk Reduction programs through various means. From the Vision 20/20 website, you can access CRR Education, examples of model programs from past Symposia, and other resources to aid understanding and application, including:

Applying Model Performance Measures Terminology to Community Risk Reduction Programs

In very simple terms, the Formative, Process, Impact and Outcome measures for fire community risk reduction programs can be described as follows:

Formative Evaluation: The measures that describe the formative research done to help us focus desired changes.
This can include a needs or a risk assessment which tells us things like fire incident rates, emergency medical incident rates, age, race and income of people affected so we know who is at highest risk. It can include incident response types, the time they occur, the location they occur so we know where our longest response times or most frequent use of the system are. It can involve more detailed examination of the demographics of the community involved so we know more about who is being affected by the emergencies we are called upon to mitigate either through an emergency response or proactive prevention efforts. The formative process includes the establishment of program goals and objectives, and development of materials and methods for intervention. Formative Measurement Examples

Process Evaluation: The measures that tell us how programs were implemented so we can quantify the number and extent of activities and resources committed to reaching the program objectives.
Commonly we look at outputs – or the numbers of things done in this category of evaluation. Process Measurement Examples

Impact Evaluation: The measures that tell us how much our programs helped to change the risks of our intended audiences.
It is measuring things like changes in knowledge or behaviors, or changes in the environment (like smoke alarm installations, or home safety visits that identify and correct fall hazards for seniors) that we know reduce risks. Impacts in this context are about risk – and do not necessarily affect outcomes. Impact Measurement Examples

Outcome Evaluation: The measures that tell us if our ultimate goals of public safety have been reached by documenting changes in fire or medical incidence, dollar losses, injuries or deaths.
We need to note that documenting changes in outcome measures need to be done over time because numbers can change from year to year due to random chance or other community variables. Outcome Measurement Examples

FORMATIVE MEASUREMENT EXAMPLES

Code Compliance Programs (ways to identify risks or needs)

- Survey records to identify buildings in need of code enforcement
- Conduct site visits of areas to determine level of hazards
- Review records of violations to find higher risk buildings
- Look at fire incident data
  - Different rates for different building/occupancy type
  - Review historical data on fire rates by building/occupancy type over time
- Interview experts in the field for their opinions

Public Education Programs (ways to identify risks or needs)

- Conduct interviews with community leaders to assess needs
- Review incident data to determine priority causes and high-risk audiences
- Review the demographics of the community to determine high-risk audiences and areas
- Work with subject matter experts to find out where high-risk audiences congregate, shop, and which groups influence them etc. (psychographics)

Plan Review Programs (ways to identify risks or needs)

- Review document records to find out which buildings have not received plan review for modern codes
- Review incident data to determine which buildings are experiencing high fire rates
- Interview subject matter experts to help prioritize plan review activities
  - Including examining model programs in other places

Fire Investigation Programs (ways to identify needs)

- Review incident data to see where fires are more prevalent
- Review national guides for best practice investigation processes (e.g. NFPA 921)
- Interview subject matter experts on best practices
- Review previous investigation reports for accuracy and cause determination percentages Emergency Response (ways to identify needs)
- Review incident data to see when and where incidents are more prevalent
- Review incident data to see what types of incidents are more prevalent and what level of care or response was needed
- Review incident data to determine what level of care or response was sent In all cases, formative research could involve reviewing best practices elsewhere, and create baseline measurements from which we can begin planning for specific goals and objectives.

PROCESS MEASUREMENT EXAMPLES

Code Compliance Programs

- Find out the percentage of fires in properties subject to inspection that were not listed in inspection files (an example of a process problem where some occupancies were missed)
- Find out the percentage of inspections for which time since the last inspection is greater than the department’s target cycle time.
- Find out the number of building systems and features, from a defined list, for which code enforcement inspections and hazard abatements were not completed.
- Numbers of inspections done – as a baseline – per inspector Public Education Programs
- Review the list of available and needed public education materials § Review the percentage of people reached in target population in previous efforts
- Determine the number of fire safety programs needed and track the ones delivered § Track the time spent on public education activities
- Track the relationship between high-risk areas, and educational outreach efforts focused on those areas and audiences

Plan Review Programs

- Determine the percentage of time turn-a-round goals for plan review – establish goals
- Determine the average time (days) from permit application to permit “decision” – establish goals
- Determine the number of plans reviewed Fire Investigation Programs
- Determine the number of fires investigated (by line personnel or designated staff)
- Determine the number of interviews conducted for fire investigations Emergency Response Programs
- Determine needed staffing based on incident volume, type, time, or area
- Compare the level of care or response needed to what was sent/offered
- Determine response times relative to established goals for fire, EMS and other incident types

IMPACT MEASUREMENT EXAMPLES

Code Compliance Programs

- Determine the percentage of code violations noted that were identified and corrected
- Determine the percentage of fires that were preventable or could have been mitigated by inspection or by the educational and motivational elements of inspection
- Determine the percentage of fires where there were pending, uncorrected violations present at the time of the fire – indicating a weakness in the compliance rates

Public Education Programs

- Measuring changes in participants’ safety knowledge, attitudes, and beliefs
- Measuring observed and documented changes in behavior (e.g., safe storage of flammables, planned and practiced escape plans) and safety devices used (e.g., smoke alarms) through surveys Plan Review Programs
- Measuring the changes in percentage of errors in plans reviewed
- Measuring the changes of observed reductions in code violations found during “acceptance” inspections – indicating quality of the plan review process

Fire Investigation Programs

- Measuring changes in the quality of investigation reports (leading to better outcomes of cause determination)
- Measuring the changes in the number of fires investigated by line personnel vs. full-time investigation staff (indicating better education programs for line personnel) Emergency Response Programs
- Measuring changes in staff time, fuel spent, and vehicle use for related incident response
- Measure changes/improvements to response times to incidents
- Measure quality improvements to emergency response procedures that are known to reduce risks for specific events/audiences (e.g. proactive response patterns for low acuity calls)
- Documenting changes in response programs such as accreditation, ISO ratings, etc.

OUTCOME MEASUREMENT EXAMPLES

Code Compliance Programs

- Measuring the total value of property losses in inspectable occupancies to fire in relation to assessed value (factored for inflation)
- Measuring the changes in the percentage of total fire losses occurring in inspectable occupancies (factored for inflation) § Determining the changes in fire deaths/100,000 residents in inspectable occupancies
- Documenting the number of structural fires/1,000 residents in inspectable occupancies Public Education Programs
- Measuring the changes in fire incidents for populations of special focus
- Documenting the changes in fire-related deaths in focused populations
- Measuring changes in fire-related injuries in focused populations
- Measuring changes in property damage costs in focused populations (factored for inflation)

Plan Review Programs

- Documenting changes in code violations of buildings properly reviewed when compared with others that were not
- Documenting changes in fire incidents in “reviewed” occupancies
  - This assumes a time frame between recent construction – and older properties that fall under a regular code compliance inspection cycle. In this context “reviewed” would be a building or portion that is approved for occupancy, and the time before it falls under a regular code compliance inspection cycle.

Fire Investigation Programs

- Documenting changes in percentage of fires where cause is determined
  - Assumes the outcome expected of investigation is to determine cause but quality control is important and can be affected by the quality (or lack thereof) of the investigation
- Changes in percentage of fire reports “cleared” without a determination of cause Emergency Response Programs
- Documenting changes in outcomes of incidents (e.g. improved cardiac arrest survival rates; improved efforts to confine fires to area of origin due to emergency response)
- Documenting savings due to emergency response changes (e.g. reduced trips to emergency room where it is more expensive to provide care)
- Documenting lives saved or injuries reduced due to emergency response programs
- Documenting property saved – instead of lost – due to effective emergency response programs.