Applying the Four Evaluation Criteria
At a school in Oaxaca, the pump that sends water up to the water tank breaks down. The principal buys a new one, but two weeks later it fails again. How can we tell whether a technical solution is really a good one? It isn't enough for it to "look good": we have to evaluate it with clear criteria. At the end of the page you will come back to this situation.
Why we evaluate
To evaluate is to check how well a technical system (a pump, an oven, a work process) works so we can improve it. It isn't for giving grades, but for deciding what to change.
A good evaluation uses the same criteria for all the options. That way we compare fairly and not according to how much we like one of them.
On this page we use four criteria: effectiveness, efficiency, reliability, and feasibility.
Effectiveness and efficiency
To measure efficiency we use a ratio: how much product comes out for each unit of resource. For example, pieces per kilogram of firewood, or liters per hour.
School example: a lab computer that opens the drawing program and lets you finish the assignment is effective. If it also does it quickly and uses little electricity, it is also efficient. An old computer can be effective (it does finish the assignment in the end) and not very efficient (it takes a long time and uses more power).
Match each evaluation criterion with the question that corresponds to it and explain the difference between effectiveness and efficiency with a school example, for…
Match each evaluation criterion with the question that corresponds to it and explain the difference between effectiveness and efficiency with a school example, for instance a computer in the lab. Criteria: effectiveness, efficiency, reliability, feasibility. Questions: does it meet the goal it was made for?; does it always work equally well, without failing?; is it possible to carry it out with the resources we have?; does it use the minimum resources needed?
A community bakery tests two firewood ovens. Oven A bakes 36 pieces with 6 kg of firewood; oven B bakes 54 pieces with the same 6 kg. Calculate the pieces per kilogram…
A community bakery tests two firewood ovens. Oven A bakes 36 pieces with 6 kg of firewood; oven B bakes 54 pieces with the same 6 kg. Calculate the pieces per kilogram of each oven, decide which one is more fuel-efficient and explain why an oven can be more efficient but not necessarily more effective (think about how the pieces come out).
Reliability
It is measured by counting failures per period: failures per year, or how many days pass between failures. Fewer failures in the same amount of time means greater reliability.
Two water pumps supply a school's rooftop tank. Pump X fails on average once every 30 days; pump Y, once every 90 days. Estimate how many failures each one would have in…
Two water pumps supply a school's rooftop tank. Pump X fails on average once every 30 days; pump Y, once every 90 days. Estimate how many failures each one would have in 365 days and decide which one is more reliable, explaining the criterion you used.
Feasibility
Here we ask: is there enough money? Are there materials nearby? Is there someone who knows how to install it and maintain it? Is there enough electricity or rain in that community?
A system can be very efficient and reliable, but not feasible. An industrial oven is very efficient, but if it costs $\$200,000$ and the bakery only has $\$20,000$, it can't be done.
A school in a community without piped water must choose between two systems. Option 1: a rooftop tank with an electric pump that costs $\$4,800.00…
A school in a community without piped water must choose between two systems. Option 1: a rooftop tank with an electric pump that costs $\$4,800.00$ and spends about $\$180.00$ of electricity per month; it delivers water whenever there is electricity, but the community has frequent outages. Option 2: a ferrocement cistern with rainwater harvesting that costs $\$9,600.00$, spends almost nothing per month, but depends on rainfall and there are 3 very dry months. The available budget is $\$10,000.00$. Evaluate both options with the four criteria (effectiveness, efficiency, reliability and feasibility) and justify your recommendation.
Deciding with the four criteria
To compare options we build a table and check each criterion. Then we decide what matters most in that context.
| Criterion | Option 1 | Option 2 |
|---|---|---|
| Effectiveness | Does it meet the goal? | Does it meet the goal? |
| Efficiency | How much does it use? | How much does it use? |
| Reliability | How many failures? | How many failures? |
| Feasibility | Are the budget and conditions enough? | Are the budget and conditions enough? |
A school in a community without piped water must choose between two systems. Option 1: a rooftop tank with an electric pump that costs $\$4,800.00…
A school in a community without piped water must choose between two systems. Option 1: a rooftop tank with an electric pump that costs $\$4,800.00$ and spends about $\$180.00$ of electricity per month; it delivers water whenever there is electricity, but the community has frequent outages. Option 2: a ferrocement cistern with rainwater harvesting that costs $\$9,600.00$, spends almost nothing per month, but depends on rainfall and there are 3 very dry months. The available budget is $\$10,000.00$. Evaluate both options with the four criteria (effectiveness, efficiency, reliability and feasibility) and justify your recommendation.
Go back to the beginning: the pump at the school in Oaxaca failed again. With the four criteria we see that the new pump was effective (it raised water), but it was not reliable (it failed quickly). We also needed to check whether the cause was in the pump or in the power outages, and improve the system.