Validating and testing prototypes

Explain what validating a prototype means and how it differs from "your team liking it". Why is it not enough to just ask your friends if your idea is good?

Explain what a test user (tester) is and why your grandmother or your best friend are suspicious testers. Describe the profile of three good testers for a school notes app (who they are and why their opinion counts more than your family's).

Distinguish validating from verifying with an example from the crêpe stall: verifying is checking that the crêpe is made according to the recipe; validating is checking that people want to buy it. Explain why a product can be 'well made' and still not sell.

You are going to test the juice stall with $5$ classmates. Describe what you would give them to sample, what three things you would record for each person (what they choose, how long they take to decide, whether they would pay the price), and why it is better to observe what they do than just ask them if they like it.

Design a test for a paper-screen school app prototype: write a script of $5$ steps that you will ask the user to complete (checking the timetable, writing down a task...), what they must do on their own (without your help), and what you will record at each step. Explain why, during the test, the user "drives" while you remain quiet.

Your first validation came out 'too well': you showed the prototype to $8$ friends explaining it yourselves, and everyone said it was 'great'. When trying it on their own, $3$ classmates from another class couldn't even get started.

1. a) Identify the three mistakes of the first test (the sample, the prior explanation, questions pushing towards praise).
2. b) Redesign the test with the correct rules (the user drives, you keep quiet, tasks instead of opinions) and write the first two sentences you will say to the tester.

Explain what a metric is and why 'it seemed people liked it' is not a metric. Write one concrete metric for each project: the breakfast stall, the notes app and the book swap point.

Your app test yields $15$ feedback comments: two people request dark mode, one asks for Alexa integration, eight complain that the main button is unclear, and four offer no comments. Explain how you distinguish signal from noise (how many people request it, how much it hinders usability) and decide which two changes you would make first, defending the order.

The team tested its app with $20$ students and gathered $25$ comments: $9$ ask for dark mode, $6$ complain that they can't find the search, $4$ ask for exam reminders, $3$ ask for stickers and $3$ said nothing.

1. a) Order the changes by signal (how many people ask for it and how much it blocks) and decide which two go into the next version, defending the criterion.
2. b) Explain what you would do with the stickers comment and why 'pleasing everyone' is the recipe for never launching.

The juice stall ran two test rounds. Round 1: $40\ \%$ of those who came over bought, an average wait of $6$ minutes, a flavour rating of $6,5$. Round 2 (with a new recipe and two jugs): $55\ \%$ buying, a wait of $3$ minutes, flavour $8,2$.

1. a) Interpret the evolution: what changed, which metric confirms it and what you would say if someone on the team claimed that 'it's perfect now'.
2. b) Propose the next metric to improve (percentage who come back, sales per peak hour) and design how you would measure it at the fair.

Two rounds of testing for the breakfast stall yield the following data: version $1$: $3$ out of $10$ customers complete the purchase and rate the waiting time at $4$ out of $10$; version $2$ (with pre-orders and two tills): $8$ out of $10$ complete, and the wait is rated $8$.

1. a) Interpret the progress: what problem did version $1$ have, what changed, and which metric proves it?
2. b) Propose the next hypothesis to test (what would further improve the purchasing experience) and design the mini-test with its success metric.

Two teams validate the same kind of stall. Team A asks: 'would you like there to be a healthy breakfast stall?' ($90\ \%$ say yes). Team B sets up a minimal stall one break time and measures: $15$ purchases out of $60$ people who passed by.

1. a) Explain what each team really measured and why team A's 'yes' is worth less than team B's $15$ purchases (the difference between stated opinion and real behaviour).
2. b) Design your definitive validation for the fair: which experiment with real money you would run beforehand, with the success threshold written down in advance ('if fewer than $X$ out of every $Y$ buy, we change product').