Exercises: Prototype validation and testing

Explain what validating a prototype means and how it differs from "your team liking it". Why is it not enough to just ask your friends if your idea is good?

You are going to test the juice stall with $5$ classmates. Describe what you would give them to sample, what three things you would record for each person (what they choose, how long they take to decide, whether they would pay the price), and why it is better to observe what they do than just ask them if they like it.

The figure shows the workflow loop of a team validating their prototype, but one of the stages is covered with a question mark.

1. a) Indicate which stage is missing in the loop (what the team does with what was learned in the test) and why the loop cannot cycle without it.
2. b) Explain what the centre of the loop represents and why it cannot be "the team's own opinion".

Explain what a test user (tester) is and why your grandmother or your best friend are suspicious testers. Describe the profile of three good testers for a school notes app (who they are and why their opinion counts more than your family's).

Distinguish validating from verifying with an example from the crêpe stall: verifying is checking that the crêpe is made according to the recipe; validating is checking that people want to buy it. Explain why a product can be 'well made' and still not sell.

Name three things you can note down in a test without asking anything (where the user hesitates, which button they press first, how long they take) and explain what advantage observing has over asking 'do you like it?'.

Explain what a metric is and why 'it seemed people liked it' is not a metric. Write one concrete metric for each project: the breakfast stall, the notes app and the book swap point.

Your team has tested the prototype and has a list of $12$ faults. Explain why fixing all $12$ at once is not advisable and in what sensible order to tackle them (first the ones that block use, then the ones that annoy a lot, last the details). Give an example of each type.

Design a test for a paper-screen school app prototype: write a script of $5$ steps that you will ask the user to complete (checking the timetable, writing down a task...), what they must do on their own (without your help), and what you will record at each step. Explain why, during the test, the user "drives" while you remain quiet.

In the first test of the crêpe stall, $3$ out of every $10$ testers ask for the savoury option, and the average rating for the sweet flavour is $7,2$ out of $10$; after changing the recipe, the average rises to $8,5$ and $5$ out of every $10$ request savoury crêpes.

1. a) Interpret the two metrics and explain what decisions you would make regarding the menu (pricing, prominence of savoury options, sweet recipe).
2. b) Propose a third useful metric for the next test (queueing time, repeat customers, accepted price) and justify it.

Your app test yields $15$ feedback comments: two people request dark mode, one asks for Alexa integration, eight complain that the main button is unclear, and four offer no comments. Explain how you distinguish signal from noise (how many people request it, how much it hinders usability) and decide which two changes you would make first, defending the order.

Design the usability test for the book swap point prototype: the $4$-step script you will ask of each user (finding the point, looking for a book, recording the handover), what they must do alone without help, what you will note down at each step and how many testers you will look for. Explain why with $5$ good testers most of the big problems are already detected.

In the breakfast stall test, $6$ out of $10$ testers complete the purchase in under $2$ minutes; $3$ ask about gluten-free options (there are none) and the average product rating is $8,1$ out of $10$.

1. a) Interpret the three metrics and decide the first two improvements, justifying the order.
2. b) Propose the hypothesis you would test in the next test ('if we add a gluten-free option, $X$ will happen') and the metric that would confirm it.

The team tested its app with $20$ students and gathered $25$ comments: $9$ ask for dark mode, $6$ complain that they can't find the search, $4$ ask for exam reminders, $3$ ask for stickers and $3$ said nothing.

1. a) Order the changes by signal (how many people ask for it and how much it blocks) and decide which two go into the next version, defending the criterion.
2. b) Explain what you would do with the stickers comment and why 'pleasing everyone' is the recipe for never launching.

Your first validation came out 'too well': you showed the prototype to $8$ friends explaining it yourselves, and everyone said it was 'great'. When trying it on their own, $3$ classmates from another class couldn't even get started.

1. a) Identify the three mistakes of the first test (the sample, the prior explanation, questions pushing towards praise).
2. b) Redesign the test with the correct rules (the user drives, you keep quiet, tasks instead of opinions) and write the first two sentences you will say to the tester.

The juice stall ran two test rounds. Round 1: $40\ \%$ of those who came over bought, an average wait of $6$ minutes, a flavour rating of $6,5$. Round 2 (with a new recipe and two jugs): $55\ \%$ buying, a wait of $3$ minutes, flavour $8,2$.

1. a) Interpret the evolution: what changed, which metric confirms it and what you would say if someone on the team claimed that 'it's perfect now'.
2. b) Propose the next metric to improve (percentage who come back, sales per peak hour) and design how you would measure it at the fair.

The team tested their study notes app with $10$ users: their $8$ best friends and $2$ family members. It received nothing but praise and zero complaints. Upon launch, students find it confusing.

1. a) Analyse why the test went "perfectly" and yet failed: what biases existed in the sample and the script.
2. b) Redesign the test: who you would select, what instructions you would give them, and what you would do to bring constructive criticism to light.

Two rounds of testing for the breakfast stall yield the following data: version $1$: $3$ out of $10$ customers complete the purchase and rate the waiting time at $4$ out of $10$; version $2$ (with pre-orders and two tills): $8$ out of $10$ complete, and the wait is rated $8$.

1. a) Interpret the progress: what problem did version $1$ have, what changed, and which metric proves it?
2. b) Propose the next hypothesis to test (what would further improve the purchasing experience) and design the mini-test with its success metric.

Validation is not just about the product: it is also about the team's process. Design a project self-assessment with five questions that each team member will answer individually (fair task distribution, meeting deadlines, quality of decisions, group atmosphere, personal learning). Explain how the answers will be shared and what you will do if a score for a question is very low.

The organisation app team launched it without validating: $6$ months of work, $200$ downloads in the first week… and $80\ \%$ uninstalled it by the second day. The reviews say 'it's incomprehensible' and 'I can't find anything'.

1. a) Reconstruct which validation would have detected the disaster in week $2$ (a test with paper screens, $5$ users, concrete tasks) and what it would have cost compared with $6$ months.
2. b) Design the rescue plan: which test you run now with the real app, with how many users, which three tasks you ask of them and which two urgent changes you expect to find before relaunching.

Two teams validate the same kind of stall. Team A asks: 'would you like there to be a healthy breakfast stall?' ($90\ \%$ say yes). Team B sets up a minimal stall one break time and measures: $15$ purchases out of $60$ people who passed by.

1. a) Explain what each team really measured and why team A's 'yes' is worth less than team B's $15$ purchases (the difference between stated opinion and real behaviour).
2. b) Design your definitive validation for the fair: which experiment with real money you would run beforehand, with the success threshold written down in advance ('if fewer than $X$ out of every $Y$ buy, we change product').

'Validating is for those who don't believe in their idea; great entrepreneurs follow their gut.'

1. a) Summarise the argument and detect what it confuses (the idea with its execution: instinct proposes, the user disposes).
2. b) Rebut it with the case of the team that 'knew' the red bracelets would be the best sellers and ended up giving them away while the blue ones sold out, and draw out a practical rule: which decisions can be left to instinct and which cannot.

Your cooperative launches v3 of the stall after two validations. The real fair, however, turns out worse than the tests: long queues, you run out of stock mid-morning and several customers leave without buying.

1. a) Explain why the tests didn't predict this (scale, rush hour, team stress: the laboratory is not the real world) and what an agile team learns from this 'failure' of the test.
2. b) Design the post-fair report: which five figures you will collect during the event (sales per hour, maximum queue, moment of selling out, complaints, what sold out first) and which three decisions you will take for the next event with those figures.

Design the complete validation system for your class's project for this school year: what is validated before building (the concept), what is validated with the prototype (the use), what is validated at launch (the real market) and what is validated afterwards (continuous improvement). For each stage state the method, the number of people involved, the success metric and the decision you will take if the result is bad. Defend why validating is an insurance and not a waste of time.