C3.1 — Product analysis and evaluation
Key concepts
Evaluation is the test of whether a design actually works. Anyone can claim a product is good; evidence-based evaluation proves it. Product analysis and evaluation is what separates professional designers from enthusiastic amateurs.
In the IA, Criterion E is dedicated to evaluation. Students who evaluate systematically, with evidence, score well; students who evaluate by opinion score poorly.
The purpose of evaluation
Evaluation answers three questions:
- Did the product meet its specification? (technical test)
- Did the product meet user needs? (user test)
- What should be improved? (forward-looking)
Evaluation is not the end of design — it is the feedback that drives the next iteration.
Evaluation frameworks
Specification check — the cleanest evaluation method. For each specification point:
- State the requirement
- Describe how it was tested
- Present the result (measurement or user feedback)
- Judge: fully met, partially met, not met
- Justify the judgement
Because the specification was written before the design (B1.1), it provides a neutral standard.
SWOT analysis — identifies Strengths, Weaknesses, Opportunities, Threats. Good for strategic-level evaluation, less useful for product specifics.
Comparative analysis — measures the new product against competitors or prior versions. Tests whether the design is better than existing alternatives.
User evaluation — structured usability testing with target users. Task completion rates, time-on-task, error counts, satisfaction ratings, think-aloud transcripts.
Lifecycle considerations — assesses environmental and social impact across the product's life.
Evidence-based vs opinion-based evaluation
Opinion-based: "The grip is comfortable and the product works well."
Evidence-based: "95% of the 20 users tested rated handle comfort as 4 or 5 on a 5-point scale. Mean grip pressure measured at 12 N, within the 8–15 N specification range. Grip fatigue onset at 23 minutes (specification: > 15 minutes)."
The evidence-based statement can be questioned but not dismissed. The opinion-based statement cannot be audited.
Every evaluative claim in an IA should be backed by something verifiable: a measurement, a user quote, a photograph, a completion time, a rating.
Testing and measurement
Design useful evaluation methods:
- Quantitative — measurements, counts, ratings, percentages (easy to compare, hard to reveal nuance).
- Qualitative — quotes, observations, photos, descriptions (rich detail, hard to summarise).
- Mixed methods — both together. Best practice.
Decide in advance what would count as success. "User completes task in under 45 seconds" is testable. "User finds it easy" is not until "easy" is defined.
Cultural and contextual evaluation
Products exist in culture and context. Evaluation should consider:
- Does it fit the user's environment (home, workplace, climate)?
- Does it respect cultural norms (colour, symbol, gesture)?
- Does it serve the full diversity of intended users?
- Does it have unintended consequences beyond the immediate use?
Being honest
Strong IAs acknowledge failures alongside successes. A product that partially meets its specification, with clear explanation of what went wrong and what would be done differently, scores better than one that claims success without evidence.
Examiners reward honest critical reflection. They penalise overclaiming.
Case studies
Apple product reviews — technical publications (Anandtech, iFixit) tear products down and measure every specification: battery life, thermal performance, durability, repairability. Users learn to expect rigorous evaluation. Design teams iterate based on these reviews.
Which? and Consumer Reports — independent testing organisations evaluate products against user-relevant criteria, publish comparative analyses. Drive design improvement through evidence-based public accountability.
Crash testing (Euro NCAP) — automotive safety evaluation through standardised destructive testing. Results published as star ratings. Has demonstrably driven safety improvements across the industry.
Dyson internal testing — thousands of hours of life testing, dust simulation, drop testing. Products not released until internal evaluation demonstrates they meet specifications. Evaluation embedded in culture.
Glossary
- Evaluation — systematic assessment of whether a design meets its objectives.
- Specification check — evaluation by comparing the product against each specification point.
- SWOT analysis — evaluation tool identifying Strengths, Weaknesses, Opportunities, Threats.
- Comparative analysis — evaluation by comparison against existing products.
- Quantitative evaluation — evaluation using numerical measurements.
- Qualitative evaluation — evaluation using descriptive observations and quotes.
- Usability testing — structured user evaluation with target users performing realistic tasks.
- Critical reflection — honest acknowledgement of strengths, weaknesses, and learning.
Check your understanding
1. Why is evidence-based evaluation more valuable than opinion-based evaluation?
Opinion is subjective, unreflectable, and carries no weight with examiners or critics. Evidence uses measurable data and documented observations that can be audited by a third party. "Users completed the task in under 45 seconds on 9 of 10 trials" is verifiable; "it is easy to use" is not. Every evaluative statement in an IA should be backed by evidence someone else could check, because that is what examiners reward.
2. Describe a systematic approach to evaluating a prototype against a specification.
Take each specification point individually. State the requirement as originally written. Describe the test method used (user test, measurement, comparison). Present the result with data (percentages, counts, ratings, quotes). Judge whether the requirement was fully met, partially met, or not met, with justification. Finally, summarise overall performance and identify the changes required in the next iteration. This produces a rigorous, auditable evaluation that an examiner can follow.
3. Distinguish between quantitative and qualitative evaluation, and explain why strong evaluation uses both.
Quantitative evaluation uses numerical data (counts, times, ratings, percentages) — easy to compare, summarise, and audit, but may miss context and nuance. Qualitative evaluation uses descriptive data (user quotes, photos, observations) — rich in detail, surfaces unexpected insights, but hard to summarise and compare. Strong evaluation uses both: quantitative data confirms the pattern; qualitative data explains why the pattern exists and what to do about it.
4. A student's IA evaluation claims "the product is a success and works really well". Rewrite this as an evidence-based statement.
"The product met 7 of 9 specification points. User testing with 12 target users showed 11/12 successfully completing the primary task within 60 seconds (specification: < 90 seconds). Grip comfort averaged 4.3/5 (specification: > 4.0). Two specification points were only partially met: durability (one failure in 500 cycles vs the 1,000-cycle target) and weight (320g vs 300g target). Users described the product as "natural to hold" and "clear to use". Main improvement needed: reinforce the hinge to meet the durability target." This statement is auditable, balanced, and rewards examiners with specific evidence.
Teacher's notes — additional examples and activities
ACCESS FM — the product analysis framework
A memorable mnemonic for structured product analysis:
- Aesthetics — visual appearance, form, colour, texture
- Cost — price, perceived value, cost to manufacture
- Customer — target user, demographics, need
- Environment — sustainability, disposal, context of use
- Size — dimensions, scale, portability
- Safety — hazards, standards, user protection
- Function — what it does, how well it does it
- Material & manufacture — what it is made of, how it is made
Without a framework, product analysis becomes a list of opinions. ACCESS FM forces breadth.
Objective vs subjective analysis
- Objective = fact (e.g. "Product X costs £49.99, weighs 340g, is 180mm tall")
- Subjective = opinion (e.g. "I like the colour", "It feels premium")
Strong analysis uses both. Facts anchor the argument; opinions capture user experience. Examiners reward students who distinguish the two clearly.
SWOT analysis
A complementary framework:
- Strengths — advantages over competitors
- Weaknesses — disadvantages
- Opportunities — room for improvement or innovation
- Threats — external risks (competitors, regulation, user trends)
ACCESS FM findings can be sorted into SWOT to identify design opportunities.
Stakeholders and data types
Different stakeholders provide different data:
| Stakeholder | Data type |
|---|---|
| User | Experience, pain points, preferences |
| Client | Business requirements, brand alignment |
| Manufacturer | Process constraints, cost limits |
| Engineer | Technical feasibility, safety limits |
Reverse engineering — the practical product analysis task
Reverse engineering is understanding how a design was made by recreating it. Gives insight into: - The balance between form and function - Where and how to innovate - Materials used - Construction methods
Task sequence: 1. Measure the dimensions of the IA product (H × W × D) 2. Identify innovation opportunities (which component or process could improve?) 3. Recreate a rapid prototype physically or in CAD 4. Produce a 3-projection orthographic drawing with labelled components 5. Label manufacturing process and most suitable material for each component
Multi-analysis comparison
Analysis 1: ACCESS FM on your chosen IA product. Analysis 2: ACCESS FM on a more complex / premium version. Analysis 3: ACCESS FM on a simpler / more minimal version. Analysis 4: Venn comparison across all three. Which is most successful? Why? What patterns emerge?
Constructive discontent
The habit of looking at products critically even when they work — could this be better? Fuels innovation. Drives students to notice opportunities ordinary users accept without question.