Chapter 14
14.1
● Evaluation = collecting and analysing data about users’ or potential users’ experiences
when interacting with a design artefact
● Goal: improve the artefacts design
● Focuses on usability and user experience
● Evaluations helps check that a design is appropriate and acceptable for the target user
population
● The type of evaluation method depends on the goals of the evaluation and can occur in
a range of places
● Usually involve observing participants and measuring their performance during usability
tests, experiments or field studies in order to evaluate the design or design concept
● Modelling user’s behaviour = provides an approximation of what users might do when
interacting with an interface
● Analytics = a way of examining the performance of an already existing product
14.2
● Well-designed products sell therefore companies want to invest in evaluating the design
of products
● Enables problems to be fixed before the product goes on sale
● Different kinds of users may use the same software in different ways, therefore it is
important to evaluate a range of types of users
● Evaluations can be carried out on lo-fi prototypes or complete systems and one function
or a whole workflow
● Different types of evaluation will be needed depending on the type of product, the
prototype or design concept and the value of the evaluation to the designers, developers
and users
● Main criteria: will the users use it? Does it do what the users need and want it to do?
● Where evaluation takes place depends on what is being evaluated (ie. in a lab versus
in-the-wild studies)
● Living labs = a compromise between the artificial, controlled context of a lab and the
natural, uncontrolled nature of in-the-wild studies
○ They provide the setting of a particular type of environment such as the home, a
workplace or a gym while also giving the ability to control, measure and record
activities through embedding technology in them
● Formative evaluations = cover a broad range of design processes (development of
sketches -> tweaking and perfecting a nearly finished design)
● Summative evaluations = carried out to assess the success of a finished product
○ If product is being updated, can be used to find improvements and fix associated
usability problems
● Common for rapid iterations of product development embed evaluations into short cycles
of design, build, test (evaluate)
, 14.3 - Types of Evaluation
● Evaluations are classified into three broad categories:
○ Controlled settings directly involving users = users activities are controlled to test
hypotheses and measure or observe certain behaviours
■ Main methods: usability testing and experiments
■ Examples: usability labs and research labs
○ Natural settings involving users = There is little to no control of users’ activities to
determine how the product would be used in the real world
■ Main methods: field studies
■ Examples: online communities, products used in public places
○ Any setting not directly involving users = consultants and researchers critique,
predict and model aspects of the interface to identify the most obvious usability
problems.
■ Main methods: inspections, heuristics, walk-throughs, models and
analytics
● Pros and cons to each evaluation category
● The correct evaluation approach to use is determined by the goals of the project and on
how much control is needed to find out whether an interface or device meets those goals
and can be used effectively
● Controlled Setting Involving Users
○ Designed to control what users do, when they do it and for how long
○ Reduce outside influence and distractions
○ Usability Testing = involves collecting data using a combination of methods in a
controlled setting
■ ie. experiments that follow basic experimental design, observation,
interviews and questionnaires
■ Often conducted in labs and increasingly remotely via digital
communication
■ Primary goal: determine whether an interface is usable by the intended
user population to carry out the tasks for which it was designed
● Compare the number and kinds of errors that users make between
versions/recording time to complete tasks/record them on video
■ Observing users reactions to an interactive product has helped
developers reach an understanding of usability issues which would be
difficult for them to glean from reading reports or listening to presentations
■ Usability testing a fundamental, essential HCI process
■ Usability specifications = a summary of findings from usability testing that
enables developers to test future prototypes or versions of the product
against it
■ Experiments are typically conducted in research labs at universities or
commercial labs to test such hypotheses
■ Remove distractions to that they can reliably say that findings arising from
the experiment are due to the particular interface feature being measured
14.1
● Evaluation = collecting and analysing data about users’ or potential users’ experiences
when interacting with a design artefact
● Goal: improve the artefacts design
● Focuses on usability and user experience
● Evaluations helps check that a design is appropriate and acceptable for the target user
population
● The type of evaluation method depends on the goals of the evaluation and can occur in
a range of places
● Usually involve observing participants and measuring their performance during usability
tests, experiments or field studies in order to evaluate the design or design concept
● Modelling user’s behaviour = provides an approximation of what users might do when
interacting with an interface
● Analytics = a way of examining the performance of an already existing product
14.2
● Well-designed products sell therefore companies want to invest in evaluating the design
of products
● Enables problems to be fixed before the product goes on sale
● Different kinds of users may use the same software in different ways, therefore it is
important to evaluate a range of types of users
● Evaluations can be carried out on lo-fi prototypes or complete systems and one function
or a whole workflow
● Different types of evaluation will be needed depending on the type of product, the
prototype or design concept and the value of the evaluation to the designers, developers
and users
● Main criteria: will the users use it? Does it do what the users need and want it to do?
● Where evaluation takes place depends on what is being evaluated (ie. in a lab versus
in-the-wild studies)
● Living labs = a compromise between the artificial, controlled context of a lab and the
natural, uncontrolled nature of in-the-wild studies
○ They provide the setting of a particular type of environment such as the home, a
workplace or a gym while also giving the ability to control, measure and record
activities through embedding technology in them
● Formative evaluations = cover a broad range of design processes (development of
sketches -> tweaking and perfecting a nearly finished design)
● Summative evaluations = carried out to assess the success of a finished product
○ If product is being updated, can be used to find improvements and fix associated
usability problems
● Common for rapid iterations of product development embed evaluations into short cycles
of design, build, test (evaluate)
, 14.3 - Types of Evaluation
● Evaluations are classified into three broad categories:
○ Controlled settings directly involving users = users activities are controlled to test
hypotheses and measure or observe certain behaviours
■ Main methods: usability testing and experiments
■ Examples: usability labs and research labs
○ Natural settings involving users = There is little to no control of users’ activities to
determine how the product would be used in the real world
■ Main methods: field studies
■ Examples: online communities, products used in public places
○ Any setting not directly involving users = consultants and researchers critique,
predict and model aspects of the interface to identify the most obvious usability
problems.
■ Main methods: inspections, heuristics, walk-throughs, models and
analytics
● Pros and cons to each evaluation category
● The correct evaluation approach to use is determined by the goals of the project and on
how much control is needed to find out whether an interface or device meets those goals
and can be used effectively
● Controlled Setting Involving Users
○ Designed to control what users do, when they do it and for how long
○ Reduce outside influence and distractions
○ Usability Testing = involves collecting data using a combination of methods in a
controlled setting
■ ie. experiments that follow basic experimental design, observation,
interviews and questionnaires
■ Often conducted in labs and increasingly remotely via digital
communication
■ Primary goal: determine whether an interface is usable by the intended
user population to carry out the tasks for which it was designed
● Compare the number and kinds of errors that users make between
versions/recording time to complete tasks/record them on video
■ Observing users reactions to an interactive product has helped
developers reach an understanding of usability issues which would be
difficult for them to glean from reading reports or listening to presentations
■ Usability testing a fundamental, essential HCI process
■ Usability specifications = a summary of findings from usability testing that
enables developers to test future prototypes or versions of the product
against it
■ Experiments are typically conducted in research labs at universities or
commercial labs to test such hypotheses
■ Remove distractions to that they can reliably say that findings arising from
the experiment are due to the particular interface feature being measured
