# jzhao.xyz


.
\_.
/


# Interviews and Data Recording

Last updated December 25, 2021

• What is data? Anything that is useful for understanding a particular design, user needs, and user behaviour.
• What does it mean to collect user data? What techniques can be used, and how useful and reliable is the data that is collected?

Five Key Issues

1. Settings Goals. Goals will influence the nature of data gathering sessions, the data gathering techniques to be used, and the analysis to be performed.
2. Identifying Participants. Those who fit the profile of types of people from whom data can be gathered are called the study population. Types of sampling are as follows:
• Probability Sampling: simple random sampling or stratified sampling
• Nonprobability Sampling: convenience sampling or volunteer panels
• Convenience Sampling: sample includes those who were available rather than those specifically selected
1. Relationship with Participants: informed consent with a clear and professional relationship between participant and researcher (however, informed consent is generally not required when gathering requirements data for commercial company where a contract usually exist between collector and provider)

Triangulation: the investigation of a phenomenon from at least two different perspectives. This is mostly focused on verification and reliability of data rather than making up for the limitations of another type of methodology

1. Triangulation of data: data is drawn from different sources at different times/places/people
2. Investigator triangulation: different researchers (observers, interviewers, and so on) have been involved in collecting and interpreting the data
3. Triangulation of theories: use of different theoretical frameworks through which to view data
4. Methodological triangulation: employ different data gathering techniques

## # 3 Methods for Data Recording

1. Notes + Photographs -> least technical and most flexible way of recording data, but can be difficult to capture the right highlights.
2. Audio + Photographs -> less intrusive than video, allows observers to focus on the activity rather than on trying to capture every spoken word.
3. Video -> highest fidelity. Things to keep in mind: 1) whether to fix cam position or use a roving recorder 2) where to point the camera 3) understanding the imapct of the recording on the participants

## # Interviews

A conversation with a purpose

Good for exploring issues, learning more about tasks, and getting inside user’s head.

Criteria for good interview: - Before: - get questions right → should support goals, should be easy to understand/answer - be organized → check equipment, have necessary docs - run pilots - During: - structure time - give participants context - use props and visuals → easier to show than tell - Listen well throughout!

1. Open-ended/unstructured -> exploratory and similar to conversation. Can be time-consuming but can also produce rich insights
2. Semi-structured -> plans basic script with both closed and open questions but probes interviewee until no new relevants info is there
3. Structured -> predetermined questions like a questionnaire, study is standardized (same questions with each participant)
4. Group interviews -> 3-10 people selected to provide a representative sample of the target population. Useful for investigating shared issues rather than individual experiences

Asking questions like “Can you tell me a bit more about…” is called probing

### # Planning

When developing Interview Questions, keep in mind open questions are best suited where the goal of the session is exploratory; closed questions are best suited where the possible answers are known in advance. Break long or compound questions into separate questions

A lot of decisions to make:

• Choosing a framework
• Level of participation to adopt
• How to make a record of the data
• How to gain acceptance into the group being studied
• How to ensure that the study uses difference perspectives

Ethnography: the description of the customs of people and cultures. A distinguishing feature of ethnographic studies compared with other data gathering is that a situation is observed without imposing any a priori structure or framework upon it, and everything is viewed as “strange”.

Technique Good for Kind of Data Advantages Disadvantages
Interviews Exploring issues Mostly qualitative (some quantitative) Interviewer can guide, encourages contact between researchers and users Artificial environment might be intimidating, remove them from usual environment
Focus Groups Collecting multiple viewpoints Mostly qualitative (some quantitative) Highlight areas of agreement/conflict, encourages contact between researchers and users Possibility of dominant characters
Questionnaires Answering specific questions Quantitative and Qualitative Can reach many people with low resource requirements Design is key, response rates may be low
Direct observation in the field Understanding context of user activity Mostly qualitative Observational insights Very time-consuming, huge amounts of data
Direct observation in a controlled environment Captural detail of individuals Quantitative and qualitative User can focus on task without interruption Data may be of limited use due to artificial environment
Indirect observation Observing users in natural environment without distraction Quantitative (logging) and qualitative (diary) Can be long due to automative recording Large amounts of data implies need for tools to support analysis, participants may exaggerate memories

### # Running the interview

Before starting, make sure that the goals of the interview have been explained to the interviewee and that they are willing to proceed. Listen more than talk, repond with sympathy but without bias, and to appear to enjoy the interview.

1. Intro
• interviewer introduces themselves
• explain why you’re doing the interview
• reassure interviewee re: ethical issues
• ask interviewee if they mind being recorded
2. Warm-up session
• easy, nonthreatening questions
3. Main session
• questions presented in logical sequence
• probing questions at the end
• order may vary in semi-structured interview
4. Cooling-off period
• easy questions to defuse any tension
5. Closing session
• interviewer thanks interviewee
• switch off recorder or put notebook away

## # Observation

Users may be observed directly by the investigator as they perform their activities or indirectly through records of the activity that are studied afterward.

Observation can also result in a lot of data to sift through and can be complicated to do well than at firs appreciated. As such, a clearly stated goal is important to have focus for an observation session.

Example frameworks:

• The person: Who is using the technology at any particular time?
• The place: Where are they using it?
• The thing: What are they doing with it?

3 common approaches

1. Simple observation: user is given a task, the evaluator just watches. This gives no insight into users’ decision process
2. Think aloud: subjects asked to say what they are thinking/doing. However, its hard to talk while concentrating and thinking may alter the way people naturally perform the task.
3. Co-discovery learning: two people work together on a task and normal conversation is monitored.

### # Degree of Participation

1. Passive Observer: observer who adopts an approach at the outsider end; does not take part in the study environment at all
2. Participant Observer: adopts an approach at the insider end; becomes a member of the group being studied

### # Coding Sheet

A data recording instrument in which a list of itemized coding options are structured

This standardizes observation practices which makes it more objective.

Consider:

• evaluation goals (break it all down!)
• stage of design
• observation method/types of data
• what would potentially be an interesting finding from this particular style?
• e.g. for think-aloud, it might be good to record action vs spoken comments

## # Questionnaire

Survey vs Questionnaire: the questionnaire is a part of the survey. The questionnaire is just the concrete things you’re asking.

Pros

• cheap
• does not require presence of evaluator
• many results can be quantified

Cons

• preparation is “expensive” → need to design questions well
• can have low response rate or low quality response
• difficult to do in-depth “probing”

A questionnaire is good when motivation is high enough without anyone else present. If persuasion is needed, a structured interview is probably better

### # Designing a Questionnaire

Keep in mind

• what info is sought?
• how would you analyze results?
• what audience do you want to reach?
• what will you do with your analysis?

Don’t use vague questions, pilot the questionnaire before testing.

Questions

• should not be transferable to other interfaces, can’t be interpreted in different ways depending on judgment (i.e. domain specific and clear wording)
• avoid double-barreled questions and leading questions
• to de-bias: neutral language, can have random order of questions for different participants
• for validity
• use previously validated questionnaires
• triangulation (ask multiple questions about the same matter)
• piloting

#### # Types of questions**

• Open-ended (hard to analyze rigorously)
• Closed (easily analyzed but can be hard to interpret if not well-designed)
• checkboxes and ranges
• range of answers to demo questions is predictable: offer a predefined list
• interval doesn’t have to be equal in all cases, depends on what you want to know
• mention how many boxes to check, be consistent with ascending/descending order
• Likert and semantic differential scale
• used for measuring opinions, attitudes, and beliefs
• widely used for evaluating user satisfaction
• Likert: a five, seven, or nine-point agreement scale used to measure respondents’ agreement with various statements
• semantic differential scale: rely on choosing pairs of adjectives to explore a range of bipolar attitudes about particular opinions
• ranked (closed)
• respondent places ordering on items in a list
• useful to indicate preferences
• forced choice
• multi-choice (closed)
• offered choice of explicit responses

Checklist

• think about ordering of questions → impact can be influenced by order
• consider if different versions are needed for different populations
• provide clear instructions on how to complete questionnaire
• eg. if answers can be saved and completed later
• think about length → avoid questions that don’t address study goals
• consider allowing respondents to opt out at different stages especially if long → better to have some than none
• think about layout and pacing

## # Data and Analysis

• Subjective: what you were told what happened
• Objective: what you captured using your senses
• Quantitative: data in the form of numbers or data that can be easily translated into numbers. Focuses on ascertain magnitude, amount, or size of something
• Qualitative: data in the form of words and images. Focuses on nature of something (themes, patterns, and stories)

Note that quantitative data is not always objective! Subjectivity can come from participants in how they express opinions or from investigators during the data capturing/interpreting/analysis process.

Similarly, it is unfair to try to quantize all qualitative data. This needs justification. Also be wary of translating small populatino sizes into percentages.

### # What to focus on

1. What are the most important needs/tasks to support?
2. What are the repeated patterns?
3. Key issues/areas that could be improved
4. What surprised you?
5. What is essential/nonessential in implementation

### # Steps

1. Initial reactions or observations (identify patterns, simple numerical analysis like averages, ratios, percentages)
2. Data cleansing (checking for erroneous entries and anomalies)
3. Analysis

### # Qualitative Analysis

#### # Thematic Analysis

Themes are a small number of high-level patterns that answer your evaluation questions.

Going from codes (descriptive labels) to categories (grouping imposed on codes) to themes (interpretive patterns). Deductive analysis is just the inverse (starting at themes and arriving at codes)

Do an initial pass to check of internal consistency: make sure themes occur across several or all participants. Then, step back to see if an overarching narrative emerges from the themes. One can them remove themes or look into why there are conflicts.

One way of doing this is using affinity diagrams:

1. record each idea/observation/problem/etc on individual card or post-it notes
2. look for notes that seem to be related
3. sort notes into groups until all used
• sort and resort as necessary

#### # Categorizing Data

• scheme of data: code the data according to categories
• if analysis frame is chosen beforehand: deductive analysis
• if study is explanatory and it is important to let themes emerge from data: inductive analysis
• can then analyze with appropriate methods like counting averages # of problems or identifying recurring patterns

### # Critical Incident Analysis

Helps identify significant subsets of data for more detailed analysis.

This is not about summarizing all incidients, more like finding gold nuggets. Incidents need not be bad all the time, can be either desirable or undesirable.