Search IconIcon to open search


Interviews and Data Recording

Last updated December 25, 2021

Five Key Issues

  1. Settings Goals. Goals will influence the nature of data gathering sessions, the data gathering techniques to be used, and the analysis to be performed.
  2. Identifying Participants. Those who fit the profile of types of people from whom data can be gathered are called the study population. Types of sampling are as follows:
  1. Relationship with Participants: informed consent with a clear and professional relationship between participant and researcher (however, informed consent is generally not required when gathering requirements data for commercial company where a contract usually exist between collector and provider)

Triangulation: the investigation of a phenomenon from at least two different perspectives. This is mostly focused on verification and reliability of data rather than making up for the limitations of another type of methodology

  1. Triangulation of data: data is drawn from different sources at different times/places/people
  2. Investigator triangulation: different researchers (observers, interviewers, and so on) have been involved in collecting and interpreting the data
  3. Triangulation of theories: use of different theoretical frameworks through which to view data
  4. Methodological triangulation: employ different data gathering techniques

# 3 Methods for Data Recording

  1. Notes + Photographs -> least technical and most flexible way of recording data, but can be difficult to capture the right highlights.
  2. Audio + Photographs -> less intrusive than video, allows observers to focus on the activity rather than on trying to capture every spoken word.
  3. Video -> highest fidelity. Things to keep in mind: 1) whether to fix cam position or use a roving recorder 2) where to point the camera 3) understanding the imapct of the recording on the participants

# Interviews

A conversation with a purpose

Good for exploring issues, learning more about tasks, and getting inside user’s head.

Criteria for good interview: - Before: - get questions right → should support goals, should be easy to understand/answer - be organized → check equipment, have necessary docs - run pilots - During: - structure time - give participants context - use props and visuals → easier to show than tell - Listen well throughout!

  1. Open-ended/unstructured -> exploratory and similar to conversation. Can be time-consuming but can also produce rich insights
  2. Semi-structured -> plans basic script with both closed and open questions but probes interviewee until no new relevants info is there
  3. Structured -> predetermined questions like a questionnaire, study is standardized (same questions with each participant)
  4. Group interviews -> 3-10 people selected to provide a representative sample of the target population. Useful for investigating shared issues rather than individual experiences

Asking questions like “Can you tell me a bit more about…” is called probing

# Planning

When developing Interview Questions, keep in mind open questions are best suited where the goal of the session is exploratory; closed questions are best suited where the possible answers are known in advance. Break long or compound questions into separate questions

A lot of decisions to make:

Ethnography: the description of the customs of people and cultures. A distinguishing feature of ethnographic studies compared with other data gathering is that a situation is observed without imposing any a priori structure or framework upon it, and everything is viewed as “strange”.

Technique Good for Kind of Data Advantages Disadvantages
Interviews Exploring issues Mostly qualitative (some quantitative) Interviewer can guide, encourages contact between researchers and users Artificial environment might be intimidating, remove them from usual environment
Focus Groups Collecting multiple viewpoints Mostly qualitative (some quantitative) Highlight areas of agreement/conflict, encourages contact between researchers and users Possibility of dominant characters
Questionnaires Answering specific questions Quantitative and Qualitative Can reach many people with low resource requirements Design is key, response rates may be low
Direct observation in the field Understanding context of user activity Mostly qualitative Observational insights Very time-consuming, huge amounts of data
Direct observation in a controlled environment Captural detail of individuals Quantitative and qualitative User can focus on task without interruption Data may be of limited use due to artificial environment
Indirect observation Observing users in natural environment without distraction Quantitative (logging) and qualitative (diary) Can be long due to automative recording Large amounts of data implies need for tools to support analysis, participants may exaggerate memories

# Running the interview

Before starting, make sure that the goals of the interview have been explained to the interviewee and that they are willing to proceed. Listen more than talk, repond with sympathy but without bias, and to appear to enjoy the interview.

  1. Intro
    • interviewer introduces themselves
    • explain why you’re doing the interview
    • reassure interviewee re: ethical issues
    • ask interviewee if they mind being recorded
  2. Warm-up session
    • easy, nonthreatening questions
  3. Main session
    • questions presented in logical sequence
    • probing questions at the end
    • order may vary in semi-structured interview
  4. Cooling-off period
    • easy questions to defuse any tension
  5. Closing session
    • interviewer thanks interviewee
    • switch off recorder or put notebook away

# Observation

Users may be observed directly by the investigator as they perform their activities or indirectly through records of the activity that are studied afterward.

Observation can also result in a lot of data to sift through and can be complicated to do well than at firs appreciated. As such, a clearly stated goal is important to have focus for an observation session.

Example frameworks:

3 common approaches

  1. Simple observation: user is given a task, the evaluator just watches. This gives no insight into users’ decision process
  2. Think aloud: subjects asked to say what they are thinking/doing. However, its hard to talk while concentrating and thinking may alter the way people naturally perform the task.
  3. Co-discovery learning: two people work together on a task and normal conversation is monitored.

# Degree of Participation

  1. Passive Observer: observer who adopts an approach at the outsider end; does not take part in the study environment at all
  2. Participant Observer: adopts an approach at the insider end; becomes a member of the group being studied

# Coding Sheet

A data recording instrument in which a list of itemized coding options are structured

This standardizes observation practices which makes it more objective.


# Questionnaire

Survey vs Questionnaire: the questionnaire is a part of the survey. The questionnaire is just the concrete things you’re asking.



A questionnaire is good when motivation is high enough without anyone else present. If persuasion is needed, a structured interview is probably better

# Designing a Questionnaire

Keep in mind

Don’t use vague questions, pilot the questionnaire before testing.


# Types of questions**


# Data and Analysis

Note that quantitative data is not always objective! Subjectivity can come from participants in how they express opinions or from investigators during the data capturing/interpreting/analysis process.

Similarly, it is unfair to try to quantize all qualitative data. This needs justification. Also be wary of translating small populatino sizes into percentages.

# What to focus on

  1. What are the most important needs/tasks to support?
  2. What are the repeated patterns?
  3. Key issues/areas that could be improved
  4. What surprised you?
  5. What is essential/nonessential in implementation

# Steps

  1. Initial reactions or observations (identify patterns, simple numerical analysis like averages, ratios, percentages)
  2. Data cleansing (checking for erroneous entries and anomalies)
  3. Analysis

# Qualitative Analysis

# Thematic Analysis

Themes are a small number of high-level patterns that answer your evaluation questions.

Going from codes (descriptive labels) to categories (grouping imposed on codes) to themes (interpretive patterns). Deductive analysis is just the inverse (starting at themes and arriving at codes)

Do an initial pass to check of internal consistency: make sure themes occur across several or all participants. Then, step back to see if an overarching narrative emerges from the themes. One can them remove themes or look into why there are conflicts.

One way of doing this is using affinity diagrams:

  1. record each idea/observation/problem/etc on individual card or post-it notes
  2. look for notes that seem to be related
  3. sort notes into groups until all used
    • sort and resort as necessary

# Categorizing Data

# Critical Incident Analysis

Helps identify significant subsets of data for more detailed analysis.

This is not about summarizing all incidients, more like finding gold nuggets. Incidents need not be bad all the time, can be either desirable or undesirable.

Interactive Graph