Exercise: Experiment with a research method

Exercise: Experiment with a research method

Materials:
Copies of the research method protocol template. Use the template to structure your experiment with one or more of the research protocols based on the cards from IDEO.

Choose at least one protocol, but even better if you try more than one, if you have the interest or time available.
Context:
Identify an object or context for field work. Possible examples include:

• Stairs
• Doorways
• Queuing in a café
• Anywhere with a queue
• Parcel drop services
• Anything with a touchscreen
• UCD directional maps online and on-campus
• Dublin Bus website 
• ATM machine 
• Train/LUAs ticketing 
• Supermarket Self-service checkout machines 
• Copier machines at the Copi-Print system 
• UCard Top-up 
• Printer
• Kindle or other E-reader 
• Brightspace (from student’s perspective) 
• Alarm System 
• Airport Self-service Checkin 
• Tesco’s Online Shopping website 
• UCD Library catalogue 

Time:
Schedule field study observations over several days.

Instructions:

1. Write your name and the research context on the form
2. Investigate and propose your own version of the procedure for the research method assigned to you. Base this procedure on readings or your own creative extrapolation of the description on the IDEO methods card.
3. Conduct a trial run of the procedure on yourself first, then involve another willing participant, and another...
4. Make hand-written notes, photos and recordings for later analysis.
5. Keep a record of evidence gathered.
6. Make an archive of data/records (paper sketches and notes, digital folder, website).
7. Present your findings and analysis at the next class.

Evaluation:

In class discussion of findings. 

Questions to ask:

1. Did you write the method protocol (recipe/steps)?

2. Did you identify a method reference (source literature/inspiration)?

3. Have provided a sample/example of data you gathered?

4. Did you try to analyse or interpret the data? (may need to apply some tool or technique)

5. What was learnt from it all? Did I learn anything? Can you decide if the method is feasible, practical, suitable for your project?

Research Protocols

Names and descriptions for the IDEO Method Cards (IDEO, 2003)
Ideo (2003) IDEO Method Cards: 51 Ways to Inspire Design. William Stout.

LEARN

Research Method	Description
Activity Analysis	A description of project relevant dynamics: actions, interactions, tasks, and objects of achieving goals.
Affinity Diagrams	Represent or diagram clustering of design elements with activities, goals, obstacles. Proximity, dependence and relationships
Anthropometric Analysis	Human factors or ergonomics to assess project relevant use factors.
Character Profiles	Personas. Archetypes of real people with real goals, lifestyle, behaviour, identities.
Cognitive Task Analysis	List, summary of all available sensory inputs decision points and actions.
Competitive Product Survey	Collect, compare, and conduct evaluations of extant and competitive products.
Cross-Cultural Comparisons	Personal accounts of differences in situations, behaviour and artefacts in different national or cultural settings.
Error Analysis	Capturing the things that actually go wrong in the project relevant setting.
Flow Analysis	The flow of information in existing and/or new system.
Historical Analysis	Identify trends and cycles of product use, customer behaviour, market, and practice. Relate to timeless goals
Long-Range Forecasts	Narratives of future scenarios complete with social and technological trends to predict behaviours.
Secondary Research	Summary analysis of existing sources and 3rd party data on project relevant areas.

LOOK

Research Method	Description
A Day in the Life	Catalogue a person’s whole day without necessarily focusing on project relevant aspects.
Behavioural Archaeology	Look at use, wear, the detailed arrangement or organisation of use objects in their use setting.
Behavioural Mapping	Map position, movement, and use of space over time.
Fly on the Wall	Observe in context without interfering.
Guided Tours	Ask the user to guide you through project relevant spaces and activities.
Personal Inventory	Ask the user to reflect on and describe the things they view as important or significant
Rapid Ethnography	Participate and experience it first-hand with the user for as long as possible.
Shadowing	Tag-along with people through the day.
Social Network Mapping	Notice the relationships between people, groups. Look for identity, profession, culture, and connections.
Still Photo Survey	Build up a visual record of key use and interaction moments over time.
Time-Lapse Video	A way of summarising activity over time, use of time, space, and location.

ASK

Research Method	Description
Camera Journal	A written and visual diary of project relevant circumstances and activities.
Card Sort	Organise cards spatially in ways that make sense. To expose mental models of device or system.
Cognitive Maps	Create a map of an existing or virtual space. The pathways they know and navigate, mental models.
Collage	Self created collage of arranged images. Used to help verbalise complex or unvocalised themes.
Conceptual Landscape	Sketch and juxtapose social and behavioural constructs from participants. People’s mental model.
Cultural Probes	A visual journal for participants to build up themselves. A self generated reflection. Gathered and compared across many participants.
Draw the Experience	Ask participants to visualise and draw the experience in their own way, with their own associations, order, relationships, theories.
Extreme User Interviews	Evaluate (ask) users at extreme ends of market (early adopters, power users, beginners) to highlight their issues.
Five Whys?	Ask “why?” in response to five consecutive answers. Expose/uncover deeper attitudes perceptions.
Foreign Correspondents	Elicit inputs from others (snowball sample) to build up varied cultural and environmental contexts.
Narration	Users perform tasks and achieve goals while describing aloud. Talk aloud protocol. Stream of consciousness.
Surveys & Questionnaires	Targeted questions to assess design, usage, interaction characterises and perceptions of users.
Unfocus Group	Gather a range of tools or materials and get diverse user group to create things relevant to the design or project.
Word-Concept Association	Users associate words with design. Cluster user perceptions to evaluate design features & concepts. Value and priority.

TRY

Research Method	Description
Behaviour Sampling	Snapshot people’s activities at different times. How do pervasive products intrude in your lives?
Be Your Customer	What is it like to purchase your product? Actual experience of searching, buying, consuming, disposing.
Bodystorming	Act out scenarios with many people, using space, place, sequence, queues, etc. Test performance in context.
Empathy Tools	Experience the range of users capability for involvement under real conditions.
Experience Prototype	Mock up a rough but workable prototype to simulate the experience of using the new product.
Informance	Act out scenarios observed in the field to interpret and analyse in the lab. Builds shared understanding and good for solution formation.
Paper Prototyping	Rapid paper based mock-ups that are manipulated to demonstrate functionality. To articulate design concepts with users.
Predict Next Year’s Headlines	Involve users in future design possibilities. Futuristic, wishing, unmet needs. Help separate what is needed now from what can wait.
Quick-and-Dirty Prototyping	Assemble a very rough mock-up of a new feature or product to help start and refine a design.
Role-Playing	Identify actors/stakeholders involved in design use, enact real activities in real or imagined context.
Scale Modelling	Simulation technique to test arrangements of space, place, context.
Scenarios	A character-rich story, to communicate (simulate) and test a plausible story in probable context.
Scenario Testing	Show depictions of possible future scenarios. Share reactions, refine concepts.
Try it Yourself	Like Microsoft’s famous ‘eating our own dog food’ being the first user.

General comments on doing field research

On learning about methods

On methods; I recommend you identify a published research example and adapt it. And look carefully at the example's own bibliography too. The methods literature is broad. You will take ownership ofdiscovering your methods' background, select an informing literature and decide for yourself about method suitability etc.

Each of us is expected to delve into the literature on the different methods available and on adapting these findings to our own topic. The reasons for this are various but primarily because research methods for product design and scientific studies is a very broad area with huge variation and there is no way to teach methods without limiting or biasing your own research journey (see comments below on the overuse of surveys and questionnaires). Educationally I expect each researcher to identify, study and develop expertise in the methods they employ. By taking ownership of this you become authentically involved in the process of professional and scholarly research. You will discover the background to various methods, select the publications that inform your own research design, decide through experimentation what methods are suitable and feasible for your own research context.

Some comments by way of general orientation to a research exercise:

• Be curious and open to discovery
• Goal oriented analysis often yields interesting findings.
• Be open to identifying insights, deep insights, particularly "what I learnt".
• Look out for the unexpected, things that are puzzling, aberrations. 
• Self-reporting can be hugely insightful, this can be researcher self-reporting or respondent/subject self-reporting.
• Don't get side-tracked by 'annoyances' when there is an elephant in the room.
• When making recommendations or advising on solutions
• Contrast with comparable experiences, goals, solutions from other industries.
• Contrast with completely incomparable solutions from completely different settings.
• Avoid 'selling' a new tech system as if it is a holy grail or nirvana solution. All tech systems have their failings, you may simply be changing the source of your pain, not taking it away.
• There is huge value in a quick-dirty prototype. 
• Use paper sketches, paper prototypes, mock-ups, from low to medium to high fidelity.
• Design ideas (sketches, mock-ups, prototypes) are used for feedback, for learning.
• If a design proposal is quite narrow, if so then it must be very focused, well argued, and insightful.
• NO DANCING BEARS! For an explanation see Alan Cooper's book "The Inmates are Running the Asylum" (search link)
• Respect your research subjects' privacy-identity
• It's generally seen as good practice to anonymise your interviewees if putting information in the public domain unless it is absolutely necessary for some reason.
• I recommend redacting personal identities from reports and papers.
• Be aware of copyright and attribution
• You cannot post other people's copyright material online.
• If you 'copy-paste' content from someone else make sure you surround it with quotation marks and include a citation or attribution.
• Do please make posts on your websites that are your own writing. 
• Be aware of and comply with copyright law and conventions for fair use, attribution etc.

On Surveys and Questionnaires

I am not particularly interested in student projects that use surveys or questionnaires as the main or even as supporting research methods. 

Surveys and questionnaires are a tertiary research method and rely on the laws of large numbers and/or access to large representative population pools. Survey/questionnaire methods are explicitly closed-ended in that they imply limited sets of allowable data/responses. Unfortunately these methods are rarely accompanied with a necessary introspection by the researcher of prior assumptions, or reflection/statement of researcher's epistemological/ontological assumptions that may skew or predispose the design of data collection to produce implicit results. A research design may in fact generate (produce, reify) the very objects it seeks to reveal.

It is also rare that we see well justified and well designed questionnaires or surveys. In fact these methods require that that the researcher has already studied (through literature review) or actually conducted extensive primary empirical research and/or carried out medium scale studies before resorting to survey/questionnaire. The findings of prior studies provide the justifications for determining what, why and who to ask. There will be clear connections between foundational research findings and the very design of a survey/questionnaire instrument. The content and sequence of each question is there for a valid research reason. The method is an inherently closed style of questioning and is therefore a kind of 'forcing function'. It produces limited responses responses to focused questions and therefore carries the risk that you will only detect what you expect to find, the method is quite literally self-determining and often results in poor science.

The design of surveys and questionnaires must needs be subjected to scrutiny, refinement and quality checking in order to avoid problems ranging from avoiding leading questions through to establishing construct validity. For example, is it essential for the research that you capture the respondents gender? Why? Is this justified? If so how many categories are you providing? Male/Female? Does the respondent have to answer or can they opt out? If so how does this affect the data and your analysis of it?

Surveys and questionnaires have their uses, for example where the concepts are well defined, not subject to misunderstanding (ref. construct validity), and the questions being addressed can usefully be asked of a large sample population. Therein lies my last bugbear with survey/questionnaire, sample size. A survey with 8 respondents is technically useless unless the whole population is 8. Statements based on small sample sizes are likewise in general useless unless you are sampling small absolute populations. Plus, survey responses are often presented in ways that mislead us as to their significance, for example, "100% of survey respondents answered yes" creates a different impression from "8 survey respondents answered yes".

So when does a survey sample size reach statistical relevance? The answer to this question depends on the margin of error threshold you seek to pass. Notice when talking about populations we generally refer to the sample size, a subset of the population. The implication being that we don't expect to be able to contact every single member of a target population, merely a subset. So what sample size do we need to reach to make reasonable inferences about the entire population? The margin of error or confidence level desired are in fact functions of population size; see "survey population/sample size calculator" for further information.

Survey population-sample size calculators

The Difference between Citing References and Field Data

References and citations should relate to the subject matter being written about. Field data and interviews etc. gathered from a field study site does not constitute reference material in the same way; in this case a quote from the data is simply data not a citation. Similarly survey results are only included where relevant to the arguments or analyses being made, the entire survey with responses should not be appended to the paper. Research instruments and the gathered results may, if desired, be included in an appendix at the end of the document.

Personal Reflection

1-page appendix item "Personal Reflection"

The aim of a personal reflection is to give the student an opportunity to relate a personal understanding of the course. To highlight not just the described learning outcomes but also draw attention to challenges and areas of difficulty. Think of it as a statement of what you determine to be the key learnings and contribution of the course. It can be critical, highlighting gaps etc. Ultimately it is a personal statement of your own (perhaps new or changed) perspective on the subject, new understandings, difficulties, and insights.

Grading criteria:

The Personal Reflection is authentic, critical, supported by evidence and descriptive, conveying your own personal learning insights.

• A single page, approximately 500 words.
• Is it original? Is it your own work? (this is a basic requirement)
• Are the insights and learning described authentic? Does it honestly communicate your personal learning on taking this class?
• Is it critical? Critique isn't a bad thing. It challenges your own and others, even the subject itself. Consider prior understandings, misunderstanding, new knowledge, or changes in understanding?
• Are statements supported with examples? For example, comments or reflections on the homework tasks, the project, themes and subject matter?
• Core concepts? At the very best the reflection offers a compelling account of the significance of some of the key ideas arising in the course.

On the subject of research writing, methods, data, analysis, assumptions...

On the subject of research writing, methods, data, analysis, assumptions... Some ideas for structuring your own research ideas, and useful tips about research design.

Working Title: Initially, phrase a research question as the title of the paper (you can change it later).
Abstract: Restate and expand on the research question in the abstract (you can change it later when you have analysed your findings).

Research Access: Make good use of your personal access to your contacts, projects or companies, past or present for providing data.

Gather data and working backwards:
What I mean by this is that you will almost certainly end up changing/revising the research question as you go along, and the abstract will need to be revised at the end. The working title and abstract written at the beginning was just a first stab at the paper.
A research question is a prerequisite and precursor to a research project. Having a question puts the focus on a few things; the kinds of data you might expect to gather which could be anything from interviews, observations, documentary/documents, to literature review/desk research etc. etc.
A (tentative) research question usually implies particular kinds data, so ask, what kind of data does the question imply? What constitutes evidence for the phenomena under investigation? What data will (or might) provide the kinds of evidence that could be used to make justifiable statements about the research context?
A clear idea of the kind of evidence and data sought leads us to consider the types of data capture methods (research methods) that can be used to produce data or the evidence sought. This in turn indicates a typical overall research model, research design and research approach. Sometimes a programme of research will involve many of these approaches. For example, literature review, a distinctive family of desk based text research, is generally a prerequisite for all other research activities; evident in an introduction or positioning section of a paper, or constitute the whole research project itself. In very general terms, the gamut of research designs or models includes - but is not limited to:

Essay: Purely theoretical, elaborating a conjecture, speculation, conceptual, philosophical argumentation, thought experiment, word games, word play, rhetoric, logicism, formalism, constructionism.
Review, meta-analysis of prior research, selective literature review, systematic literature review.
Empirical: descriptive, idiosyncratic, case-study, naturalistic observation, questionnaire survey.
Correlational-causal: studies adopting systems model views, inputs factors, outcomes, case-control study, observational study, survey, structural equation modelling.
Statistical meta-analytic: research derived from other research, findings based on aggregations of other research.
Semi-experimental: research styled on intervention in settings, field experimentation not amenable to laboratory control, quasi-experiment, trials, living labs.
Experimental: classical closed system model of scientific discovery, reproducible experiment, blind experimentation, random assignment experiments.

(partially derived from the Wikipedia article on Research Design):

These various research models involve or require differing philosophical commitments and assumptions or beliefs around the nature reality and the world (worldview, ontology), and of the nature of knowledge (epistemology is the theory and nature of knowledge; objective, subjective, social). Usually it is sufficient to simply acknowledge the stance adopted for the purpose of the research, be it: interpretive, critical, critical realism, naive realism, post-modern, deconstruction, construction, positivist, aesthetic, utilitarian, speculative, qualitative, quantitative.

Improve the draft:

Commence your paper with an introductory/positioning piece that incorporates a short selective review of relevant recent literature critically addressing the topic areas you are working with.

Provide a presentation to peers or colleagues if possible, talking through your ideas, the data, your analysis and findings. Doing so will inevitably help refine how well you communicate the story, your message, convincing evidence, the key points, highlight your contribution and findings.

Use a Template

Start using a scientific conference template for writing up. I recommend you write using a journal or conference template. Two examples in the IS field below...

This is an example from the ECIS 2015 conference...

(http://www.ecis2015.eu/participation/submissions.html)

This is a pointer to the live HICSS template in Word format or in LaTeX format on the HICSS conference website. See the "Author Instructions" section...

(https://hicss.hawaii.edu/authors/)

The HICSS conference style in LaTeX is provided/shared via Overleaf. Overleaf is an online writing/editing service that uses LaTeX, the defacto standard for most scientific research writing...  

Further reading

Links to various related conference templates below:
A LaTeX and a Word version of an ECIS Template - from Muenster 2015 - recommended!!
A LaTeX version of an ICIS Template - Ryan Schuetzler - a bit gnarly. A Word version of an ICIS Template - it's Word :-(