Thursday, 24 November 2016

General comments on doing field research

This is just a list of comments by way of general feedback on the research exercise:
  • Be curious and open to discovery
    • Goal oriented analysis often yields interesting findings.
    • Be open to identifying insights, deep insights, particularly "what I learnt".
    • Look out for the unexpected, things that are puzzling, aberrations. 
    • Self-reporting can be hugely insightful, this can be researcher self-reporting or respondent/subject self-reporting.
    • Don't get side-tracked by 'annoyances' when there is an elephant in the room.
  • When making recommendations or advising on solutions
    • Contrast with comparable experiences, goals, solutions from other industries.
    • Contrast with completely incomparable solutions from completely different settings.
    • Avoid 'selling' a new tech system as if it is a holy grail or nirvana solution. All tech systems have their failings, you may simply be changing the source of your pain, not taking it away.
    • There is huge value in a quick-dirty prototype. 
    • Use paper sketches, paper prototypes, mock-ups, from low to medium to high fidelity.
    • Design ideas (sketches, mock-ups, prototypes) are used for feedback, for learning.
    • If a design proposal is quite narrow, if so then it must be very focused, well argued, and insightful.
    • NO DANCING BEARS! For an explanation see Alan Cooper's book "The Inmates are Running the Asylum" (search link)
  • Respect your research subjects' privacy-identity
    • It's generally seen as good practice to anonymise your interviewees if putting information in the public domain unless it is absolutely necessary for some reason.
    • I recommend redacting personal identities from reports and papers.
  • Be aware of copyright and attribution
    • You cannot post other people's copyright material online.
    • If you 'copy-paste' content from someone else make sure you surround it with quotation marks and include a citation or attribution.
    • Do please make posts on your websites that are your own writing. 
    • Be aware of and comply with copyright law and conventions for fair use, attribution etc.

On Surveys and Questionnaires

It is rare to see a well justified and well designed questionnaire or survey.
Every single survey and questionnaire question should be there for a reason. For example, is it essential for the research that you capture the respondents gender? Why? Is your response justified? If so how many categories are you providing? Male/Female? Does the respondent have to answer? What about giving an opt out like Male/Female/No response? The closed style of questioning used for survey's is a kind of "forcing function" for the types of answers you are expecting, this carries the risk that you will only detect what you expect to find, the method is quite literally self-determining and in my opinion, usually poor science.

Surveys and questionnaires have their uses, for example where the constructs are well defined, not subject to misunderstanding, and the question being addressed can usefully be asked of a large sample population. Therein lies my last bugbear with survey/questionnaire, sample size. A survey with 8 respondents is technically useless unless the whole population is 8. Statements based on small sample sizes are likewise in general useless unless you are sampling small absolute populations. Plus, survey responses are often presented in ways that mislead us as to their significance, for example, "100% of survey respondents answered yes" creates a different impression from "8 survey respondents answered yes".

So when does a survey sample size reach statistical relevance? The answer to this question depends on the margin of error threshold you seek to pass. Notice when talking about populations we generally refer to the sample size, a subset of the population. The implication being that we don't expect to be able to contact every single member of a target population, merely a subset. So what sample size do we need to reach to make reasonable inferences about the entire population? The margin of error or confidence level desired are in fact functions of population size; see "survey population/sample size calculator" for further information.
Survey population-sample size calculators

The Difference between Citing References and Field Data

References and citations should relate to the subject matter being written about. Field data and interviews etc. gathered from a field study site does not constitute reference material in the same way; in this case a quote from the data is simply data not a citation. Similarly survey results are only included where relevant to the arguments or analyses being made, the entire survey with responses should not be appended to the paper. Research instruments and the gathered results may, if desired, be included in an appendix at the end of the document.