On learning about methods
On methods; I recommend you identify a published research example and adapt it. And look carefully at the example's own bibliography too. The methods literature is broad. You will take ownership ofdiscovering your methods' background, select an informing literature and decide for yourself about method suitability etc.
Each of us is expected to delve into the literature on the different methods available and on adapting these findings to our own topic. The reasons for this are various but primarily because research methods for product design and scientific studies is a very broad area with huge variation and there is no way to teach methods without limiting or biasing your own research journey (see comments below on the overuse of surveys and questionnaires). Educationally I expect each researcher to identify, study and develop expertise in the methods they employ. By taking ownership of this you become authentically involved in the process of professional and scholarly research. You will discover the background to various methods, select the publications that inform your own research design, decide through experimentation what methods are suitable and feasible for your own research context.
Some comments by way of general orientation to a research exercise:
- Be curious and open to discovery
- Goal oriented analysis often yields interesting findings.
- Be open to identifying insights, deep insights, particularly "what I learnt".
- Look out for the unexpected, things that are puzzling, aberrations.
- Self-reporting can be hugely insightful, this can be researcher self-reporting or respondent/subject self-reporting.
- Don't get side-tracked by 'annoyances' when there is an elephant in the room.
- When making recommendations or advising on solutions
- Contrast with comparable experiences, goals, solutions from other industries.
- Contrast with completely incomparable solutions from completely different settings.
- Avoid 'selling' a new tech system as if it is a holy grail or nirvana solution. All tech systems have their failings, you may simply be changing the source of your pain, not taking it away.
- There is huge value in a quick-dirty prototype.
- Use paper sketches, paper prototypes, mock-ups, from low to medium to high fidelity.
- Design ideas (sketches, mock-ups, prototypes) are used for feedback, for learning.
- If a design proposal is quite narrow, if so then it must be very focused, well argued, and insightful.
- NO DANCING BEARS! For an explanation see Alan Cooper's book "The Inmates are Running the Asylum" (search link)
- Respect your research subjects' privacy-identity
- It's generally seen as good practice to anonymise your interviewees if putting information in the public domain unless it is absolutely necessary for some reason.
- I recommend redacting personal identities from reports and papers.
- Be aware of copyright and attribution
- You cannot post other people's copyright material online.
- If you 'copy-paste' content from someone else make sure you surround it with quotation marks and include a citation or attribution.
- Do please make posts on your websites that are your own writing.
- Be aware of and comply with copyright law and conventions for fair use, attribution etc.
On Surveys and Questionnaires
I am not particularly interested in student projects that use surveys or questionnaires as the main or even as supporting research methods.
Surveys and questionnaires are a tertiary research method and rely on the laws of large numbers and/or access to large representative population pools. Survey/questionnaire methods are explicitly closed-ended in that they imply limited sets of allowable data/responses. Unfortunately these methods are rarely accompanied with a necessary introspection by the researcher of prior assumptions, or reflection/statement of researcher's epistemological/ontological assumptions that may skew or predispose the design of data collection to produce implicit results. A research design may in fact generate (produce, reify) the very objects it seeks to reveal.
It is also rare that we see well justified and well designed questionnaires or surveys. In fact these methods require that that the researcher has already studied (through literature review) or actually conducted extensive primary empirical research and/or carried out medium scale studies before resorting to survey/questionnaire. The findings of prior studies provide the justifications for determining what, why and who to ask. There will be clear connections between foundational research findings and the very design of a survey/questionnaire instrument. The content and sequence of each question is there for a valid research reason. The method is an inherently closed style of questioning and is therefore a kind of 'forcing function'. It produces limited responses responses to focused questions and therefore carries the risk that you will only detect what you expect to find, the method is quite literally self-determining and often results in poor science.
The design of surveys and questionnaires must needs be subjected to scrutiny, refinement and quality checking in order to avoid problems ranging from avoiding leading questions through to establishing construct validity. For example, is it essential for the research that you capture the respondents gender? Why? Is this justified? If so how many categories are you providing? Male/Female? Does the respondent have to answer or can they opt out? If so how does this affect the data and your analysis of it?
Surveys and questionnaires have their uses, for example where the concepts are well defined, not subject to misunderstanding (ref. construct validity), and the questions being addressed can usefully be asked of a large sample population. Therein lies my last bugbear with survey/questionnaire, sample size. A survey with 8 respondents is technically useless unless the whole population is 8. Statements based on small sample sizes are likewise in general useless unless you are sampling small absolute populations. Plus, survey responses are often presented in ways that mislead us as to their significance, for example, "100% of survey respondents answered yes" creates a different impression from "8 survey respondents answered yes".
So when does a survey sample size reach statistical relevance? The answer to this question depends on the margin of error threshold you seek to pass. Notice when talking about populations we generally refer to the sample size, a subset of the population. The implication being that we don't expect to be able to contact every single member of a target population, merely a subset. So what sample size do we need to reach to make reasonable inferences about the entire population? The margin of error or confidence level desired are in fact functions of population size; see "survey population/sample size calculator" for further information.
It is also rare that we see well justified and well designed questionnaires or surveys. In fact these methods require that that the researcher has already studied (through literature review) or actually conducted extensive primary empirical research and/or carried out medium scale studies before resorting to survey/questionnaire. The findings of prior studies provide the justifications for determining what, why and who to ask. There will be clear connections between foundational research findings and the very design of a survey/questionnaire instrument. The content and sequence of each question is there for a valid research reason. The method is an inherently closed style of questioning and is therefore a kind of 'forcing function'. It produces limited responses responses to focused questions and therefore carries the risk that you will only detect what you expect to find, the method is quite literally self-determining and often results in poor science.
The design of surveys and questionnaires must needs be subjected to scrutiny, refinement and quality checking in order to avoid problems ranging from avoiding leading questions through to establishing construct validity. For example, is it essential for the research that you capture the respondents gender? Why? Is this justified? If so how many categories are you providing? Male/Female? Does the respondent have to answer or can they opt out? If so how does this affect the data and your analysis of it?
Surveys and questionnaires have their uses, for example where the concepts are well defined, not subject to misunderstanding (ref. construct validity), and the questions being addressed can usefully be asked of a large sample population. Therein lies my last bugbear with survey/questionnaire, sample size. A survey with 8 respondents is technically useless unless the whole population is 8. Statements based on small sample sizes are likewise in general useless unless you are sampling small absolute populations. Plus, survey responses are often presented in ways that mislead us as to their significance, for example, "100% of survey respondents answered yes" creates a different impression from "8 survey respondents answered yes".
So when does a survey sample size reach statistical relevance? The answer to this question depends on the margin of error threshold you seek to pass. Notice when talking about populations we generally refer to the sample size, a subset of the population. The implication being that we don't expect to be able to contact every single member of a target population, merely a subset. So what sample size do we need to reach to make reasonable inferences about the entire population? The margin of error or confidence level desired are in fact functions of population size; see "survey population/sample size calculator" for further information.
 |
| Survey population-sample size calculators |
