“GOOD” RESEARCH

Research leads to knowledge and insights, which are important sources of
competitive advantage. Your insights differentiate you from your competitors
and establish the value of the design services that you provide.
No matter how you are defining research, it is always possible to categorize
it as “good” or “not-so-good.” Good research answers the questions it has
chosen to investigate. It is appropriate to the situation. If the design goal is to create a space that supports an organizational culture, that organizational culture
must be identified using credible techniques. Even good techniques can
be flawed if used incorrectly:
• Time utilization studies and behavior mapping uncover where people are,
but not why they’re there or if the place they’re in is working for them or
against them.
• Confidential survey questions, answered by well-meaning respondents,
will give you an idea of what people think they do, but people regularly
do not remember their own actions accurately. For example, if you want
to learn how people respond to particular colors, study participants must
see those colors and experience them in context; it’s not enough to just
describe them with words.
• It is great when people volunteer to participate in a research project, but
collecting information from too many volunteers is also a red flag that
something may be amiss. Volunteers sometimes have a hidden agenda
(e.g., they want to prove their employer is abusive) that they will pursue
relentlessly throughout the research process (e.g., perhaps sacrificing
truthful responses to particular survey questions).
• People analyzing the behaviors of groups from another culture need to
acknowledge their own cultural biases. Our individual knowledge and
prior experiences influence our subsequent thoughts and actions, even
about how others perceive the design.
When research is done well, researchers identify any personal factors
(theirs or clients’) that might have affected the approach that they took to
developing the research plan or that might have influenced their analysis of
the data.
The most successful design research studies people who are most like those
who will ultimately use a space. Clearly, using college freshmen to determine
how professionals will respond to a new workspace requires several leaps of
faith. Likewise, mothers giving birth/caregivers/friends and family all react
differently to a birthing suite—and all those reactions are different from those
of the single male architect designing the space. Extrapolating from the experiences
of North Americans in a particular sort of healthcare environment to
those of Middle Easterners in the same sort of environment is not prudent, as
discussed earlier.
In a good research report, all of the details of the research and analysis
tools are presented in clear, straightforward prose and diagrams. This makes
the research process transparent to observers. A good report also discusses
sources of information and appropriate design implications. This means that the
research work that is the foundation for the current project is documented, and
ways that the collected information can be applied in the future are enumerated.
Insights that add value and stand the test of time are based on “good”
information. Good information comes from investigations that are, in sciencespeak,
reliable and valid. Reliability and validity are related to the way that
the investigations are done and not the specific research tools applied. No
matter what the methodology, the reliability and validity of a study can be
determined.
Good information is drawn from valid studies. That means the tools
and analysis methods used are appropriate for the questions being asked.
Researchers define all different sorts of validity, but the core idea is the same
for each type—people are learning about what they think they’re studying in
sound studies—a questionnaire probing organizational culture really is finding
out about organizational culture, for example.
Several specific types of validity will be particularly important to you as
you read and conduct research: If a study involves observations in a real world
location, it has ecological validity—the situation investigated is so close to
everyday life so that you can be comfortable applying the information leaned.
The number of odd bits of data introduced into peoples’ responses by tests
in a laboratory, for example, would be minimal in this case. Internal validity
describes the logical consistency of the reasoning within a study (is a finding
logically consistent with the sort of people who participated in the study,
the ways that data were collected, etc.), while external validity is the extent
to which study findings can reasonably be applied in the real world. They
are generalizable. Content validity is the extent to which a study thoroughly
addresses the issues in question.
It is also important to consider:
• Changes in instruments/methodologies can influence results. If researchers
compare two sets of responses to a questionnaire, the survey questions
need to be identical, for example, or any differences seen might be attributable
to differences in the questions asked.
• Selection related issues may affect study outcomes. If researchers compare
answers/responses of two sets of people, those people need to be identical
on relevant criteria (e.g., national culture but not hair color)—results can
also be distorted if members of one group were more likely to quit a study
early or not comply with the conditions of the study. Study participants
should also be randomly assigned to different test conditions whenever
possible, so that the assignment process isn’t responsible for differences
seen. In design research this would mean assigning different sets of work
teams to different sorts of test environments by drawing their names out
of a hat, not having all accounting groups work in one sort of space for a
period of time, all marketing teams in another, etc. If it will be possible to
customize work areas for groups doing different sorts of work, the level at
which randomization must occur moves down a notch, to types of accounting
and marketing teams.
• Sometimes the Hawthorne effect is present and changes in participant
behaviors are found because they were involved in a study, this is difficult
to determine from the research reports of others but you may be able to
spot it in your own work
• Subjects behaving in ways the experimenter wants them to behave are
another way that information collected may be corrupted.
• Subjects may be concerned about being evaluated, so they respond in a
“desirable” way. In this situation the importance of anonymity (versus
confidentiality) of data collected come to the fore.
• Changes in the pre- and post-test or as a study progressed that might have
influenced the information collected need to be considered. A change
could be a society-wide influence, such as 9/11, or something more specific
to situation, such as the death of a company executive and resulting modifications
in standard operating procedures.
• Pretesting of survey instruments and other tools can result in modification
of behaviors by the people they were pretested on. Answering survey
questions on a particular topic can move it top-of-mind, even though it is
not generally consciously considered.
• Misunderstandings about findings can lead to incorrect ideas about causation.
For example, findings of correlation between factors cannot be used
to determine which factor caused the other. Also, when multiple factors in
an environment are modified, it is not possible to attribute any resulting
changes in behavior to a single one of those changes; the entire set must
be credited.
Reliability basically means repeatability or consistency. An investigation
or instrument is reliable if duplicating it or using it again would produce the
same results. Since studies don’t often get repeated, discussions of reliability
in the applied world quickly turn to the idea of concurrent validity. Concurrent
validity is seen when studies/instruments that, because of their similarity,
should get the same results, actually do. Concurrent validity is found when the
same thing (national culture or morale, as examples) is measured using several
different tools and the same classification results in both cases.
When a study is done that seems similar to other studies in terms of how it
was conducted, but it reaches different conclusions from those of other studies,
it deserves special scrutiny. The conclusions drawn might be entirely appropriate—
methods advance, the world changes—but those different results might
have arisen because of an error on the part of the researchers. Regularly,
newer studies find and correct issues in the ways that earlier studies were
conducted. It is also true, however, that newer studies might define problems
differently than previous ones, which leads to differences in findings. The more recent researchers may also be studying a slightly different population
than previous researchers or using a different sort of research methodology,
for example—which might be more or less appropriate and applicable to any
design effort. Good practice suggests design researchers should remain openminded
to changing how they think about objects, spaces, and places, but be
critical as well.
Knowing about reliability and validity will make you a more informed
reader of study critiques and give you a mental framework to structure your
own reviews of investigations that you do yourself. When you’re reviewing
research that other people have done, ask yourself if the study seems appropriate
based on what you know about the field and if reasonable conclusions are
being drawn from the work done.
There are some additional general rules to use when you’re deciding if a
study you are referencing is strong or weak. A thorough discussion of the topic
of interest is a good sign, and so is a head-on discussion of other studies or
common professional practices that are inconsistent with the study results. If
you can identify topics that should have been addressed but were not, it’s time
to resume the literature review.
More recent studies are more desirable than those that are older—but
sometimes, classic studies stand the test of time. When people repeat an earlier
study and get the same results as the original researchers, that’s often not
very interesting to people who edit journals, so those corroborating studies
often don’t get reported. Older studies shouldn’t be dismissed out of hand—
particularly if you can’t find more recent ones on the same topic. Also, research
topics go in and out of fashion—a subject, such as the effects of ceiling height
on behavior, will be hot for a few years, and then related research may die
off for a decade. More recent studies will reflect recently completed research
by the authors of the article you are reading, as well as their peers. They also
might utilize more recent data-collection tools—brain MRIs instead of participants’
impressions of their responses, for example.
Good science also recognizes that not all human actions and feelings are
logical and consistent—sometimes valid research findings may seem illogical.