Science & Research: How to Test What You Think You Know

Why believe that a study technique works? A teacher recommends it, friends repeat it, and it sounds logical. Those are starting points. Research asks a further question: what evidence would distinguish a real benefit from a convincing story?

Learning goals and Feynman practice

By the end, explain a fair test, distinguish correlation from causation, identify selection and expectancy biases, and interpret a statistical result without claiming certainty. Read each section, close the notes, explain it to a beginner, and invent your own example. Mark the step you cannot explain, revisit it, and try again.

1. Four paths to knowledge

Path	Everyday example	What to check
Authority	A respected teacher recommends rereading.	Expertise and evidence behind the recommendation.
Repetition or tenacity	Everyone keeps saying it works.	Repetition can preserve an error.
A priori reasoning	Familiar material feels easier, so it must be learned.	Sound reasoning needs sound premises; intuition needs testing.
Scientific analysis	Compare delayed recall under defined study conditions.	Measurement, fair comparison, uncertainty, and replication.

Science resembles checking a recipe by cooking it under specified conditions. Agreement about how delicious it sounds cannot replace observing what happens. Authority and logic still help generate ideas; empirical testing makes those ideas accountable to observations.

Falsifiability means specifying observations that could count against a claim. “This technique improves average recall after a week” can be tested. “It always works, unless invisible forces prevent it” protects itself against every outcome.

Science can study consequences of ethical choices or people’s religious experiences. Evidence alone cannot decide what we ought to value, settle every metaphysical claim, or determine artistic worth.

2. Build a fair experiment

Suppose you test retrieval practice against rereading. Randomly assign volunteers to one method, give both groups the same material and study time, and measure recall a week later.

• Independent variable (IV): the assigned study method.
• Dependent variable (DV): the number of items correctly recalled.
• Operational definition: the exact procedure used to represent a concept such as learning.
• Control condition: a comparison that helps isolate the effect being investigated; it need not mean doing nothing.

Random assignment makes pre-existing differences less likely to systematically favour one condition. It does not guarantee identical groups. Random sampling addresses a different problem: whether participants represent the population.

Teach it back: Why would letting students choose their method weaken the test?

Self-check: explain the alternative cause

Motivation or prior ability might influence both method choice and recall. The groups could differ before studying. Random assignment helps break that connection; measuring only the final scores does not.

3. Correlation is a clue about causes

Ice-cream sales and swimming accidents may rise together because hot weather increases both. The association alone does not show that ice cream causes accidents. Other threats include reverse causation and differences in who enters each group.

A well-controlled randomized experiment offers strong causal evidence because the researcher changes a factor while reducing competing explanations. Manipulation alone is insufficient if other things change at the same time. Causal inference is also possible from carefully designed natural or quasi-experiments, but depends on explicit assumptions.

You cannot randomly assign someone’s age or childhood culture. Comparing age groups is usually observational; it does not automatically become a quasi-experiment. Quasi-experiments use a design feature, such as an external policy change, to support a causal comparison without random assignment.

4. What a null-hypothesis test says

For the study-method experiment, a null hypothesis might specify no average difference in delayed recall. A p-value describes how unusual a result at least as extreme as the observed one would be under that null model and its assumptions.

A small p-value can justify rejecting the null under a prespecified rule. It is not the probability that the null is true, and rejection does not prove the preferred explanation. Failure to reject does not establish no effect: the study may be too imprecise. Also examine effect sizes, confidence intervals, design quality, and replication. Statistical testing is one research tool, rather than the definition of science.

Teach it back: Can a tiny effect become statistically significant in a very large sample? Explain why practical importance needs a separate judgment.

5. Bias: when the comparison quietly changes

Problem	How it misleads	Useful protection
Self-selection	Volunteers or adherent participants differ from others.	Random assignment and careful recruitment; analyse assigned groups appropriately.
Unrepresentative sample	Results from one student population are generalized to everyone.	Diverse recruitment and explicit limits on generalization.
Demand characteristics	Participants guess the aim and adjust behaviour.	Neutral instructions and suitable masking.
Experimenter expectancy	Researchers unintentionally cue or rate expected behaviour.	Masked assessors, standardized procedures, and objective measures.
Artificial task	Laboratory behaviour may not transfer to everyday life.	Test transfer and compare findings across settings.

A double-blind design masks assignment from participants and the relevant researchers or assessors. Some interventions cannot hide their condition from participants; masking outcome assessors can still help. Blinding and random assignment solve different problems.

The Coronary Drug Project provides a striking warning. Its 1980 report found five-year mortality of 15.1% among good placebo adherers versus 28.3% among poor adherers. These were post-randomization subgroups, not randomly assigned levels of adherence. The contrast does not show that taking an inert pill caused longer survival. It shows how adherence comparisons can be confounded, even within a randomized trial. [1]

6. A short history of asking measurable questions

Wundt established the Leipzig psychology laboratory in 1879, helping institutionalize experimental psychology. Mental chronometry uses reaction times to investigate processing; Donders had used reaction-time subtraction earlier. If choosing between responses takes longer than making a simple response, the difference may inform a model of additional processing, provided its assumptions hold.

Behaviorism emphasized observable relationships between stimuli and responses. The cognitive revolution renewed interest in internal representations and processes inferred through behaviour and other measurements. An invisible process can be investigated scientifically when a model makes testable predictions.

7. Two examples of intuition meeting evidence

Disorder and littering: Keizer and colleagues’ 2008 field experiments examined whether visible norm violations encouraged other violations. The bicycle-flyer example compares littering in orderly and graffiti conditions. Treat it as evidence from specific settings, rather than proof of a universal theory about crime. Ask whether conditions, observation procedures, and opportunities to litter were comparable, and whether the result replicates. [2]

Changing test answers: Kruger and colleagues’ research challenges the blanket advice to keep your first answer. Their paper describes evidence that most answer changes are from wrong to right, and investigates why regrettable changes are especially memorable. This supports reconsidering an answer when you identify a reason; it does not support changing answers randomly or promise a fixed improvement ratio. [3]

8. Feynman workshop: design, explain, repair

Design a test of “phone notifications reduce learning.” Specify the IV, DV, comparison, assignment method, and one confound. Then explain the design aloud without using the words “scientific” or “objective.” Those labels cannot substitute for explaining the controls.

One possible design

Randomly assign participants to study the same material with standardized notifications enabled or disabled. Hold study time and equipment constant and measure delayed recall with masked scoring. Prior knowledge could affect scores; random assignment helps balance it on average, and a baseline measure can improve precision. The result concerns this notification procedure and task, not every kind of phone use.

Explain why a control group alone does not establish causation

A comparison group may differ in motivation, health, or prior skill. Causal interpretation depends on how groups were formed and what else differs, not merely on naming one group the control.

Explain why “not significant” does not mean “identical”

A noisy or small study can miss a meaningful difference. An uncertainty interval helps show which effect sizes remain compatible with the data under the model.

Retrieval and review

From memory, write the four paths to knowledge, define IV and DV using a new example, distinguish sampling from assignment, and explain one alternative cause of a correlation. Repeat tomorrow and in a week. Repair the weakest explanation before adding more terminology.

One-minute explanation: Research turns a plausible claim into a comparison that could reveal an error. We define what changes, measure what follows, reduce alternative explanations, and report uncertainty. A persuasive story becomes more useful when we know what evidence would make us revise it.

Sources and further reading

The supplied outline’s numbered citations did not include a bibliography. These references support the research examples; the worked study designs above are teaching examples.

1. Coronary Drug Project Research Group (1980). Influence of adherence to treatment and response of cholesterol on mortality.
2. Keizer, K., Lindenberg, S., and Steg, L. (2008). The spreading of disorder.
3. Kruger, J., Wirtz, D., and Miller, D. T. (2005). Counterfactual thinking and the first instinct fallacy.