Scientific testing and troubleshooting

A useful experiment answers a question that could change your design. Write the prediction before looking at the outcome. Preserve inconvenient results; otherwise the experiment describes selected successes rather than the actual system.

Design a fair comparison

Name the independent variable, measured response, fixed conditions and predeclared criterion. “Try several commands” becomes: “Compare distances for commands 200, 400 and 600 at a fixed 0.5 s pulse, three repetitions each, on one marked mat.” Those commands are examples; pilot actual conservative limits with the hardware guide first.

Use a pilot to choose a safe setup and practical measurement method. Keep pilot data labelled separately. Alternate conditions when battery state, operator practice or lighting could change over time. If you cannot control a factor, record it and narrow the conclusion.

A compact test card

• Question and hypothesis.
• Source revision, firmware, map and calibrated settings.
• Conditions, start/reset method and sample count.
• Changed factor and fixed factors.
• Measured fields, units and independent truth method.
• Acceptance criterion, timeout and abort conditions.
• Full outcome table including stopped/failed/missing runs.
• Analysis and retain/revert/next-test decision.

Use the notebook template and experiment report. A partially completed physical batch must state its actual count; do not fill remaining rows with invented results.

Development versus validation

Tune on development trials. Freeze the selected code/settings before held-out validation. A new route or start condition can test generalization within the claimed envelope. An out-of-envelope test is an exploratory robustness test, not an ordinary pass/fail against a promise you never made.

If validation exposes a defect, preserve that batch, repair under a new revision and begin a fresh validation batch. Do not silently exclude failures or combine old/new versions into one fraction. Compare versions on identical replay fixtures for diagnosis, then use new physical trials for a final claim.

Software, simulation, replay and hardware

Unit calculation: checks a small function against hand results. Simulation: tests behaviour under a mathematical model with synthetic inputs. Replay: repeats decisions on a saved input trace. Hardware trial: observes actual robot behaviour. Each is valuable; they answer different questions.

A simulated PID plot cannot establish real motor friction or battery behaviour. A device import check cannot establish safe stopping distance. A clean replay can reveal algorithm changes without reintroducing floor variation. Label the evidence type in filenames/plots and conclusions.

Diagnostic examples

Veering: first compare command signs, encoder signs and start angle with wheels raised. Then compare measured wheel travel using the same short pulse. Change one calibration at a time. Noisy threshold: collect labelled light/dark readings before changing a threshold; compare distributions and lighting. Wrong waypoint: inspect cell-to-world conversion and radians/degrees before retuning gains.

A cause is confirmed only when evidence distinguishes it from plausible alternatives. “It drifted because batteries were low” is a hypothesis until battery condition or supply evidence supports it. Do not bypass a guard to complete a trial.

Failure analysis

Record first divergence, last valid sample, stop reason, likely subsystem and evidence. Separate observed facts from causal guesses. Create a minimal synthetic replay if possible. Repair one defect and rerun its reproduction plus the small regression matrix, including normal cases. Keep a known-limitations list so future work starts from evidence.

Acceptance and uncertainty

Choose tolerances larger than meaningless measurement precision and compatible with robot clearance. A ±1 mm requirement measured by an ambiguous hand-placed ruler is not credible. Report mean, range, individual values and failures for small samples. Distinguish calibration bias, timing uncertainty, repeatability and model error; see the math primer.

A review that takes five minutes

Can another reader find the raw data, tested version and criterion? Can they reproduce one calculation? Do plots show units and conditions? Are failed runs retained? Does the conclusion stay within the tested envelope? If any answer is no, fix the evidence record before adding features.

Save a CSV and make your first plot

Create a text file with a header such as trial,distance_m,time_s,condition,outcome,source_revision. Each following line is one attempted trial. Keep missing measurements blank and explain them in outcome/notes. CSV is a data format, not a screenshot. Verify that your editor saved the intended filename rather than adding .txt.

Download plot-trials.py to student code/, then run:

python3 code/plot-trials.py data/trials.csv --x trial --y distance_m --output plots/distance.svg

Open the SVG in your browser and link it from the notebook. The tool plots individual numeric pairs, labels the column names/ranges and reports missing/invalid pairs. Put units in headers; add conditions, tested version and your prediction to the notebook caption. Do not combine different conditions in one unexplained plot. The printed mean/spread describes plotted pairs only; report every stopped/missing attempt separately. Raw CSV remains unchanged. For a comparison, create separate derived condition CSVs while retaining the original complete file, and use matched axis ranges when drawing a comparison figure.