Technical research and experimentation

Capability level: Established — delivered in prior roles and independent projects.

Deciding whether to adopt a technical approach is itself a research question. This capability reviews the available technical evidence, compares implementation options, tests feasibility, and documents the tradeoffs so an engineering or product decision rests on something.

Typical work

• Comparisons of candidate approaches against the same data or workload
• Feasibility and performance testing before a build commitment
• Reviews of technical literature, tooling, and vendor claims
• Written tradeoff documentation for an engineering or product decision

Evaluate the evaluation first

A comparison is only as good as the measure it is scored on, and a measure that cannot separate the options will report a tie regardless of how differently they behave. Choosing a recommender by evaluating the evaluation is a worked example: three recommendation approaches were run on one dataset, and the finding was that the metric, not the model, was the thing that needed fixing.

The general form of that argument is set out in Evaluation.

Possible pilot outputs

A pilot may produce a technical assessment, a reproducible benchmark, annotated analysis code, or a memo recommending against a build. A negative result is a valid output and is reported as one.

Limits

This is engineering and product research rather than academic study. Findings describe what held on the systems and data actually tested, and the conditions that would change the answer are stated with them, following Data.