Assess a research engineer by how they turn a research question into reliable experiments and interpretable evidence. A publication, benchmark result, or familiar lab name does not establish the person’s individual engineering contribution.
Clarify whether the role emphasizes implementing ideas, building experimental infrastructure, reproducing results, or developing new methods.
Follow one experiment
Ask the candidate to choose work they are authorized to discuss. Have them explain the question, implementation, comparison, and conclusion.
| Area | Question |
|---|---|
| Contribution | Which code, infrastructure, or experimental decisions were yours? |
| Baseline | What did you compare against, and why was it appropriate? |
| Controls | What stayed constant between comparisons? |
| Reproducibility | What would another person need to rerun the work? |
| Interpretation | What did the result support, and what did it not establish? |
| Failure | Which experiment or assumption did not work? |
Avoid asking for private datasets, unpublished confidential results, or restricted code.
Inspect the experimental system
Discuss configuration, data versions, dependencies, logs, and how results were associated with the correct run. Ask how the candidate detected invalid experiments.
A research prototype can appropriately differ from production software. The relevant question is whether its engineering supports the conclusions being drawn.
Use a fictional result critique
Provide a small synthetic comparison in which an apparent improvement also changes the dataset or computational budget. Ask the candidate to identify the ambiguity and propose a cleaner comparison.
Then ask what they would prioritize under a limited experiment budget. Look for the experiment that most reduces uncertainty, not simply the most elaborate plan.
Separate engineering from scientific claims
A candidate may have implemented a method brilliantly without originating its central idea. Another may have shaped the method but relied on collaborators for infrastructure.
Both can be valuable. Match the evidence to the actual mandate rather than compressing all contributions into “did research.”
Record uncertainty honestly
A strong review describes the supported capabilities and the remaining gaps. Do not infer general scientific judgment from one successful result or reject someone for an experiment that failed for well-understood reasons.
Use the research scientist versus AI engineer comparison and inference engineering assessment. Discuss a research engineering search with Refery.