Task config

The eval_config defines what clinicians see and fill in. Key fields:

• purpose: evaluate (grade a model output), label (categorise / annotate data), or create (produce gold answers, preferences, or ratings). Defaults to evaluate; it sets the reviewer workflow and the deliverable.
• instructions: your rubric, shown to clinicians at the top of every task — what to evaluate, the standard, what counts as an error, edge cases. Optional, but it’s the single biggest lever on answer quality and reviewer agreement. Use line breaks to separate points. (Templates ship with a starter rubric you can tune.)
• adjudicate: true holds any item where reviewers disagree for a senior reviewer to resolve, instead of shipping the majority vote. Recommended for judgment work; the judgment templates set it for you. Optional (default false).
• auto_deliver: by default a finished batch is held for a human sign-off before it’s released to you (status stays in_review until then). Set true for hands-off delivery the moment every item is done. Optional (default false).
• input: text (shows the context fields), image (each item needs an image URL), or audio / video (each item needs an audio / video URL; a clinician plays it, streamed straight from your storage).
• context: for text tasks, which data keys to show the clinician, in order.
• classes: the label set used by from_classes and structured fields.
• case_id_field: which item field ties a result back to your own record.
• field_order: the order clinicians are asked the fields, as a list of field names. Worth setting — configs are stored as JSON, whose key order is not preserved, so without it the form is ordered arbitrarily and a clinician can be asked for a rationale before the verdict it explains. Templates set it for you; names you leave out are asked last.
• Each field takes a label (the question a clinician reads — without one the field name is prettified, so correct_label reads as “Correct label”) and an optional hint.
• primary_field: which answer decides reviewer agreement, and so which items are held for adjudication. Defaults to the first required single / from_classes field.

fields is a map of what the clinician fills. Each has a type:

• single: choose one of options.
• from_classes: choose one of the project classes.
• structured: yes or no, plus which finding (from classes).
• scale: a rating from 1 to max.
• flag: a single checkbox.
• text: free-text notes (rows sets the box height for long-form).
• spans: highlight text in the model output and tag each span with one of options (text input only).

Any field can add required: true, visible_when: "field!=value", and hint: "..." (a one-line note shown under the field’s label to guide the clinician).