1. Evaluation scope
EvalDock evaluates Agents on specific tasks and under specific operating conditions. Assessment starts with task requirements: check whether the actual delivery meets those requirements, then use available execution traces to explain the process. Tasks, configurations and evidence jointly define the scope of each conclusion.
Published reports use scoring rules appropriate to their tasks. The DSH plugin-configuration evaluation (Chinese) uses DSH configuration dimension scoring 2.0; the four-office-Agent evaluation uses file checks and scoring rules for five task suites. This page explains their shared task and evidence framework. Each report identifies the protocol versions it actually uses.
2. Tasks and operating conditions
Before evaluation, define what counts as completing a task and how each requirement will be verified. For file-generation tasks, for example, specify the output location, format, required content, and any required or prohibited tools.
| Item | Required record |
|---|---|
| Task and inputs | Task goals, input materials, delivery requirements and operating restrictions. |
| Configuration | The Agent and its version, plugin configuration, tools and operating environment. |
| Completion criteria | Checks and verification methods for each requirement, defined before execution. |
| Evidence scope | The inputs, outputs, artifacts, tool calls and environment records actually retained, with missing coverage identified. |
When execution traces are available, inspect the inputs received by tools, their actual returns, and how the Agent uses that information to proceed. When traces are unavailable, report only conclusions supported by inputs, outputs and artifacts, without inferring internal processes. Trace availability depends on what was actually collected, not whether the product is open source.
Observability: black box, gray box and white box
Record observability for each tested workflow. A product’s web interface, CLI, API or permission modes may expose different evidence, so the classification applies to the specific test configuration.
| Level | Collection boundary |
|---|---|
| Black box · visible results | Inputs and final outputs are recorded, with artifacts and externally measurable elapsed time retained as appropriate. No verifiable execution trace is available. |
| Gray box · partial process visibility | Some verifiable actions, tool activity or logs are retained alongside inputs and outputs. Gaps in key steps must be identified individually. |
| White box · traceable workflow | Tasks can be linked to key tool calls and their inputs and returns, execution order, errors and final artifacts, with coverage and blind spots documented. |
| Unconfirmed | Collection scope still requires checking. Classify the workflow once the covered steps have been confirmed. |
White box means that the tested workflow is traceable. It does not require an open-source product or imply complete transparency into the model’s internal reasoning. Page progress indicators and Agent self-reports cannot replace verifiable execution records. State collected evidence and publicly released evidence separately.
Select the test runner according to access and collection capabilities. All modes use shared result-acceptance rules, with process analysis reported separately. Observability levels do not earn extra result points. Comparisons require consistent tasks, operating conditions and scoring rules, plus confirmation that collection methods did not alter execution. Mark unavailable process metrics as missing.
3. Verification and scoring
Verifying delivered results
Compare the actual delivery with each task requirement, checking file location, format and content. For programming tasks, also use actual runtime output and verification records where available. A file’s existence proves only that it was generated; content correctness and support for citations require separate checks.
Scoring and aggregation in the current DSH report
The DSH plugin-configuration report combines model-generated dimension scores with delivery verification. Dimension scores describe performance on each dimension; delivery verification checks whether specific task requirements were met. The statistical rules are as follows.
| Item | Rule used in the report |
|---|---|
| Unit | A task instance. The same task under different configurations counts as a separate instance for each configuration. |
| Score source | deepseek-flash assigns raw scores from 0 to 5 using dimension-specific rules, task requirements, execution records and delivered artifacts. |
| Inclusion | The run ended normally, all required test conditions were available, scores for every applicable dimension were complete, and independent-answer requirements were met. |
| Dimension mean | Task scores within the same configuration and dimension receive equal weight. Raw means are shown to two decimal places. |
| Sample size n | The number of scores actually included for that configuration and dimension; this is also the denominator of its dimension mean. |
Read the report’s methodology and statistical rules (Chinese).
Scoring and aggregation in the office Agent report
Each of four products completed 15 tasks in each of five suites: 75 distinct tasks and 300 delivery results in total. Office-file and spreadsheet suites count passed tasks; PPT editing, PPT generation and document question answering use their respective task scoring rules. The report converts each metric to a 0–100 scale using its theoretical range.
The overall score weights the five suites equally at 20% each. For PPT generation, the converted content, design and coherence scores are averaged before inclusion in the overall score. Execution allows necessary human assistance, continuation and retries, and evaluates final delivery quality. Tasks were selected using categories, difficulty and other information; this is not a simple random sample. This evaluation did not include repeated-run experiments.
Read the office report’s sampling, scoring and verification notes; see the 100-point conversions and overall-score formula.
4. Interpreting and comparing results
Read scores together with their dimensions, sample sizes and conditions. First identify which tasks were tested and which criteria were used, then decide whether the findings apply to your own setting.
- Interpret each score according to its metric. Dimension means describe scoring performance; pass rates describe the proportion of tasks meeting acceptance criteria. Assess completion using the specific requirements and verification results.
- Compare differences under comparable conditions. Check task composition, sample size, operating conditions and scoring configuration first. Plugin effects require a controlled comparison under the same conditions; score differences across configurations alone do not establish causation.
- Match conclusions to the collected evidence. Trace analysis covers verifiable actions, tool returns and artifacts. The model’s internal reasoning is outside that evidence scope.
Each report identifies the evidence actually collected and released. The DSH report is based on per-task scores and execution traces; the office Agent report is based on task inputs, final answers or delivered files. Public pages provide configurations, statistical rules and case-verification summaries. Complete raw records are not publicly released.
When selecting a tool, prioritize cases with similar tasks, input materials and delivery requirements. After making improvements, retest with the same tasks, environment and scoring standards, and record the changes and their effects.
5. Protocol versions and report references
The version identified by each report governs its specific rules. The protocols below include those already used in published reports. The general input-output evaluation and execution-trace analysis protocols remain drafts and have not been used for published results. Detailed protocol pages are currently in Chinese.
- DSH configuration dimension scoring (Chinese)2.0 · Used in a published report
- The scoring rules used in the DSH plugin-configuration report. deepseek-flash assigns dimension scores from 0 to 5 based on task requirements, execution records and delivered artifacts.
- Office Agents · file delivery (Chinese)2026-09-24.1 · Used in a published report
- Uses adapted OfficeBench file-delivery tasks and counts passed tasks using the corresponding adapted checks.
- Office Agents · spreadsheet delivery (Chinese)2026-09-24.1 · Used in a published report
- Uses a SpreadsheetBench Verified subset, compares the target regions of delivered files with expected results, and counts passed tasks.
- Office Agents · PPT editing (Chinese)2026-09-24.1 · Used in a published report
- Uses a Microsoft PPT-Eval subset and the same automated checks and AI judge configuration for this evaluation to assess requested edits, reporting the mean score across 15 tasks.
- Office Agents · PPT generation (Chinese)2026-09-24.1 · Used in a published report
- Uses a PPTAgent / PPTEval subset. Content and design are aggregated from slide-level scores; coherence is scored per presentation. Each retains its 1–5 scale.
- Office Agents · document image question answering (Chinese)2026-09-24.1 · Used in a published report
- Uses a DocVQA subset and divides accumulated ANLS by 15 to report mean answer-text similarity.
- Input-output evaluation (Chinese)1.0-draft · Method draft
- Verify task completion through inputs, outputs and delivered artifacts under consistent tasks and operating conditions. Applicable when complete execution traces are unavailable.
- Execution-trace analysis (Chinese)1.0-draft · Method draft
- Alongside result verification, analyze execution using the tool calls and environment records actually collected. Applicability depends on available evidence, not whether a product is open source.
Report using scoring 2.0: DSH plugin-configuration evaluation (Chinese).