The full instrument used to score MediVerse against a general-purpose LLM comparator on six immersive-visualisation metrics, across two paediatric cohorts.
This 50-query instrument measures the six IVEM (Immersive Visualisation Evaluation Metrics) scores for MediVerse alongside a comparator system, on the same paediatric datasets and the same natural-language prompts. Each system runs every query three times so that Encoding Consistency can be observed. The full spreadsheet - with the yellow-cell scoring workflow, one worked example row, and an auto-computed Summary tab - is available on request; this page reproduces the query list and the metric definitions so results can be cited without shipping an Excel file.
A first-pass instrument. AI Encoding Accuracy and Visualisation Reasoning Score require aggregation across three or more expert raters for the values reported in the paper; the intent here is to fix the queries, thresholds, and score keys so different raters and different systems can be compared like-for-like.
100 acute lymphoblastic leukaemia patients from three CSVs:
PatientData.csv, PatientTreatments.csv, and
processedPatientData.csv. Fields cover phenotype (FAB L1/L2),
risk stratification, treatment protocol, initial WBC and blast counts,
CNS involvement, and relapse status.
101 RMS samples: SVD on top 258.csv holds three principal components
(V1/V2/V3), histology (ERMS vs ARMS), COG risk classifier, age status,
and survival time; Williamson.csv is the accompanying
gene-expression matrix (PAX3, FOXO1, etc.).
Yellow cells on the Scoring sheet feed into the Summary tab, which computes each metric automatically. Thresholds below are the excellent band; the paper reports full distributions rather than pass/fail.
| Metric | Abbrev. | What it measures | Excellent threshold |
|---|---|---|---|
| Intent Recognition Accuracy | IRA | Query intent correctly interpreted (1) or not (0), averaged over 50 queries. | ≥ 0.95 |
| AI Encoding Accuracy | AEA | Chosen visual encoding: optimal (1), acceptable (0.5), or incorrect (0). Mean of ≥3 expert raters in the final paper. | ≥ 0.90 |
| Update Latency | UL | End-to-end wall-clock time from spoken/typed query to rendered visualisation, in milliseconds. | < 1000 ms |
| Encoding Consistency Score | ECS | Fraction of queries for which the system chose the same encoding across all 3 repeated runs. | ≥ 0.95 |
| Visualisation Reasoning Score | VRS | Fraction of 4 reasoning components present in the system's explanation (why-questions only). | higher is better |
| Scene Stability Index | SSI | Property-update queries only: does the scene retain camera pose, selection, and axes after re-encoding? Not applicable to 2D dashboards. | ≥ 0.90 |
Every query is run three times on each system for the Encoding Consistency observation.
| ID | Class | Query |
|---|---|---|
| A1 | Descriptive | Show all patients in a 3D scatter plot grouped by gender. |
| A2 | Descriptive | Show the distribution of patients by phenotype (FAB L1 vs L2). |
| A3 | Descriptive | Colour the scatter by relapse status. |
| A4 | Descriptive | How many patients are there in total? |
| A5 | Descriptive | Show the number of patients per treatment protocol. |
| A6 | Descriptive | Show patient counts by ethnicity. |
| A7 | Descriptive | Show initial WBC counts across all patients. |
| A8 | Descriptive | Show the proportion of patients with CNS involvement. |
| A9 | Filtering | Show only patients who relapsed. |
| A10 | Filtering | List all female patients with FAB L2 phenotype. |
| A11 | Filtering | Show high-risk patients only. |
| A12 | Filtering | Filter to patients treated with BMT. |
| A13 | Filtering | Show patients with splenomegaly greater than 10 cm. |
| A14 | Filtering | List patients on the BFM 95 protocol. |
| A15 | Filtering | Show patients with blast percentage above 90. |
| A16 | Comparative | Compare initial WBC between relapsed and non-relapsed patients. |
| A17 | Comparative | Compare blast percentage across risk stratification groups. |
| A18 | Comparative | Compare male vs female counts by protocol. |
| A19 | Comparative | Compare platelet counts between standard and high risk groups. |
| A20 | Comparative | Compare CNS involvement rates between phenotypes. |
| A21 | Diagnostic | Why do relapsed patients cluster together in the scatter? |
| A22 | Diagnostic | Why was a scatter plot selected to visualise all patients? |
| A23 | Diagnostic | What factors are associated with high-risk stratification? |
| A24 | Diagnostic | Recolour the scatter by treatment status. |
| A25 | Diagnostic | Change point colour to risk stratification. |
| ID | Class | Query |
|---|---|---|
| B1 | Descriptive | Show a 3D scatter of V1, V2, V3 coloured by histology. |
| B2 | Descriptive | Colour all samples by age status. |
| B3 | Descriptive | Show the number of samples by clinical group. |
| B4 | Descriptive | Show sample counts by primary site. |
| B5 | Descriptive | Show a line chart of PAX3 expression across all samples. |
| B6 | Descriptive | Show the distribution of survival time. |
| B7 | Descriptive | Colour samples by COG risk classifier. |
| B8 | Descriptive | Show the count of ERMS vs ARMS samples. |
| B9 | Filtering | Show only ERMS samples. |
| B10 | Filtering | List all ERMS patients with AgeStatus unfavourable and survival_time under 2 years. |
| B11 | Filtering | Filter to metastatic (M1) samples. |
| B12 | Filtering | Show high-risk (COG) samples only. |
| B13 | Filtering | List samples from parameningeal primary sites. |
| B14 | Filtering | Show ARMS samples with unfavourable age status. |
| B15 | Filtering | Filter to samples that are dead from disease. |
| B16 | Comparative | Compare PAX3 expression between ERMS and ARMS. |
| B17 | Comparative | Compare survival time between favourable and unfavourable age status. |
| B18 | Comparative | Compare V1 values across histological subtypes. |
| B19 | Comparative | Compare metastasis rates between clinical groups. |
| B20 | Comparative | Compare FOXO1 expression between COG risk groups. |
| B21 | Diagnostic | Why was a line chart chosen to show PAX3 expression? |
| B22 | Diagnostic | Why do ARMS samples separate from ERMS in the scatter? |
| B23 | Diagnostic | What distinguishes high-risk from low-risk samples? |
| B24 | Diagnostic | Recolour the RMS scatter by survival status. |
| B25 | Diagnostic | Change encoding to colour by primary site. |
The 50 queries above, the six IVEM metrics, and the datasets they run against are open for other researchers to reproduce and extend. If you want the full scoring spreadsheet (with the auto-computed Summary tab, worked example row, and colour-coded input cells) - or want to compare a different system to MediVerse - please get in touch.
Email Rani for the spreadsheet Read the future-research directions