MVision AI Contour+ (Guideline-Based AI Segmentation): highlights from an independent validation study using a comprehensive and curated data set

Overall, 87% of the radiation oncologists who participated in the study gave the MVision AI Contour+ a score of 4 or 5 (minor or no editing required).

How to properly test an AI-based auto-contouring solution?

Artificial-intelligence applications are becoming increasingly popular in medicine, including Radiotherapy. Before AI-based tools can be widely implemented, they need to be carefully evaluated in terms of safety and efficacy. Auto-contouring solutions are usually evaluated for time saving and quality. The perceived performance of the AI models can be judged statistically according to the spatial similarity with a reference set of manually contoured structures and qualitatively for clinical acceptance.

If the team chooses to test the new solution by comparing it to clinical contours from previously treated patients, the results might be misleading due to the heterogeneity of the reference data. The well-known interobserver variability of manual contours and possible deviations from international standards and guidelines make the use of retrospective clinical data problematic.

A better solution would be the creation of a well curated reference dataset generated by experts although this requires substantially more time and effort. Even if subjectivity cannot be completely avoided, this approach offers more reliable results than using uncurated retrospective clinical data.

 

What method has been successfully used?

A group of experienced medical physicists and radiation oncologists from IRCCS Azienda Ospedaliero-Universitaria di Bologna published this year the results of a complex evaluation of MVision AI models’ quality (1). The authors used multiple parameters to assess the clinical acceptability and the potential for improving their daily activity. 

Measuring distances, overlapping surfaces and analyzing percentages is something that we can use for objective judgment, but sometimes the numbers are not enough to estimate their practical significance. The time saved by using automated predictions and the grade of Radiation Oncologists’ satisfaction represent pragmatic modalities of evaluation. Consistency, which should be as high as possible, is an important aspect in radiotherapy. 

To assess the impact of using Contour+, the team evaluated the interobserver variability between the manual contours and between the manually adjusted MVision AI predictions, respectively.

The complex approach of the Italian group is reflected also by the wide variety of analyzed cases. One hundred and eleven cases which had a tumor located in one of the most frequent anatomical sites  were selected. Head and neck, breast, thorax, abdomen, male or female pelvis cases were manually delineated according to institutional protocols which were based on ESTRO and RTOG guidelines. 

MVision AI Contour+ provided the study contours for 59 different structures – organs at risk or CTV structures as lymph node regions and seminal vesicles. It is important to mention that the patients included were scanned on two different machines, with variable slice thickness, and that for 67% of the patients from the abdominal site, 80% from those from the head and neck and 90% from those with thoracic cancers had contrast medium used. 

What were the results?

Such an ample evaluation brings an equally substantial amount of data, which cannot be covered in a short summary. We chose to mention some of the results that have practical importance.

The quality of the automatic contours was graded as 1 if unusable; 2, 3 and 4 if various grades of adjustments were needed (2 – major; 3 – some; 4 – minor) and 5 if there was no need of editing. The satisfactory grade was paired with the Lickert scale, ranging from 1 (poor) to 5 (excellent).  

Overall, 44% of the radiation oncologists who participated in the study gave the MVision AI solution a score of 4 and 43% gave the maximum of scoring, meaning 5. The score given by senior radiation oncologists was significantly higher than the scoring made by junior radiation oncologists. However, the time for manually adjusting the MVision AI contours did not differ according to physicians’ experience, which shows that the solution is equally helpful for all users. The median time for manual contouring was 25 minutes, ranging from 8 to 115 minutes. 

The median time for MVision AI contours generation was 2.3 minutes, ranging from 1.2 to 8 minutes, and for manual corrections the median needed time was 10 minutes, ranging from 0.3 to 46.3 minutes. Details on the evaluation and time save on each evaluated site can be found in Figure 1 and Table 1.

Slice thickness and the use of contrast did not interfere with the performance of the MVision AI models. Manually adjusting MVision AI contours significantly decreased interobserver variability compared to the manual contours.

Figure 1. Overview of the time save and satisfaction with MVision AI models in a comprehensive dataset (*head and neck median time save value differs from the source article, being provided by the authors)

What can be concluded?

Complex and extensive evaluation of MVision AI solution was highly appreciated by clinicians, reduced contouring time and decreased interobserver variability.

Guideline-consistent ground truth contours and rigorous peer-review are a distinctive hallmark of  MVision AI’s GBS (guideline based segmentation) approach to training AI models for auto- contouring. MVision AI is continuously improving its models, so newer versions are expected to perform even better. These excellent results and many similar others, motivate the MVision AI team to continue developing in the same direction and support clinicians to improve cancer care.

Table 1. Time save, satisfaction grades for the anatomical site groups included in the analysis by Strolin et al, 2023

VOI – volume of interest

 

Reference

Strolin S, Santoro M, et al (2023) How smart is artificial intelligence in organ delineation? Testing a CE and FDA-approved Deep-Learning tool using multiple expert contours delineated on planning CT images. Front. Oncol. 13:1089807. doi: 10.3389/fonc.2023.1089807

Our Newsletter

Subscribe to get information, latest news and other interesting offers about MVision AI

Related Posts

20.6.2024

World Kidney Cancer Day 2024: “Listen”

World Kidney Cancer Day is observed every year on the second Thursday of June. “Listening” is this year’s theme, which refers to the benefits of shared decision making, patient’s empowerment to seek a second opinion and creating awareness around misinformation and how to check if sources are reliable (1). More…

News

14.6.2024

MVision AI Appoints Antti Pirskanen as New Chief Financial Officer

Helsinki, Finland – June 14, 2024 – MVision AI, a pioneering company in the development of innovative AI solutions for radiation oncology is pleased to announce the appointment of Antti Pirskanen as its new Chief Financial Officer (CFO). Antti will assume his responsibilities immediately, bringing extensive expertise in financial management…

Press Releases

14.6.2024

MVision AI Receives MFDS Approval and Market Authorization for South Korea

HELSINKI, Finland – June 14, 2024 MVision AI is pleased to announce that our application for MVision AI Segmentation (Contour+) in South Korea has been accepted by the Ministry of Food and Drug Safety (MFDS) under the “Image Segmentation Application Software” for image analysis algorithms to be used in radiation…

Press Releases