Brain tumor MRI analysis asks two questions: which type of tumor is present, and where it is. They are usually answered by a classifier and a separate segmentation network that never share information, so their answers can contradict each other. We adapt MedPLIB-7b-2e, a biomedical multimodal large language model, so that one model answers both questions in one response. It writes the tumor type followed by a special <SEG> token, and the hidden state of that token is decoded into the tumor mask.
Method
One model, one answer, two outputs.
The class and the mask come from one answer
Given an MRI slice \(x\) and a fixed instruction \(q\), MedPLIB writes an answer of the form “\(c\) <SEG>”, where \(c\) is one of the four classes. A projector \(P\) maps the hidden state \(h\) of the <SEG> token to a prompt vector. The SAM-Med2D mask decoder \(D\) combines this prompt with the image features \(F = E(x)\):
The prompt takes the place of the click or box that a user would normally give to SAM. The mask is therefore drawn from what the language model has just decided.
The answer is chosen by scoring
Only four answers are valid, so we do not parse free text. Each candidate \(a^{(c)}\) is scored by its log-likelihood and the best one is kept:
The mask comes from the <SEG> state of the winning answer. No rule removes it afterwards, so an empty mask on a non-tumorous slice has to come from the model itself.
Two-stage adaptation
Stage A trains only the modules that connect the parts: the image projector, the router, the <SEG> projector and the mask decoder. Stage B adds LoRA, \(W = W_0 + \tfrac{\alpha}{r}BA\) with \(r = 16\), to all 32 language model layers and switches on the adapters of the SAM-Med2D encoder. It minimizes
where the first term trains the answer and the other three train the mask. Non-tumorous slices are trained with an empty mask. Half of the training prompts ask for the class and the mask, a quarter for the class only and a quarter for the mask only.
Ask the model
Pick a test slice. The model scores each of the four valid answers, keeps the most likely one, and draws the mask from that answer's <SEG> token.
Results at a glance
| Model | Accuracy | Tumor Dice | Dice, all slices | False masks | Contradictions |
|---|---|---|---|---|---|
| MedPLIB, no adaptation | 0.294 | 0.040 | 0.085 | 63.6% | 7.0% |
| MedPLIB, unified, ours | 0.980 | 0.854 | 0.874 | 0.0% | 0.0% |
| EfficientNet-B0 + U-Net | 0.993 | 0.881 | 0.812 | 61.4% | 9.0% |
| EfficientNet-B0 + U-Net, with a hand-written rule | 0.993 | 0.881 | 0.898 | 0.0% | 0.4% |
Dice over all slices also scores the 140 non-tumorous slices, where an empty mask counts as correct. The hand-written rule removes the U-Net mask whenever EfficientNet-B0 predicts no tumor. Accuracy of the unified model is about the same on all three imaging planes: 0.980 on axial, 0.984 on coronal and 0.976 on sagittal slices. The full comparison is in the README.
What adaptation changes
The same model on the same 1,000 test slices, before and after our two training stages.
Real answers before and after adaptation
The first two cases show the gain from adaptation. The third shows its cost: training on one narrow task made the model lose some general skills, and its free descriptions shrank to a single word. Mixing general question and answer data into training is one way to keep these skills.
The model knows when it is unsure
The four answer scores give a confidence for every prediction, at no extra cost.
Case gallery
For each tumor type, the test slices where the unified model scored its best, median and worst Dice, and the two non-tumorous slices with the largest false masks before adaptation. Nothing is hand picked.