Early vs. Late Fusion in Clinical ML: How to Choose

Search for early fusion vs late fusion and most of what comes back is academic papers and reviews. They are useful, but they rarely say how to pick one on a real clinical dataset, with real gaps in it. The question sits at the center of my current work. As an AI researcher at the University of Alberta, I built a benchmark for fusing 12-lead ECG waveforms with clinical text, and I have developed and evaluated 8 multimodal fusion architectures. This essay is the plain-language version of what I wish I had read first: what each fusion strategy assumes, where each one breaks, and how I would choose.

I will keep the findings qualitative. The research is not published, so there are no AUROCs here, only the reasoning.

Early, intermediate and late fusion, in one example

Take a concrete case: predicting a cardiovascular outcome for a patient from two sources. One is a 12-lead ECG, a dense multichannel waveform. The other is the clinical text written about that patient, which is long, irregular and full of abbreviations. These are about as different as two inputs can be.

This is how I use the three terms in this essay:

Early fusion. Combine the inputs, or very low-level features of them, before most of the learning happens, and train one model over the joint representation. The model can learn interactions between the ECG and the text from the first layer onward.
Intermediate fusion. Give each modality its own encoder, then merge the learned representations partway through the network, by concatenation, attention or a gating layer, and keep training the whole thing together.
Late fusion. Train a separate model per modality, each producing its own prediction, and combine the predictions at the end: an average, a weighted vote or a small model stacked on top.

The names describe where the information meets. That one choice decides what the model can learn, what it needs from your data, and what happens on the day a modality is missing.

What early fusion buys you, and what it costs

The case for early fusion is that the interesting signal may live in the interaction. An ECG pattern might matter a great deal for one patient history and very little for another. A model that sees both inputs together from the start can, in principle, learn that.

The costs, in my experience, are practical ones:

Alignment. You need to decide what "together" means. An ECG is a timestamped recording. Notes are written over hours or days, across one or more hospital episodes. Fusing early forces you to define a time window, and that definition is an easy place for future information to slip in.
Missing modalities. A jointly trained model expects every input. When a patient has an ECG but no usable note, you have to impute, mask or drop the patient, and each of those choices changes who your model is evaluated on.
One modality can dominate. If one input is much easier to learn from, a joint model can lean on it and barely use the other. The result looks like a multimodal model and behaves like a single-modality one.

I am currently working on an early-fusion model myself: raw 12-lead ECG, echocardiography video and structured clinical data, predicting one-year cardiovascular mortality with survival heads (DeepHit, discrete-time survival and Cox partial likelihood). It is in progress, so I can't report results.

What late fusion buys you, and what it costs

Late fusion is the conservative choice, and I mean that as a compliment.

It degrades gracefully. If the text model has nothing to say about a patient, you still have the ECG model's prediction. Missing data is handled by construction rather than by imputation.
It is easy to debug. Each unimodal model can be evaluated on its own, so you always know what each modality contributes before you combine them.
It is cheap to change. Swapping the text encoder doesn't mean retraining the ECG side.

The cost is the mirror image of early fusion's benefit. Each model makes its prediction without seeing the other input, so any signal that exists only in the interaction is lost before the combination step. If the two modalities carry mostly separate evidence, that loss is small. If the value is in how they relate, late fusion leaves it on the table.

Intermediate fusion is the attempt to keep most of both: separate encoders that can each be pretrained and inspected, plus a learned merge that can still capture interactions. In my opinion it is the right default for most clinical teams, which is also why it is the hardest to evaluate honestly. More moving parts means more ways for a reported gain to come from something other than the fusion.

What benchmarking 8 fusion architectures taught me

The most useful decision in the benchmark was not an architecture. It was the comparison set. Multimodal models were compared at the patient level against ECG-only and text-only models, across the same outcomes: AF outcomes, heart failure, cardiovascular diagnosis and long-term prognosis.

Two findings I can state. First, combining ECG and clinical text substantially improves cardiovascular outcome prediction. Second, there are information ceilings for rare stroke and bleeding outcomes. For those outcomes, the honest reading is that the inputs may not contain enough signal, and no fusion strategy will invent it.

If the inputs don't carry the signal, the fusion strategy can't add it.

In my opinion, that second finding reframes the whole question. "Early or late?" is the second question. The first is "Is there anything here to fuse?", and you only answer it by testing each modality on its own before you combine them.

The other lessons were about evaluation, not fusion:

Patient-disjoint splits are not optional. A patient with several hospital episodes can't appear in both training and test. Otherwise the model learns the patient, not the condition, and every fusion strategy looks better than it is.
Leakage audits come before comparisons. Clinical text can mention the outcome you are trying to predict. Early fusion makes that easier to miss, because the leaky feature is mixed into a joint representation from the start.
Differences between architectures need intervals. I used patient-level bootstrap confidence intervals and DeLong testing to compare models, plus seed-noise analysis. In my view, two fusion designs a fraction of a point apart, run on different seeds, are often the same model.
Reproducibility is what makes 8 architectures a benchmark. Reproducible experiment pipelines and automated tests are what let you compare the eighth design to the first one fairly, across large multi-GPU runs.

The same habit of distrusting a clean number applies outside research. I wrote about that on the factory side in production ML on a factory floor, where the hard constraints were physical precision and speed rather than missing notes. The fiber organizer case study has the details.

A decision table for clinical fusion

This is the table I would use to choose a starting point. It is a set of defaults, not a law, and you should still benchmark the alternatives.

Your situation	Start with	Why
Modalities often missing at prediction time	Late fusion	Each model still predicts on its own
Small dataset, many features per modality	Late or intermediate	A joint model has more to learn from fewer patients
Value likely lies in how the modalities interact	Intermediate, then early	The merge can learn interactions
Inputs share a clock (signals recorded together)	Early or intermediate	Alignment is natural, not imposed
Inputs on very different timescales (ECG vs notes)	Intermediate	Separate encoders, with a deliberate time window
You need to explain each modality's contribution	Late fusion	Per-modality predictions are directly inspectable
One modality is far stronger than the other	Late fusion, then test a merge	Check that the weaker one adds anything at all

If two rows point in different directions, I would favor the one about missing data. A model that works on the patients who have everything and fails quietly on everyone else is the worst outcome in a clinical setting, because nobody notices it.

How to apply this

If you are about to build a multimodal clinical model, this is the order I would work in:

Train and evaluate each modality on its own, on patient-disjoint splits. These are your baselines, and they tell you whether there is a ceiling.
Audit for leakage before any fusion experiment: time windows, outcome mentions in text, patients shared across splits.
Build late fusion first. It is cheap, and it gives you a floor that any more complex design must beat.
Try intermediate fusion next, and early fusion only if the inputs align naturally or the late-fusion gap suggests interactions matter.
Compare every design with patient-level confidence intervals and more than one seed. Report the gain over the best single modality, not over the weakest.
Decide how the model behaves when a modality is missing, and evaluate that case explicitly.

None of this is specific to ECGs and notes. The same logic holds for imaging and labs, or video and structured data. The details of my research roles are in the Experience section, and if you are working on a similar problem, the contact section is the place to reach me.