As detailed in our Mixer Basics Primer, the isolations of a mixer refer to how well a balanced mixer cancels out each of the arms inputs from the outputs of the other arms. There are three isolations of different importance:
1) LO-RF Isolation: This is most important because the LO tends to be very close to the RF (usually the IF is low frequency). Therefore the LO will be in the band of the following components and contaminate the RF circuitry, whether in an upconversion or a downconversion.
2) LO-IF Isolation: This is also important as it is the worst of the isolations. The LO can contaminate the IF circuitry, especially if it is wideband, or cause conversion loss problems if it is reflected from the IF circuitry.
3) RF-IF Isolation: This is less of a problem because the LO is always the strongest signal in the system.
So how do you determine what the isolations are? For a given frequency conversion it is simple: measure the output of the RF or IF on a spectrum analyzer and measure the leaking component. LO power input divided by output power at the LO frequency at the RF or IF port output (or minus, in dBm units) gives the isolation. This is what matters ultimately for the system.
For making a mixer datasheet, however, things are more complicated. We want to know what the isolation is across the entire operating band of the mixer. The obvious way to do it is to hook it up to a network analyzer and measure the leakage. However, since the mixer is by definition a non-linear element, we have to pay attention to the parameters we choose. The power levels usually used to measure linear devices are much too low to measure a mixer, because the mixer diodes don’t turn on until the rated drive level of the mixer (around +10-+13 dBm for a typical double balanced L diode mixer). Some VNAs cannot produce that much power across the whole band. Fortunately ours generally can. Below are two graphs of the isolations of an M1-0412LA mixer, one with measurement power of -5 dBm, and one with measurement power of more than 10 dBm.
As you can see the isolations can vary dramatically with input power. Increasing the input power by 20 dB in this case can lead to an isolation improvement of 20 dB or better!
The last conundrum is how to measure RF-IF isolation, since the normal operating mode of a mixer is with a strong LO present. The truth of this is that generally the LO and RF ports of a mixer are interchangeable. This is actually a trick that we use sometimes to improve the spurious performance of a mixer (sometimes it works better backwards!). So if you measure the RF-IF isolation the same way as the LO-IF isolation (with a high power RF signal input), then it should replicate closely the actual performance you would see under operating circumstances. In any case, this is the least important isolation for most applications anyways.