Intermodulation distortion is a tough thing to measure. We encounter this problem every day trying to measure the IP3 of the T3, the world’s highest linearity mixer. This is also why we developed the PBR series of power combiners. The high isolation allows us to measure very high IP3 values without distortion from the synthesizers affecting the measurement.
The other half of this measurement is the spectrum analyzer. Until someone releases a spectrum analyzer with a T3 as the mixer front end, you will be limited in how much power you can put in without creating internal two tone distortion. The main way to combat this is with an attenuator at the front end. If you increase the attenuation and the two tone goes down, then you know that it is caused by the spectrum analyzer and not the device under test.
To help, we’ve created a block diagram of how to measure TOI or IP3 or IIP (or, if you really want to do an apples to apples comparison between measurements, use OIP, which accounts for variations in conversion loss).
Note how the high isolation power divider prevents intermodulation from the synthesizer by preventing the signal from synthesizer A getting into synthesizer B. The other element is the attenuator on the spectrum analyzer, which prevents the mixer on the SA from causing it’s own intermodulation products. This is also important for spur measurements.
Rohde and Schwarz has a nice white paper that goes into more depth and analyzes the problem with a little more of an eye towards communication applications (we’re looking at the problem more generally, with radar and electronic warfare applications considered as well). If you’re trying to make accurate linearity measurements I recommend taking the time to read it, at least to hold you over until we can finish our write up of it.