A digital processing technique for separating two cochannel frequency-modulation (FM) signals involves a partial algebraic solution that gives the phases of the two signals to within one of two possibilities, plus the use of a two-state trellis algorithm to trace the most likely correct sequence of possibilities. Other techniques for separating cochannel FM signals do not yield perfect separation under any circumstances; however, the present technique can yield perfect separation in the absence of noise.

The mathematical derivation of the technique begins with the complex-amplitude baseband representation of the pair of cochannel signals sampled at small time intervals. The total signal at the *n*th sampling interval is given by

,

where *A* and *B* are the known magnitudes of the two signals, and *θ* and *φ* are the unknown phases of the signals. Initially, it is assumed that there is no noise and that *A* and B vary slowly, relative to* θ* and *φ*. The problem is to estimate *θ * and *φ*, given *A*, *B*,

and *r*. The equation above can be manipulated algebraically to obtain the following equations:

,

where

.

Thus, each of the two unknown phases has been determined to within two possible exact values.

For a single sample *r*(*n*), there is no reason to prefer one of the two possibilities over the other. However, one can choose a sequence of solutions ...*θ*(*n* –2), *θ*(*n* –1), *θ*(*n*)... and ...*φ*(*n* –2), *φ*(*n* –1), *φ*(*n*)... that yields the bandwidth or the spectral density expected of the phase modulation. The sequence can be chosen with the help of a two-state trellis, in which the first state represents the solution

and the second state represents the solution

.

The sequence of solutions is traced through the trellis by use of a Viterbi algorithm. The solution chosen for each time step *n* is the one for which the instantaneous frequency disagrees minimally with a value predicted from values of instantaneous frequency chosen tentatively for previous time steps. In these computations, the instantaneous frequencies at the time steps are approximated by use of finite differences between tentative phase solutions. The predicted instantaneous frequency is the finite-difference value obtained by applying, to the phase samples, an *m*th-order (where *m* is an integer > 1) Levinson-Durbin linear predictive coder (LPC), which is a linear minimum-mean-squared-error estimator.

The metric for the trellis branch from the *k*th state at time *n* –1 to the *l*th state at time *n* is the square of [(the frequency for the hypothesized *θ* solution for state *l* at time *n*) – (the frequency predicted by the Levinson-Durbin LPC for time *n* on the basis of *θ* solutions on the path leading up to state *k* at time *n* –1)] plus the square of [the corresponding quantity for *φ*. The Viterbi algorithm operates by computing all four branch metrics at each time step, storing an accumulated metric, and tracing backwards through the trellis to find the correct sequence.

This technique was tested in a computational simulation, using five test cases involving two FM voice signals sampled at a rate of 132.3 kHz and a 5th-order LPC. In all cases, the two signals were separated perfectly; that is, to within the floating-point precision of the computer. In other words, the correct branch of the trellis was chosen at every time step. Additional research would be needed to provide for the case in which estimation of *A* and *B* in situations in which they are unknown, and to determine whether the technique is robust in the presence of noise.

*This work was done by Jon Hamkins of Caltech for *NASA's Jet Propulsion Laboratory*. For further information, access the Technical Support Package (TSP) *free on-line at www.nasatech.com/tsp * under the Information Sciences category. NPO-20483*