ICA: Independent Component Analysis

Figure 1 shows the linear mixing system with the ICA attempting separating the sources. For a set of independent sources, x(t) = {x₁(t), x₂(t), … , x_N(t)}, the mixing system produces output set, 

y(t) = {y₁(t), y₂(t), … , y_M(t)},

where H is the linear mixing matrix and N(t) is the noise terms for each channel. The goal is to discover the mixing matrix or to estimate the source signals.

To make sure the problem is solvable, the following two restrictions are assumed,

1. all sources are mutually statistically mutual independent,
2. the mixing matrix has full column ranks, that is, all columns must be independent.

In most practical applications, the restrictions are not satisfied. However, the restricted ideal formulation sets a theoretical benchmark and serves as a guide for practice.

1. Let us revisit the cocktail party problem. Assume that several people are speaking concurrently in the same room, as in a cocktail party. The problem is to separate the voices of different speakers using the recordings from several microphones. Humans have remarkable skills in separating target speech from a complex audio mixtures. However, the cocktail party problem is still posed as one of the most challenging problems in blind source separation for machines.

If we assume that a linear operator W can force out a statistically independent separation, that is,

z(t)=Wx(t).

Now the question becomes how to find W such that the statistical independence between each z is maximized.

Three measures are often used,

1. Mutual Information,
2. Entropy or Negentropy,
3. Maximum Likelihood.

Each of these measures can be used to implement iterative algorithms to maximize the independence of the output. Specifically implementations will be written in a separate document.