Assume that gi (x) = 1 (hence gk (x) = 0, k = i), update the expert i based on output error. Update gating network so that gi (x) is even closer to unity. Alternatively, a batch training method can be adopted: 1. Apply a clustering algorithm to cluster the set of training samples into n clusters. Use the membership information to train the gating network. 2. Assign each cluster to an expert module and train the corresponding expert module. 3. Fine-tune the performance using gradient-based learning. Note that the function of the gating network is to partition the feature. | Assume that gi x 1 hence gk x 0 k i update the expert i based on output error. Update gating network so that gi x is even closer to unity. Alternatively a batch training method can be adopted 1. Apply a clustering algorithm to cluster the set of training samples into n clusters. Use the membership information to train the gating network. 2. Assign each cluster to an expert module and train the corresponding expert module. 3. Fine-tune the performance using gradient-based learning. Note that the function of the gating network is to partition the feature space into largely disjointed regions and assign each region to an expert module. In this way an individual expert module only needs to learn a subregion in the feature space and is likely to yield better performance. Combining n expert modules under the gating network the overall performance is expected to improve. Figure shows an example using the batch training method presented above. The dots are the training and testing samples. The circles are the cluster centers that represent individual experts. These cluster centers are found by applying the k-means clustering algorithm on the training samples. The gating network output is proportional to the inverse of the square distance from each sample to all three cluster centers. The output value is normalized so that the sum equals unity. Each expert module implements a simple linear model a straight line in this example . We did not implement the third step so the results are obtained without fine-tuning. The corresponding MATLAB m-files . Illustration of mixture of expert network using batched training method. Support Vector Machines SVMs A support vector machine 14 has a basic format as depicted in Figure where yk x is a nonlinear transformation of the input feature vector x into a high-dimensional space new feature vector y x 1 x p2 x . pp x . The output y is computed as p y x wk k x b y x W b k 1 where w w1 w2 . wp is
