SVM parameter tuning and number of SVs (Matlab libsvm)

[Scribe]

Hello,

Several questions have popped during some classification experiments I have been running lately. I tried to follow the guidelines in svmlib documentation about how applying SVMs (for beginners, which I still am), but these questions were not answered.

Some info about the data
Input data has 35 features (dim), 945 points, 5 classes (all points labellled). Data is split in half randomly for training and testing set (random selection is done class by class).

Q1
Feature selection and SVM parameter tuning (C, gamma) have to be done but in which order ? I started with cross-validation on training set to choose C and gamma, using all the features. Then feature selection was performed, using my own implementation of Sequential Forward Selection using cross-validation accuracy returned by svmtrain.

Q2
I chose the 10 best features out of 35, and trained again SVM with that set of features and the chosen C, gamma parameters. Training and testing accuracy are between 75%-80% (satisfactory for my application), but I end up with about half of training points as support vectors. Is that too many Support Vectors in the model ? Is there a way/rule of thumb to tell whether the model is overfitting - the consistency between training and testing accuracy doesn't suggest that there is overfitting.

Thanks in advance for your help.

[Scribe]

229 views and no replies, was my question so stupid ?

[Sasha]

No, your questions are very interesting, but they haven't evident answers. By the way, question to you - what was Training and Testing accuracy before your reduction of feature amounts? Do you see significant improvment for testing accuracy, did you see early (for 35 features) large difference between training and validation sets accuracy?
About your questions:
1. You can use PCA (or your private method) before SVM using, but also can use SVM results for feature amount reduction (e.g., it is very simple for Linear Kernel using). But I don't understand (see my above question) - is it really necessary to reduce amount of features. For my point of view, it isn't large value for comparison with amoint of points (35/945), but is is also depends of each class positive/negative (imbalanced or not).
2. "Half of training points as support vectors. Is that too many Support Vectors in the model?". Generally speaking - no, it is Okay. But it is depends of your real situation and kernel type .

[Scribe]

Thanks Sasha for your time and answers.

Sasha Wrote:By the way, question to you - what was Training and Testing accuracy before your reduction of feature amounts? Do you see significant improvment for testing accuracy, did you see early (for 35 features) large difference between training and validation sets accuracy?

Using my own Sequential Forward Feature Selection algorithm based on SVM (gaussian kernel), I plotted a curve of cross-validation testing error vs number of features. It looked as taken from a textbook on feature selection, strong increase of accuracy with first features, no significant improvement after 10 features. We chose the 10 or 11 "best" features after that experiment.

PCA is excluded in this application. The input features have physical meaning, we can keep some and discard others, but recombining them or finding PCA is not an option.

Still about feature selection, a conceptual question. It seems to me that one should not select features with a given classifier (say Linear Discriminant Analysis LDA), then train another classifier (here SVM with gaussian kernel) with the selected features. What would justify that features that are good for LDA are good for SVM ? In a first series of experiments features were selected in SPSS but they did not give the best results on my SVM classifier. Which makes me thinks it is better to use the same classification scheme for feature selection and model building, is it correct or irrelevant ?

Quote:is it really necessary to reduce amount of features. For my point of view, it isn't large value for comparison with amoint of points (35/945), but is is also depends of each class positive/negative (imbalanced or not).

Precisely, I forgot to mention the classes are imbalanced. First three classes have roughly equal number of samples, and constitute most of the data. Two remaining classes may have 20-30 samples in the training set.

Quote:"Half of training points as support vectors. Is that too many Support Vectors in the model?". Generally speaking - no, it is Okay. But it is depends of your real situation and kernel type .

Kernel type was gaussian, with C = 256 and g = 2^(-8). Data has been scaled to [-1 1].

[kap]

Hi, there are two types of feature selection; filter and wrapper. What you did with SPSS was a filter and with SVM CV was wrapper (if I got your point). Wrappers are obviously favourable if time is not an issue. Another option would be to use Recursive Feature Elimination (SVM-RFE) which eliminates least important features in a recursive fashion. But for this you need to use linear kernel. If you want to stick with gaussians then I will say use wrappers. You can also use something randomized algorithm (e.g. Genetic Algorithm) for that. Look at Yiming Yang's publictaions on feature selection to see how different strategies affect SVM (e.g. http://nyc.lti.cs.cmu.edu/yiming/Publications/icml97.ps ).

[Scribe]

Thanks a bunch kap, I will look into this.

[kavnik]

Hello,
Can anyone tell me which kernels is best to be used for support vector machine and their limitation. If possible pls tell me where to find these informations.....