Jika saya mengerti dengan benar, nested-CV dapat membantu saya mengevaluasi model dan proses penyetelan hyperparameter apa yang terbaik. Loop dalam ( GridSearchCV
) menemukan hyperparameter terbaik, dan loop outter ( cross_val_score
) mengevaluasi algoritma tuning hyperparameter. Saya kemudian memilih yang tuning / model combo dari loop luar yang meminimalkan mse
(Saya sedang melihat classifier regresi) untuk tes model akhir saya.
Saya sudah membaca pertanyaan / jawaban tentang validasi silang-silang, tetapi belum melihat contoh pipeline lengkap yang memanfaatkan ini. Jadi, apakah kode saya di bawah ini (abaikan rentang hyperparameter yang sebenarnya - ini hanya sebagai contoh) dan proses berpikir masuk akal?
from sklearn.cross_validation import cross_val_score, train_test_split
from sklearn.grid_search import GridSearchCV
from sklearn.metrics import mean_squared_error
from sklearn.ensemble import RandomForestRegressor
from sklearn.svm import SVR
from sklearn.datasets import make_regression
# create some regression data
X, y = make_regression(n_samples=1000, n_features=10)
params = [{'C':[0.01,0.05,0.1,1]},{'n_estimators':[10,100,1000]}]
# setup models, variables
mean_score = []
models = [SVR(), RandomForestRegressor()]
# split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.3)
# estimate performance of hyperparameter tuning and model algorithm pipeline
for idx, model in enumerate(models):
clf = GridSearchCV(model, params[idx], scoring='mean_squared_error')
# this performs a nested CV in SKLearn
score = cross_val_score(clf, X_train, y_train, scoring='mean_squared_error')
# get the mean MSE across each fold
mean_score.append(np.mean(score))
print('Model:', model, 'MSE:', mean_score[-1])
# estimate generalization performance of the best model selection technique
best_idx = mean_score.index(max(mean_score)) # because SKLearn flips MSE signs, max works OK here
best_model = models[best_idx]
clf_final = GridSearchCV(best_model, params[best_idx])
clf_final.fit(X_train, y_train)
y_pred = clf_final.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print('Final Model': best_model, 'Final model RMSE:', rmse)