はじめに
約1年前にも同様のことをしています。touch-sp.hatenablog.com
MXNetもAutoGluonもバージョンが新しくなっているので1年ぶりに再トライしました。
学習データを使って学習
今回は presets='best_quality' を追加しました。import pandas as pd from autogluon.tabular import TabularDataset, TabularPredictor all_data = pd.read_csv('train.csv', index_col=0) train_data = TabularDataset(all_data) save_path = 'ag-predict' predictor = TabularPredictor(label='Survived', path=save_path).fit( train_data = train_data, presets='best_quality' )
モデルは自動的に保存されます。
出力結果は記事の最後に載せておきます。
推論と提出用CSVファイル作成
保存された学習済みモデルを読み込んでテストデータに対して推論を行います。結果はKaggle提出用にCSVで保存するようにしています。
import pandas as pd from autogluon.tabular import TabularDataset, TabularPredictor all_data = pd.read_csv('test.csv', index_col=0) test_data = TabularDataset(all_data) save_path = 'ag-predict' predictor = TabularPredictor.load(save_path) y_pred = predictor.predict(test_data) result = pd.DataFrame(y_pred, columns=['Survived'], index = all_data.index) result.to_csv('result.csv')
Kaggleに提出、その結果は？
出来上がった「result.csv」を提出しました。結果は・・・
0.77033
8969位 / 13844
分母が以前より小さくなっているのはなぜでしょう？
正解率は前回よりわずかに改善しています。
欠損値の補完などなにも考えていないのでこんなものでしょう。
モデルの選定すら自分ではしていません。
すべてAutoGluonまかせです。
学習結果のサマリー
学習結果のサマリーを見るときはこのようにします。from autogluon.tabular import TabularPredictor save_path = 'ag-predict' predictor = TabularPredictor.load(save_path) summary = predictor.fit_summary()
実行すると以下のように出力されます。
*** Summary of fit() ***
Estimated performance of each model:
model score_val pred_time_val fit_time pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order
0 LightGBM_BAG_L2 0.879910 1.190925 17.071198 0.037570 0.910563 2 True 16
1 WeightedEnsemble_L3 0.879910 1.192192 17.354336 0.001267 0.283138 3 True 26
2 CatBoost_BAG_L2 0.877666 1.198847 18.396205 0.045492 2.235571 2 True 19
3 LightGBMXT_BAG_L2 0.872054 1.188943 16.984035 0.035589 0.823401 2 True 15
4 XGBoost_BAG_L2 0.872054 1.199248 16.953660 0.045893 0.793026 2 True 23
5 NeuralNetFastAI_BAG_L2 0.870932 1.270180 18.256152 0.116825 2.095518 2 True 22
6 WeightedEnsemble_L2 0.869809 0.246599 3.910404 0.001069 0.314337 2 True 14
7 LightGBMLarge_BAG_L2 0.868687 1.197225 18.086509 0.043870 1.925874 2 True 25
8 NeuralNetFastAI_BAG_L1 0.865320 0.084898 1.929984 0.084898 1.929984 1 True 10
9 RandomForestEntr_BAG_L2 0.856341 1.211845 16.921467 0.058490 0.760833 2 True 18
10 ExtraTreesGini_BAG_L2 0.855219 1.213233 16.958869 0.059878 0.798235 2 True 20
11 NeuralNetMXNet_BAG_L2 0.855219 1.821195 23.737728 0.667841 7.577094 2 True 24
12 RandomForestGini_BAG_L2 0.852974 1.211821 16.983857 0.058466 0.823223 2 True 17
13 ExtraTreesEntr_BAG_L2 0.852974 1.212818 16.948936 0.059464 0.788302 2 True 21
14 CatBoost_BAG_L1 0.847363 0.073447 1.556921 0.073447 1.556921 1 True 7
15 LightGBM_BAG_L1 0.843996 0.032330 0.676245 0.032330 0.676245 1 True 4
16 LightGBMLarge_BAG_L1 0.843996 0.053450 1.562874 0.053450 1.562874 1 True 13
17 XGBoost_BAG_L1 0.837262 0.043221 0.550573 0.043221 0.550573 1 True 11
18 LightGBMXT_BAG_L1 0.835017 0.035059 0.682306 0.035059 0.682306 1 True 3
19 RandomForestGini_BAG_L1 0.826038 0.072360 0.859407 0.072360 0.859407 1 True 5
20 RandomForestEntr_BAG_L1 0.822671 0.079357 0.803799 0.079357 0.803799 1 True 6
21 NeuralNetMXNet_BAG_L1 0.820426 0.534656 5.913800 0.534656 5.913800 1 True 12
22 ExtraTreesEntr_BAG_L1 0.818182 0.058779 0.812680 0.058779 0.812680 1 True 9
23 ExtraTreesGini_BAG_L1 0.818182 0.066245 0.806615 0.066245 0.806615 1 True 8
24 KNeighborsDist_BAG_L1 0.670034 0.008914 0.002877 0.008914 0.002877 1 True 2
25 KNeighborsUnif_BAG_L1 0.659933 0.010638 0.002553 0.010638 0.002553 1 True 1
Number of models trained: 26
Types of models trained:
{'StackerEnsembleModel_KNN', 'StackerEnsembleModel_LGB', 'StackerEnsembleModel_NNFastAiTabular', 'StackerEnsembleModel_TabularNeuralNet', 'StackerEnsembleModel_XGBoost', 'WeightedEnsembleModel', 'StackerEnsembleModel_RF', 'StackerEnsembleModel_XT', 'StackerEnsembleModel_CatBoost'}
Bagging used: True (with 8 folds)
Multi-layer stack-ensembling used: True (with 3 levels)
Feature Metadata (Processed):
(raw dtype, special dtypes):
('category', []) : 3 | ['Ticket', 'Cabin', 'Embarked']
('float', []) : 2 | ['Age', 'Fare']
('int', []) : 3 | ['Pclass', 'SibSp', 'Parch']
('int', ['binned', 'text_special']) : 9 | ['Name.char_count', 'Name.word_count', 'Name.capital_ratio', 'Name.lower_ratio', 'Name.special_ratio', ...]
('int', ['bool']) : 1 | ['Sex']
('int', ['text_ngram']) : 9 | ['__nlp__.henry', '__nlp__.john', '__nlp__.master', '__nlp__.miss', '__nlp__.mr', ...]
/mnt/wsl/PHYSICALDRIVE2p1/legacymxnet/lib/python3.8/site-packages/autogluon/core/utils/plots.py:138: UserWarning: AutoGluon summary plots cannot be created because bokeh is not installed. To see plots, please do: "pip install bokeh==2.0.1"
warnings.warn('AutoGluon summary plots cannot be created because bokeh is not installed. To see plots, please do: "pip install bokeh==2.0.1"')
*** End of fit() summary ***プロットしたければbokehをインストールするようにと警告がでます。
インストールした後に再度実行すると「SummaryOfModels.html」が保存されます。
開くと以下のような図になっています。
動作環境
Intel Core i7-7700K RAM 32G NVIDIA GTX 1080 (VRAM 8G)
Ubuntu 20.04 on WSL2 Python 3.8.10
autogluon==0.3.2b20220104 mxnet-cu112==1.9.0
出力結果
学習した時の出力です。Presets specified: ['best_quality']
Beginning AutoGluon training ...
AutoGluon will save models to "ag-predict/"
AutoGluon Version: 0.3.2b20220104
Python Version: 3.8.10
Operating System: Linux
Train Data Rows: 891
Train Data Columns: 10
Preprocessing data ...
AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).
2 unique label values: [0, 1]
If 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])
Selected class <--> label mapping: class 1 = 1, class 0 = 0
Using Feature Generators to preprocess the data ...
Fitting AutoMLPipelineFeatureGenerator...
Available Memory: 6555.41 MB
Train Data (Original) Memory Usage: 0.31 MB (0.0% of available memory)
Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
Stage 1 Generators:
Fitting AsTypeFeatureGenerator...
Note: Converting 1 features to boolean dtype as they only contain 2 unique values.
Stage 2 Generators:
Fitting FillNaFeatureGenerator...
Stage 3 Generators:
Fitting IdentityFeatureGenerator...
Fitting CategoryFeatureGenerator...
Fitting CategoryMemoryMinimizeFeatureGenerator...
Fitting TextSpecialFeatureGenerator...
Fitting BinnedFeatureGenerator...
Fitting DropDuplicatesFeatureGenerator...
Fitting TextNgramFeatureGenerator...
Fitting CountVectorizer for text features: ['Name']
CountVectorizer fit with vocabulary size = 8
Stage 4 Generators:
Fitting DropUniqueFeatureGenerator...
Types of features in original data (raw dtype, special dtypes):
('float', []) : 2 | ['Age', 'Fare']
('int', []) : 3 | ['Pclass', 'SibSp', 'Parch']
('object', []) : 4 | ['Sex', 'Ticket', 'Cabin', 'Embarked']
('object', ['text']) : 1 | ['Name']
Types of features in processed data (raw dtype, special dtypes):
('category', []) : 3 | ['Ticket', 'Cabin', 'Embarked']
('float', []) : 2 | ['Age', 'Fare']
('int', []) : 3 | ['Pclass', 'SibSp', 'Parch']
('int', ['binned', 'text_special']) : 9 | ['Name.char_count', 'Name.word_count', 'Name.capital_ratio', 'Name.lower_ratio', 'Name.special_ratio', ...]
('int', ['bool']) : 1 | ['Sex']
('int', ['text_ngram']) : 9 | ['__nlp__.henry', '__nlp__.john', '__nlp__.master', '__nlp__.miss', '__nlp__.mr', ...]
0.2s = Fit runtime
10 features in original data used to generate 27 features in processed data.
Train Data (Processed) Memory Usage: 0.07 MB (0.0% of available memory)
Data preprocessing and feature engineering runtime = 0.16s ...
AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'
To change this, specify the eval_metric argument of fit()
AutoGluon will fit 2 stack levels (L1 to L2) ...
Fitting 13 L1 models ...
Fitting model: KNeighborsUnif_BAG_L1 ...
0.6599 = Validation score (accuracy)
0.0s = Training runtime
0.01s = Validation runtime
Fitting model: KNeighborsDist_BAG_L1 ...
0.67 = Validation score (accuracy)
0.0s = Training runtime
0.01s = Validation runtime
Fitting model: LightGBMXT_BAG_L1 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.835 = Validation score (accuracy)
0.68s = Training runtime
0.04s = Validation runtime
Fitting model: LightGBM_BAG_L1 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.844 = Validation score (accuracy)
0.68s = Training runtime
0.03s = Validation runtime
Fitting model: RandomForestGini_BAG_L1 ...
0.826 = Validation score (accuracy)
0.86s = Training runtime
0.07s = Validation runtime
Fitting model: RandomForestEntr_BAG_L1 ...
0.8227 = Validation score (accuracy)
0.8s = Training runtime
0.08s = Validation runtime
Fitting model: CatBoost_BAG_L1 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.8474 = Validation score (accuracy)
1.56s = Training runtime
0.07s = Validation runtime
Fitting model: ExtraTreesGini_BAG_L1 ...
0.8182 = Validation score (accuracy)
0.81s = Training runtime
0.07s = Validation runtime
Fitting model: ExtraTreesEntr_BAG_L1 ...
0.8182 = Validation score (accuracy)
0.81s = Training runtime
0.06s = Validation runtime
Fitting model: NeuralNetFastAI_BAG_L1 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.8653 = Validation score (accuracy)
1.93s = Training runtime
0.08s = Validation runtime
Fitting model: XGBoost_BAG_L1 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.8373 = Validation score (accuracy)
0.55s = Training runtime
0.04s = Validation runtime
Fitting model: NeuralNetMXNet_BAG_L1 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.8204 = Validation score (accuracy)
5.91s = Training runtime
0.53s = Validation runtime
Fitting model: LightGBMLarge_BAG_L1 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.844 = Validation score (accuracy)
1.56s = Training runtime
0.05s = Validation runtime
Fitting model: WeightedEnsemble_L2 ...
0.8698 = Validation score (accuracy)
0.31s = Training runtime
0.0s = Validation runtime
Fitting 11 L2 models ...
Fitting model: LightGBMXT_BAG_L2 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.8721 = Validation score (accuracy)
0.82s = Training runtime
0.04s = Validation runtime
Fitting model: LightGBM_BAG_L2 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.8799 = Validation score (accuracy)
0.91s = Training runtime
0.04s = Validation runtime
Fitting model: RandomForestGini_BAG_L2 ...
0.853 = Validation score (accuracy)
0.82s = Training runtime
0.06s = Validation runtime
Fitting model: RandomForestEntr_BAG_L2 ...
0.8563 = Validation score (accuracy)
0.76s = Training runtime
0.06s = Validation runtime
Fitting model: CatBoost_BAG_L2 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.8777 = Validation score (accuracy)
2.24s = Training runtime
0.05s = Validation runtime
Fitting model: ExtraTreesGini_BAG_L2 ...
0.8552 = Validation score (accuracy)
0.8s = Training runtime
0.06s = Validation runtime
Fitting model: ExtraTreesEntr_BAG_L2 ...
0.853 = Validation score (accuracy)
0.79s = Training runtime
0.06s = Validation runtime
Fitting model: NeuralNetFastAI_BAG_L2 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.8709 = Validation score (accuracy)
2.1s = Training runtime
0.12s = Validation runtime
Fitting model: XGBoost_BAG_L2 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.8721 = Validation score (accuracy)
0.79s = Training runtime
0.05s = Validation runtime
Fitting model: NeuralNetMXNet_BAG_L2 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.8552 = Validation score (accuracy)
7.58s = Training runtime
0.67s = Validation runtime
Fitting model: LightGBMLarge_BAG_L2 ...
Fitting 8 child models (S1F1 - S1F8)
ParallelLocalFoldFittingStrategy is used to fit folds
0.8687 = Validation score (accuracy)
1.93s = Training runtime
0.04s = Validation runtime
Fitting model: WeightedEnsemble_L3 ...
0.8799 = Validation score (accuracy)
0.28s = Training runtime
0.0s = Validation runtime
AutoGluon training complete, total runtime = 53.65s ...
TabularPredictor saved. To load, use: predictor = TabularPredictor.load("ag-predict/")すべての学習に1分かかっていないのが驚きです。
