Tune a model via grid search#

Note

The tutorial is for FuxiCTR v1.0.

This tutorial shows how to tune model hyper-parameters via grid search over the specified tuning space.

We provide a useful tool script run_param_tuner.py to tune FuxiCTR models based on YAML config files.

–config: The config file that defines the tuning space
–gpu: The available gpus for parameters tuning and multiple gpus can be used (e.g., using –gpu 0 1 for two gpus)
–tag: (optional) Specify the tag to determine which expid to run (e.g. 001 for the first expid). This is useful to rerun one specific experiment_id that contains the tag.

In the following example, we use the hyper-parameters of FM_test in ./config as the base setting, and create a tuner config file FM_tuner_config.yaml in benchmarks/tuner_config, which defines the tuning space for parameter tuning.

# FM_tuner_config.yaml
base_config: ../config/ # the location of base config
base_expid: FM_test # the expid of default hyper-parameters
dataset_id: taobao_tiny_data # the dataset_id used, which overwrites the dataset_id key in FM_test

tuner_space:
    model_root: './tuner_config/' # the value will override the default value in FM_test
    embedding_dim: [16, 32] # the values in the list will be grid-searched
    regularizer: [0, 1.e-6, 1.e-5] # the values in the list will be grid-searched
    learning_rate: 1.e-3 # it is equivalent to [1.e-3]
    batch_size: 128 # the value will override the default value in FM_test

Specifically, if a key in tuner_space has values stored in a list, those values will be grid-searched. Otherwise, the default value in FM_test will be applied.

Run the following command to start:

!cd benchmarks
!python run_param_tuner.py --config ./tuner_config/FM_tuner_config.yaml --gpu 0 1

After finished, all the searched results can be accessed from FM_tuner_config.csv in the ./benchmarks folder.

Note that if you want to run only one group of hyper-parameters in the search space, you can use --tag to specify which one to run. In the following example, 001 means the expid (i.e., FM_test_001_7f7f3b34) corresponding to the first group of hyper-parameters. It is useful when one needs to rerun an expid for reproduction.

!cd benchmarks
!python run_param_tuner.py --config ./tuner_config/FM_tuner_config.yaml --tag 001 --gpu 0 1

While the above example config file shows how to import base_expid and dataset_id from the base_config folder, it is also flexible to directly expand the base setting in the tunner config file. Both configurations are the same.

# This example load base_expid and dataset_id from the same file
base_expid: FM_test # the expid of default hyper-parameters
dataset_id: taobao_tiny_data # the dataset_id used, which overwrites the dataset_id key in FM_test

model_config:
    FM_test:
        model_root: '../checkpoints/'
        workers: 3
        verbose: 1
        patience: 2
        pickle_feature_encoder: True
        use_hdf5: True
        save_best_only: True
        every_x_epochs: 1
        debug: False
        model: FM
        dataset_id: taobao_tiny_data
        loss: binary_crossentropy
        metrics: ['logloss', 'AUC']
        task: binary_classification
        optimizer: adam
        learning_rate: 1.0e-3
        regularizer: 1.e-8
        batch_size: 128
        embedding_dim: 4
        epochs: 1
        shuffle: True
        seed: 2019
        monitor: 'AUC'
        monitor_mode: 'max'
    
dataset_config:
    taobao_tiny_data:
        data_root: ../data/
        data_format: csv
        train_data: ../data/tiny_data/train_sample.csv
        valid_data: ../data/tiny_data/valid_sample.csv
        test_data: ../data/tiny_data/test_sample.csv
        min_categr_count: 1
        feature_cols:
            - {name: ["userid","adgroup_id","pid","cate_id","campaign_id","customer","brand","cms_segid",
                      "cms_group_id","final_gender_code","age_level","pvalue_level","shopping_level","occupation"], 
                      active: True, dtype: str, type: categorical}
        label_col: {name: clk, dtype: float}

tuner_space:
    model_root: './tuner_config/' # the value will override the default value in FM_test
    embedding_dim: [16, 32] # the values in the list will be grid-searched
    regularizer: [0, 1.e-6, 1.e-5] # the values in the list will be grid-searched
    learning_rate: 1.e-3 # it is equivalent to [1.e-3]
    batch_size: 128 # the value will override the default value in FM_test

In what follows, we show a real example to tune FM model on Criteo_x4. Criteo_x4 is a reusable dataset split of the widely-used Criteo data, which can be obtained from the BARS benchmark. After downloading, you can put the csv data into data/Criteo/Criteo_x4.

Then, you can save the following tunner config to ./benchmarks/tuner_config/FM_criteo_x4_tuner_config_02.yaml

# FM_criteo_x4_tuner_config_02.yaml
base_config: ../config/
base_expid: FM_base
dataset_id: criteo_x4

dataset_config:
    criteo_x4:
        data_root: ../data/Criteo/
        data_format: csv
        train_data: ../data/Criteo/Criteo_x4/train.csv
        valid_data: ../data/Criteo/Criteo_x4/valid.csv
        test_data: ../data/Criteo/Criteo_x4/test.csv
        min_categr_count: 10
        feature_cols:
            - {name: [I1,I2,I3,I4,I5,I6,I7,I8,I9,I10,I11,I12,I13],
               active: True, dtype: float, type: categorical, preprocess: convert_to_bucket, na_value: 0}
            - {name: [C1,C2,C3,C4,C5,C6,C7,C8,C9,C10,C11,C12,C13,C14,C15,C16,C17,C18,C19,C20,C21,C22,C23,C24,C25,C26],
               active: True, dtype: str, type: categorical, na_value: ""}
        label_col: {name: Label, dtype: float}

tuner_space:
    model_root: './Criteo/FM_criteo_x4_001/'
    embedding_dim: 16
    regularizer: [0, 1.e-6, 1.e-5, 1.e-4]
    learning_rate: 1.e-3
    batch_size: 10000
    seed: 2019

Run the following command, you can obtain the experimental results in FM_criteo_x4_tuner_config_02.csv.

!cd benchmarks
!nohup python run_param_tuner.py --config ./tuner_config/FM_criteo_x4_tuner_config_02.yaml --gpu 0 1 > run.log &

time	reproducing command	expid	dataset_id	train	validation	test
20210831-232555	[command] python run_expid.py –version pytorch –config Criteo/FM_criteo_x4_001/FM_criteo_x4_tuner_config_02 –expid FM_criteo_x4_001_df584306 –gpu 0	[exp_id] FM_criteo_x4_001_df584306	[dataset_id] criteo_x4_9ea3bdfc	[train] N.A.	[val] logloss: 0.449649 - AUC: 0.803118	[test] logloss: 0.449396 - AUC: 0.803503
20210901-024437	[command] python run_expid.py –version pytorch –config Criteo/FM_criteo_x4_001/FM_criteo_x4_tuner_config_02 –expid FM_criteo_x4_002_4661e593 –gpu 0	[exp_id] FM_criteo_x4_002_4661e593	[dataset_id] criteo_x4_9ea3bdfc	[train] N.A.	[val] logloss: 0.444749 - AUC: 0.807179	[test] logloss: 0.444486 - AUC: 0.807512
20210901-115913	[command] python run_expid.py –version pytorch –config Criteo/FM_criteo_x4_001/FM_criteo_x4_tuner_config_02 –expid FM_criteo_x4_003_3da0082a –gpu 0	[exp_id] FM_criteo_x4_003_3da0082a	[dataset_id] criteo_x4_9ea3bdfc	[train] N.A.	[val] logloss: 0.443421 - AUC: 0.808198	[test] logloss: 0.443109 - AUC: 0.808607
20210902-091353	[command] python run_expid.py –version pytorch –config Criteo/FM_criteo_x4_001/FM_criteo_x4_tuner_config_02 –expid FM_criteo_x4_004_3402a9bc –gpu 0	[exp_id] FM_criteo_x4_004_3402a9bc	[dataset_id] criteo_x4_9ea3bdfc	[train] N.A.	[val] logloss: 0.449190 - AUC: 0.801985	[test] logloss: 0.448863 - AUC: 0.802439

.

For more running examples, please refer to the BARS-CTR-Prediction benchmark.

FuxiCTR V1.0

Tune a model via grid search

Tune a model via grid search#