General Parameters

Controls the booster type in the model which eventually drives overall functioning

1. booster

For classification problems, we can use gbtree parameter. In gbtree a tree is grown one after other and attempts to reduce misclassification reate in subsequent iterations. The next tree is built by giving a higher weight to misclassified points by the previous tree.

For regression problems, we can use gbtree and gblinear. In gblinear, it builds a generalized linear model and optimizes it using regularization and gradient descent. The next model will built on residuals generated by previous iterations.

2. nthread

To enable parallel computing. The default is the maximum number of cores available

3. verbosity

Verbosity to display warning messages.The default value is 1 (warning), 0 for silent, 2 for info, and 3 for debug.

Booster Parameters:

Controls the performance of the selected booster

1. Eta

The range of eta is 0 to 1 and default value is 0.3. It controls the maximum number of iterations, the lower eta will generate the slower computation.

2. Gamma

The range of gamma is 0 to infinite and default value is 0 (no regularization). The higher gamma is the higher regularization, regularization means penalizing large coefficients that don’t improve the model’s performance.

3. nrounds

it refers to controls the maximum number of iterations.

4. nfold

The number of observation data is randomly partitioned into nfold equal size subsamples

5. max_depth

Maximum depth of a tree. The range of max_depth is 0 to infinite and default value is 6, increasing this value will make the model more complex and more likely to overfit.

6. Min_child_weight

The range of min_child_weight is 0 to infinite and default value is 1. If the leaf node has a minimum sum of instance weight lower than min_child_weight in the tree partition step than the process of splitting the tree will stop growing.

7. subsample

The range of subsample is 0 to 1 and default value is 1. It controls the number of ratio observations to a tree. If the value is set to 0.5 means that XGboost would randomly sample half of the training data prior to growing trees and this will prevent overfitting. subsample will occur once in every boosting iteration.

8. colsampe_bytree

The range of colsample_bytree is 0 to 1 and default value is 1. It controls the subsample ratio of columns when constructing each tree.

Learning Task Parameters

Sets and evaluates the learning process of booster from the given data.

1. Objective

• reg:squarederror for regression with squared loss
• binary:logistic for binary classification

2. eval_metric

Evaluation metrics for validaton data. The default is RMSE for regression case and error for classification case.

Next, we define the parameter will be used:

params <- list(booster = "gbtree",
               objective = "binary:logistic",
               eta=0.1, 
               gamma=10, 
               max_depth=10, 
               min_child_weight=1, 
               subsample=1, 
               colsample_bytree=1)

One of the simplest way to see the training progress is to set the verbose option as TRUE.

tic()
xgbcv <- xgb.cv( params = params, 
                 data = dtrain,
                 nrounds = 1000, 
                 showsd = T, 
                 nfold = 5,
                 stratified = T, 
                 print_every_n = 50, 
                 early_stopping_rounds = 20, 
                 maximize = F)

#> [1]  train-error:0.159371+0.000575   test-error:0.162041+0.002767 
#> Multiple eval metrics are present. Will use test_error for early stopping.
#> Will train until test_error hasn't improved in 20 rounds.
#> 
#> [51] train-error:0.134139+0.000721   test-error:0.140127+0.002438 
#> [101]    train-error:0.122489+0.000533   test-error:0.132233+0.001973 
#> Stopping. Best iteration:
#> [114]    train-error:0.121790+0.000932   test-error:0.131793+0.002073

print(xgbcv)

#> ##### xgb.cv 5-folds
#>     iter train_error_mean train_error_std test_error_mean test_error_std
#>        1        0.1593710    0.0005746874       0.1620406    0.002766615
#>        2        0.1589574    0.0008488257       0.1616114    0.002560948
#>        3        0.1582874    0.0008635417       0.1612242    0.002240224
#>        4        0.1574078    0.0012001357       0.1600936    0.002275888
#>        5        0.1563270    0.0017212650       0.1593920    0.001805679
#> ---                                                                     
#>      130        0.1217530    0.0009838978       0.1318562    0.002019269
#>      131        0.1217504    0.0009883673       0.1318562    0.002019269
#>      132        0.1217504    0.0009883673       0.1318562    0.002019269
#>      133        0.1217504    0.0009883673       0.1318562    0.002019269
#>      134        0.1217504    0.0009883673       0.1318562    0.002019269
#> Best iteration:
#>  iter train_error_mean train_error_std test_error_mean test_error_std
#>   114        0.1217898    0.0009316736       0.1317934    0.002072701

toc()

#> 72.96 sec elapsed

tic()
xgb1 <- xgb.train (params = params, 
                   data = dtrain, 
                   nrounds = xgbcv$best_iteration, 
                   watchlist = list(val=dtest,train=dtrain),
                   print_every_n = 100, 
                   early_stoping_rounds = 10, 
                   maximize = F , 
                   eval_metric = "error",
                   verbosity = 0)

#> [1]  val-error:0.157641  train-error:0.158900 
#> [101]    val-error:0.125644  train-error:0.121533 
#> [114]    val-error:0.124011  train-error:0.119858

toc()

#> 15.86 sec elapsed

xgbpred_prob <-predict(object = xgb1, newdata = dtest)
xgbpred <- ifelse (xgbpred_prob > 0.5,1,0)

In this section, we evaluate the performance of XGBoost model

confusionMatrix(as.factor(xgbpred), as.factor(label_test))

#> Confusion Matrix and Statistics
#> 
#>           Reference
#> Prediction     0     1
#>          0 13902  1830
#>          1  1131  7014
#>                                                
#>                Accuracy : 0.876                
#>                  95% CI : (0.8717, 0.8801)     
#>     No Information Rate : 0.6296               
#>     P-Value [Acc > NIR] : < 0.00000000000000022
#>                                                
#>                   Kappa : 0.7297               
#>                                                
#>  Mcnemar's Test P-Value : < 0.00000000000000022
#>                                                
#>             Sensitivity : 0.9248               
#>             Specificity : 0.7931               
#>          Pos Pred Value : 0.8837               
#>          Neg Pred Value : 0.8611               
#>              Prevalence : 0.6296               
#>          Detection Rate : 0.5822               
#>    Detection Prevalence : 0.6589               
#>       Balanced Accuracy : 0.8589               
#>                                                
#>        'Positive' Class : 0                    
#> 

let’s check the variable importance from the model:

var_imp <- xgb.importance(model = xgb1,
                          feature_names = dimnames(dtrain)[[2]])
var_imp %>% 
  mutate_if(is.numeric, round, digits = 2)

#>                            Feature Gain Cover Frequency
#>  1:                   deposit_type 0.35  0.07      0.01
#>  2:                        country 0.13  0.12      0.10
#>  3:                      lead_time 0.09  0.11      0.13
#>  4:                 market_segment 0.07  0.06      0.04
#>  5:      total_of_special_requests 0.06  0.06      0.03
#>  6:    required_car_parking_spaces 0.05  0.08      0.02
#>  7:         previous_cancellations 0.04  0.05      0.03
#>  8:              arrival_date_year 0.04  0.04      0.06
#>  9:                            adr 0.03  0.10      0.14
#> 10:                  customer_type 0.02  0.03      0.04
#> 11:       arrival_date_week_number 0.02  0.04      0.09
#> 12:             reserved_room_type 0.02  0.03      0.03
#> 13:                booking_changes 0.01  0.03      0.03
#> 14:             assigned_room_type 0.01  0.05      0.03
#> 15: previous_bookings_not_canceled 0.01  0.02      0.02
#> 16:                          hotel 0.01  0.02      0.02
#> 17:           stays_in_week_nights 0.01  0.02      0.03
#> 18:             arrival_date_month 0.01  0.01      0.02
#> 19:      arrival_date_day_of_month 0.01  0.01      0.03
#> 20:        stays_in_weekend_nights 0.01  0.01      0.02
#> 21:                           meal 0.01  0.02      0.02
#> 22:           distribution_channel 0.00  0.01      0.01
#> 23:                         adults 0.00  0.01      0.02
#> 24:                       children 0.00  0.01      0.01
#> 25:              is_repeated_guest 0.00  0.01      0.01
#> 26:           days_in_waiting_list 0.00  0.01      0.01
#> 27:                         babies 0.00  0.00      0.00
#>                            Feature Gain Cover Frequency

The function of xgb.importancedisplays the result importance values calculated with different importance metrics:

• The gain value means the percentage contribution of the feature for each tree in the model

• The cover value means the percentage represents the number of observations for each feature from all trees. From example, if we have 100 observations with 3 tree and each tree have 5, 8, and 10 observations for feature “A”. The cover value will calculate 5+8+10 = 23 observations from all trees for each feature. In this case, the feature “A” has a 0.23 cover value.

• The frequency value means the percentage representing the number of times a feature will splits in the trees of the model. For example, feature “A” occurred in 3 splits, 2 splits, and 2 splits for each tree. So the value of frequency feature “A” is 3+2+2=7 splits divide with all total numbers splits for all features.

xgb.ggplot.importance(var_imp,top_n = 10) + theme_minimal()

The graph shows the variable importance used the gain value by default and it also displays the cluster of features that have similar feature importances value. From 10 above features means their features have a significant impact on the result of prediction.

Next, we evaluate the perfomance model on the ROC curver

xgb_result <- data.frame(class1 = xgbpred_prob, actual = as.factor(label_test))

auc_xgb <- roc_auc(data = xgb_result, truth = actual,class1) 

result <- rbind(auc_ada, auc_xgb) %>% 
          mutate(model = c("AdaBoost", "XGBoost")) %>% 
          select(model, everything())
result

#> # A tibble: 2 x 4
#>   model    .metric .estimator .estimate
#>   <chr>    <chr>   <chr>          <dbl>
#> 1 AdaBoost roc_auc binary         0.955
#> 2 XGBoost  roc_auc binary         0.948

The AUC results show that AdaBoost and XGBoost model have similar value 0.94 and 0.95. To obtain the AdaBoost model we need to run model for 60 minutes, while the XGBoost model only need ~60 seconds. We can say that XGBoost works better than AdaBoost for speed.