Machine Learning | Chen Xing

The Kernel Trick

Sat, 31 Jan 2026 00:00:00 +0000

TL;DR: A mathematical shortcut that allows linear algorithms (like SVMs) to solve non-linear problems by operating in a high-dimensional feature space without ever explicitly calculating the coordinates in that space.

1. Motivation: The Interaction Problem

The Issue: Most real-world data (especially in marketing) is not linearly separable.
- Example: Customer churn isn’t just about Age or Income, but the complex interaction Age * Income^2.
The Naive Solution: Manually create interaction features (e.g., add columns for $x_1^2, x_1x_2, x_2^2$).
The Bottleneck (Curse of Dimensionality):
- If you have $d$ features and want $n$-th order interactions, the number of dimensions grows combinatorially ($O(d^n)$).
- Result: Computational explosion. You run out of RAM trying to store the expanded feature matrix.

2. Intuition: The “Cheat Code”

The Insight: Many algorithms (SVM, PCA) do not need the data points themselves ($x$). They only need the angles/distances between points, represented by the Dot Product ($x^T y$).
The Trick: We replace the expensive dot product in high-dimensional space: $\phi(x)^T \phi(y)$, with a cheap function in low-dimensional space: $K(x, y)$.
Analogy: Determining if two people are distant cousins.
- Explicit: Sequencing their entire DNA (High Comp Cost).
- Kernel: Mixing a drop of blood and seeing if it changes color (Low Comp Cost, same result).

3. How It Works (The Algebra)

The algebraic identity that makes the trick possible. The order of operations changes the cost.

Explicit Mapping ($\phi$):
1. Expand inputs: $x \to [x_1^2, \sqrt{2}x_1x_2, x_2^2]$
2. Compute dot product.
- Cost: High.
Kernel Function ($K$):
1. Compute dot product of inputs: $x^T y$.
2. Apply non-linear function (e.g., square it): $(x^T y)^2$.
- Cost: Low (Linear).

The Polynomial Proof:

$$(x^T y + c)^d = \sum (\text{All interactions of degree } d)$$

Computing the left side (Kernel) automatically generates the sum on the right side.

4. The “Magic”: Infinite Dimensions (RBF)

The Radial Basis Function (Gaussian) Kernel allows us to operate in infinite dimensions.

$$K(x, y) = e^{-\gamma ||x - y||^2}$$

Why infinite? The Taylor expansion of $e^x$ is an infinite sum ($1 + x + \frac{x^2}{2!} + \dots$).
Takeaway: We can use an infinite-parameter model to fit complex data using a finite amount of memory.

5. Statistics Connection

Mercer’s Theorem: Guarantees that if a Kernel matrix is Positive Semi-Definite (PSD), it corresponds to a valid dot product in some feature space.
- Statisticians view: It’s a Covariance Matrix.
Kernel Selection = Prior Selection:
- Linear Kernel: Prior belief that relationships are simple/additive.
- Polynomial Kernel: Prior belief that specific interactions drive the outcome.
- RBF Kernel: Prior belief in smoothness (points close in $x$-space have similar $y$-values). This is the default “Neighborhood” prior.

6. Deep Learning vs. Kernels

Kernels: You manually choose the function $K$ (hard-coding the feature space $\phi$). “I believe the world works like a Gaussian.”
Deep Learning: You let the Neural Network learn the function $\phi$. “I will let the data teach me the shape of the feature space.”

Introduction to Latent Dirichlet Allocation (LDA) Model

Fri, 02 May 2025 00:00:00 +0000

Motivation

Uncovering the insights from User-generated content (UGC) is important for researchers as UGC provides a rich information about consumer’s experiences with quality.

Tirunillai and Tellis (2014) propose a unified framework:

Extract the latent dimensions of quality from UGC
Ascertain the valence, labels, validity, importance, dynamics, and heterogeneity of those dimensions
Use those dimensions for strategy analysis (e.g., brand positioning)

LDA Model

LDA is a generative probabilistic model used primarily for topic modeling in text. This model helps us discovering the hidden thematic structure in a large collection of documents.

Key Idea

Every document is a mixture of topics.
Every topic is a mixture of words.

LDA is a mathematical method for estimating both of these at the same time: finding the mixture of words that is associated with each topic, while also determining the mixture of topics that describes each document.

LDA R Code Example

Many awesome R packages to implement LDA. For example, this blog post tutorial by Julia Silge walks through how to build a structural topic model and then how to understand and interpret it.

library(stm)
topic_model <- stm(lyrics_data, K = 4)

The most important parameter when training a topic modeling is k, the number of topics.
This is like k in k-means in that it is a hyperparamter of the model and we must choose this value ahead of time.
We can find the best value for k using data-driven methods.

Dirichlet and Categorical distributions

As we are familiar with Beta and Bernoulli distributions, it is not hard to understand the Dirichlet and Categorical distributions, which are natural extensions of previous ones.

Mathematical Components of LDA

0. Model Setup:

Topics: $K$ topics.
Documents: $D$ documents.
Words: $V$ words in the vocabulary.

1. Topic Distribution for Each Document:

Each document $d$ has a topic distribution $\theta_d$.
$\theta_d \sim \text{Dirichlet}(\alpha)$.
Here, $\alpha$ is a $K$-dimensional vector, where $K$ is the number of topics.

2. Word Distribution for Each Topic:

Each topic $k$ has a word distribution $\phi_k$.
$\phi_k \sim \text{Dirichlet}(\beta)$.
$\beta$ is a $V$-dimensional vector, where $V$ is the vocabulary size.

3. Generative Process for Each Document:

For each document $d$:
- Draw topic distribution $\theta_d \sim \text{Dirichlet}(\alpha)$.
For each word $n$ in document $d$:
- Draw topic $z_{d,n} \sim \text{Multinomial}(\theta_d)$.
- Draw word $w_{d,n} \sim \text{Multinomial}(\phi_{z_{d,n}})$.

4. Inference Problem:

Compute posterior distribution:

$P(\theta, z | w, \alpha, \beta) = \frac{P(\theta, z, \phi, w | \alpha, \beta)}{P(w | \alpha, \beta)} = \frac{ P(w | z, \phi) , P(z | \theta) , P(\theta | \alpha) , P(\phi | \beta)}{P(w | \alpha, \beta)}$
Exact computation is intractable. Gibbs sampling or variational inference are used.

5. Hyperparameters $\alpha$ and $\beta$:

$\alpha$ and $\beta$ are hyperparameters of the Dirichlet distributions.
$\alpha$: Affects the mixture of topics in each document.
$\beta$: Affects the distribution of words in each topic.

6. Why Dirichlet Distribution?

In LDA, topics and words are assumed to be multinomially distributed.
Dirichlet is the conjugate prior to the multinomial, which simplifies computation, especially for Bayesian inference.
$X \sim Dirichlet(\alpha)$ with pdf $$f(X; \alpha) = \frac{1}{B(\alpha)} \prod_{i=1}^{K} x_i^{\alpha_i - 1}$$
Dirichlet distribution is a generalization of the Beta distribution.
- Binomial Likelihood 🤝 Beta Prior
- Multinomial Likelihood 🤝 Dirichlet Prior

Reference

Tirunillai, S., & Tellis, G. J. (2014). Mining Marketing Meaning from Online Chatter: Strategic Brand Analysis of Big Data Using Latent Dirichlet Allocation. Journal of Marketing Research, 51(4), 463–479. https://doi.org/10.1509/jmr.12.0106
Julia Silge’s tutorial Topic modeling for #TidyTuesday Taylor Swift lyrics

Interpretable ML — PDP, ALE and SHAP

Sun, 16 Feb 2025 00:00:00 +0000

1. Partial Dependence Plots (PDPs)

Partial Dependence Plots (PDPs) are a tool for interpreting the relationship between a subset of input features and the predictions of a machine learning model.

The key intuition is to isolate the marginal effect of one or more features on the model’s output by averaging out the influence of all other features. Here’s how it works intuitively:

Marginalization Over Other Features:
For a chosen feature (or set of features), the PDP shows how the model’s predictions change as the feature varies, while averaging over the observed values of all other features. This effectively “controls” for the other variables, revealing the average behavior of the model with respect to the feature of interest.
Visualization of Trends:
By plotting the average prediction against the feature’s value, PDPs reveal whether the relationship is linear, monotonic, or nonlinear. For example, in a house price prediction model, a PDP for the “number of bedrooms” might show if increasing bedrooms consistently raises prices or if the effect plateaus.
Handling Interactions (Optional):
PDPs can also be extended to two features to visualize interaction effects (e.g., how “square footage” and “number of bedrooms” jointly affect house prices).

Mathematical Explanation

Formulation of Partial Dependence

Let the model prediction function be denoted as $ f(\mathbf{X}) $ , where $ \mathbf{X} = (X_1, X_2, \dots, X_p) $ is the feature vector. Partition $ \mathbf{X} $ into two subsets:

$ X_S $ : The features of interest (e.g., $ X_1 $ and $ X_2 $ ).
$ X_C $ : The complement set of all other features ($ C = \{1, \dots, p\} \setminus S $ ).

The partial dependence function is defined as the expected value of $ f(\mathbf{X}) $ over the marginal distribution of $ X_C $ , conditioned on fixing $ X_S = \mathbf{x}_S $ :

$$ \text{PD}_S(\mathbf{x}_S) = \mathbb{E}_{X_C}\left[ f(\mathbf{x}_S, X_C) \right] = \int f(\mathbf{x}_S, \mathbf{x}_C) \, dP(\mathbf{x}_C), $$ where $ P(\mathbf{x}_C) $ is the marginal distribution of $ X_C $ .

Empirical Approximation

In practice, we approximate the integral using the observed data. For a dataset with $N$ samples, the empirical partial dependence is:

$$ \widehat{\text{PD}}_S(\mathbf{x}_S) = \frac{1}{N} \sum_{i=1}^N f(\mathbf{x}_S, \mathbf{x}_C^{(i)}), $$

where $ \mathbf{x}_C^{(i)} $ are the observed values of $ X_C $ for the $ i $ -th instance. For each value of $ \mathbf{x}_S $ , we:

Replace $ X_S $ with $ \mathbf{x}_S $ in all instances.
Compute predictions $ f(\mathbf{x}_S, \mathbf{x}_C^{(i)}) $ .
Average these predictions across all instances.

Key Assumptions and Limitations

Independence Assumption:
PDPs assume $ X_S $ and $ X_C $ are independent. If they are correlated (e.g., “bedrooms” and “square footage”), the PDP may extrapolate to unrealistic combinations of features.
Masking Interactions:
PDPs show average effects, potentially obscuring interactions. For example, if $ X_1 $ has opposite effects on predictions depending on $ X_2 $ , the PDP for $ X_1 $ might show a flat line due to averaging.
Computational Cost:
Evaluating $ \widehat{\text{PD}}_S $ requires $ N \times \text{(grid size of } \mathbf{x}_S) $ predictions, which can be expensive for large datasets.

Connection to Random Forests

Random Forests are ensembles of decision trees. The PDP for a Random Forest works by:

Fixing $ X_S = \mathbf{x}_S $ across all trees.
For each tree, using its structure to compute predictions with $ \mathbf{x}_S $ .
Averaging predictions across all trees and instances.

Why It Works for Random Forests

Trees naturally handle mixed data types and nonlinearities, making PDPs effective for visualizing complex relationships.
The averaging over trees and data instances aligns with the empirical approximation of $ \mathbb{E}_{X_C}[f(\mathbf{x}_S, X_C)] $ .

Example and Visualization

Suppose we train a Random Forest to predict house prices ($ Y $ ) using features $ X_1 $ (square footage) and $ X_2 $ (number of bedrooms). A PDP for $ X_1 $ would:

Create a grid of $ X_1 $ values (e.g., 500, 1000, 1500 sq.ft).
For each grid value $ x_1 $ :
- Set $ X_1 = x_1 $ for all instances in the dataset.
- Compute predictions using the Random Forest.
- Average predictions across all instances.
Plot $ x_1 $ vs. the average prediction.

The resulting curve might show that prices increase with square footage but plateau beyond 2000 sq.ft.

Summary

Intuition: PDPs visualize the average effect of a feature on predictions, marginalizing over other features.
Mathematics: They approximate $ \mathbb{E}_{X_C}[f(\mathbf{x}_S, X_C)] $ using the empirical distribution of the data.
Use Case: Effective for interpreting complex models like Random Forests, but caveats apply (e.g., feature correlations). For interactions, use 2D PDPs or ICE plots.

2. Accumulated Local Effects (ALE) plots

Motivation Behind ALE (relax independence assumption)

Partial Dependence Plots (PDPs) are great for showing average feature effects, but they rely on an independence assumption—averaging over the marginal distribution of other features. This can lead to unrealistic combinations when features are correlated.

Accumulated Local Effects (ALE) plots are designed to overcome this limitation. Instead of averaging over all observations globally, ALE focuses on differences in predictions to isolate each feature’s effect.

Compute Local Derivatives: They first estimate the local effect (i.e., the derivative) of the feature on the model prediction in small intervals (bins) of the feature’s range.
Accumulate the Effects: Then they integrate (accumulate) these local effects over the feature’s range, starting from a reference point (often the minimum value).
Centering: Finally, the accumulated effect is centered so that the overall average effect is zero, making it easier to compare across features.

This method only relies on differences between nearby observations—where the joint distribution is supported—thus avoiding unrealistic extrapolations.

Mathematical Explanation

Naive summary of computation steps for ALE plots:

Divide the Feature Range:

Split the feature’s values into intervals (bins).

Calculate Local Differences:

Within each bin, for each instance, compute the change in the model’s prediction when the feature’s value is replaced with the bin’s upper and lower boundaries.
Average these differences within the bin to determine the feature’s local effect.

Accumulate Effects:

Sum these local effects across the bins to visualize the feature’s overall impact on the model’s predictions.

By focusing on prediction differences, ALE plots effectively account for feature correlations, providing a clearer understanding of each feature’s true effect on the outcome.

Interpretation

“The value of the ALE can be interpreted as the main effect of the feature at a certain value compared to the average prediction of the data. For example, an ALE estimate of -2 at $x_j=3$ means that when the j-th feature has value 3, then the prediction is lower by 2 compared to the average prediction.” – IML textbook

3. SHAP Values

Intuition:
SHAP (SHapley Additive exPlanations) values attribute the difference between a model’s prediction for an instance and the average prediction to each feature. Inspired by cooperative game theory (Shapley values), SHAP fairly distributes the “contribution” of each feature to the prediction.

Key Properties:

Additivity: The sum of SHAP values for all features equals the difference between the prediction and the baseline (average prediction).
Local Accuracy: Each feature’s contribution is computed for a single instance.
Consistency: If a feature’s impact increases, its SHAP value won’t decrease.

Mathematical Formulation:
For a model $f$, the SHAP value $\phi_i$ for feature $i$ is:

$$ \phi_i = \sum_{S \subseteq F \setminus {i}} \frac{|S|! (|F| - |S| - 1)!}{|F|!} \left[ f(S \cup {i}) - f(S) \right] $$

$F$: Set of all features.
$S$: Subset of features excluding $i$.
$f(S)$: Expected prediction when only features in $S$ are known.

Interpretation:

$\phi_i > 0$: Feature $i$ increases the prediction.
$\phi_i < 0$: Feature $i$ decreases the prediction.

4. ALE vs. SHAP: When to Use Which?

When to Use ALE:

You care about global feature effects (e.g., “Does income positively affect loan approval?”).
Features are correlated, and you want reliable estimates.
You need a clear visual of the feature’s directional trend.

When to Use SHAP:

You need instance-level explanations (e.g., “Why was this loan rejected?”).
You want to compare feature importance across the dataset (aggregated SHAP).
Your model has nonlinear interactions best captured per-instance.

Final Recommendation

Use ALE if your goal is to understand the overall directional relationship between a feature and the outcome, especially with correlated features.
Use SHAP if you need to explain individual predictions or compare feature importance globally.

For example:

A bank might use ALE to audit whether income has a fair directional effect on loan approvals.
The same bank could use SHAP to explain to a customer why their specific loan application was denied.

R packages 📦

Here are some R packages to implement:

References

More theory behind ALE plot, check this paper: Apley, D. W., & Zhu, J. (2020). Visualizing the effects of predictor variables in black box supervised learning models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(4), 1059-1086.

“8.1 Partial Dependence Plot (PDP)” in IML textbook by Christoph Molnar

“8.2 Accumulated Local Effects (ALE) Plot” in IML textbook

Interpretable Machine Learning (IML) – online tutorial

scikit-learn tutorial: 4.1. Partial Dependence and Individual Conditional Expectation plots

Notes on CLIP model

Wed, 18 Sep 2024 00:00:00 +0000

The following tutorial is from the conversation with ChatGPT o1.

Understanding the CLIP Model in Simple Terms

Imagine you’re sorting through a massive collection of photos and captions from the internet. You want a system that can look at a picture and tell you what it’s about in words, or read a description and find the matching image. That’s essentially what the CLIP model does.

What is CLIP?

CLIP stands for Contrastive Language-Image Pre-training. It’s a type of artificial intelligence model developed by OpenAI that learns to connect images and text descriptions. Think of it as a bilingual person fluent in both the language of pictures and the language of words, able to translate between the two seamlessly.

How Does CLIP Work?

Learning from Pairs: CLIP is trained on a vast dataset of image-caption pairs from the internet. For each image, there’s a corresponding text description.
Creating a Shared Space: The model learns to represent both images and text descriptions in a shared mathematical space called an embedding space. In this space, related images and texts are positioned close to each other, while unrelated ones are far apart.
- Imagine plotting images and captions on a 3D map where similar items cluster together.
Contrastive Learning: CLIP uses a technique called contrastive learning. It trains itself to:
- Pull Together: Bring matching images and captions closer in the embedding space.
- Push Apart: Separate non-matching images and captions.
Understanding Content: Once trained, CLIP can:
- Image to Text: Look at a new image and find the best-matching description from a set of options.
- Text to Image: Read a description and find the image that best fits it.

Why is CLIP Important in Image Processing?

Versatility: Unlike traditional models that are trained for specific tasks (like identifying cats vs. dogs), CLIP can understand a wide range of concepts without task-specific training.
Zero-Shot Learning: CLIP can recognize objects and concepts it wasn’t explicitly trained on. This means it can perform tasks out-of-the-box without additional training data.
- For example, it can identify a “teapot” in an image even if it hasn’t seen labeled examples of teapots before.

An Analogy

Think of CLIP as a matchmaker at a multilingual conference. It listens to conversations in one language (like images) and pairs them with the right conversations in another language (like text descriptions). It knows which pairs belong together because it has learned the patterns of both languages through extensive experience.

Statistical Perspective

You can view CLIP as modeling the association between two variables: images and text. It uses high-dimensional representations (embeddings) to capture the complex relationships between visual and textual data. The contrastive learning approach optimizes the model to increase the statistical dependence between matching image-text pairs while minimizing it for non-matching pairs.

Practical Applications

Image Search: Type a description, and CLIP can find relevant images.
Content Moderation: Automatically identify inappropriate images based on their content.
Assisting Creativity: Help artists and marketers find visual inspiration based on textual ideas.

Summary

CLIP bridges the gap between images and text by learning from a large collection of image-caption pairs. It creates a shared space where both images and texts are represented, enabling it to understand and match them effectively. This makes it a powerful tool in image processing tasks that involve understanding and generating descriptions of visual content.

Auto-Encoder study notes

Sun, 03 Mar 2024 00:00:00 +0000

Basic Idea of Auto-encoder

Concept	Key Features
Encoder	Transforms input data into a compressed, lower-dimensional representation.
Decoder	Reconstructs input data from the compressed form to closely resemble the original data.
Embedding	The compressed vector representation of input data created by the encoder.
Auto-Encoders	A neural network architecture for learning efficient data codings in an unsupervised manner. Utilizes encoder to compress data and decoder to reconstruct it, aiming to minimize the difference between the original and reconstructed data.

Feature Disentangle

Idea: what’s in the embedding? Extract the information of different aspects.

For example:

Discrete Representation

Vector Quantized Variational Auto-encoder (VQVAE)

Why TBATS?

Sat, 07 May 2022 00:00:00 +0000

I read Hyndman’s “Forecasting time series with complex seasonal patterns using exponential smoothing” paper again these days and provide some notes.

What is TBATS model?

TBATS: Trigonometric seasonality, Box-Cox transformation, ARMA errors, Trend and Seasonal components.

Motivation

For time series with Complex Seasonality, the common model like ETS, ARIMA, etc. CANNOT handle.

What do I mean Complex Seasonality?
- non-integer seasonality
  - eg. weekly time series w/ annual seasonal pattern 365.25/7 ≈ 52.18.
- multiple seasonality
  - eg. hourly time series.
What do I mean CANNOT handle?
- Too Many seasonal components lead to over-parameterization.

Idea behind TBATS

Idea: Modification of the state space models for exponential smoothing!

Box-Cox transformation enable “non-linearity”.
Use ARMA process to model error.
Trigonometric representation of seasonal components based on Fourier series.

BATS Model

\[ \begin{aligned}y_{t}^{(\omega)} &=\left\{\begin{array}{ll}\frac{y_{t}^{\omega}-1}{\omega} ; & \omega \neq 0 \\\log y_{t} & \omega=0\end{array}\right.\\y_{t}^{(\omega)} &=\ell_{t-1}+\phi b_{t-1}+\sum_{i=1}^{T} s_{t-m_{i}}^{(i)}+d_{t} \\\ell_{t} &=\ell_{t-1}+\phi b_{t-1}+\alpha d_{t} \\b_{t} &=(1-\phi) b+\phi b_{t-1}+\beta d_{t} \\s_{t}^{(i)} &=s_{t-m_{i}}^{(i)}+\gamma_{i} d_{t} \\d_{t} &=\sum_{i=1}^{p} \varphi_{i} d_{t-i}+\sum_{i=1}^{q} \theta_{i} \varepsilon_{t-i}+\varepsilon_{t}\end{aligned} \]

Note that:

The equations above do the following:
- Box-Cox transformation
- State Space Model framework
- level component
- trend component
- seasonal component
- ARMA process modeling error
$\alpha, \beta, \gamma$ are smoothing parameters.
$\phi$ is damping parameter for damping trend.
$\epsilon_t \sim \text{Gaussian-WN}(0, \sigma^2)$

REMARK:

BATS treats seasonal component as same as ETS model. This implies that BATS can only model integer period lengths. Approach taken in BATS requires m_i seed states for season i, if this season is long the model may become intractable.

Therefore BATS DOES NOT solve the complex seasonality problem!

TBATS Model

The difference between BATS and TBATS is the first letter “T”. Specifically, TBATS use the trigonometric representation of seasonal components based on Fourier series. Therefore, the first letter “T” is for “Trigonometric”.

BATS differs from TBATS only in the way it models seasonal effects.For TBATS, we use:

\[ \begin{aligned}s_{t}^{(i)} &=\sum_{j=1}^{k_{i}} s_{j, t}^{(i)} \\s_{j, t}^{(i)} &=s_{j, t-1}^{(i)} \cos \lambda_{j}^{(i)}+s_{j, t-1}^{*(i)} \sin \lambda_{j}^{(i)}+\gamma_{1}^{(i)} d_{t} \\s_{j, t}^{*(i)} &=-s_{j, t-1} \sin \lambda_{j}^{(i)}+s_{j, t-1}^{*(i)} \cos \lambda_{j}^{(i)}+\gamma_{2}^{(i)} d_{t}\end{aligned} \]

Note:

We describe the stochastic level of the $i$th seasonal component by $s_{j, t}^{(i)}$.
The stochastic growth in the level of the $i$th seasonal component that is needed to describe the change in the seasonal component over time by $s_{j, t}^{*(i)}$
The number of harmonics required for the $i$th seasonal component is denoted by $k_i$. When $k_i = m_i/2$ for even values of $m_i$, and $k_i = (m_i - 1)/2$ for odd values of $m_i$.
Why using Fourier terms (Harmonics)?
- It is anticipated that most seasonal components will require fewer harmonics, thus reducing the number of parameters to be estimated.
KEY Advantage:
- Use harmonics to reduce number of parameters to be estimated, solve over-parameterization.
- Based on trigonometric functions, it can be used to model non-integer seasonal frequencies.

How Does TBATS Choose The Final Model

Under the hood TBATS will consider various alternatives and fit quite a few models. It will consider models:

with Box-Cox transformation and without it.
with and without Trend
with and without Trend Damping
with and without ARMA(p,q) process used to model residuals
non-seasonal model
various amounts of harmonics used to model seasonal effects

The final model will be chosen using Akaike information criterion (AIC).

In particular auto ARMA is used to decide if residuals need modeling and what p and q values are suitable.

R Code

Loading package:

library(zetaEDA)
library(zeta.forecast)
library(forecast)
enable_zeta_ggplot_theme()

Check data:

autoplot(weeksale)

splits <- ts_split(weeksale, 8)
train <- splits$train
test <- splits$test

BATS/TBATS

For BATS Model,

$\operatorname{BATS}\left(\omega, \phi, p, q, m_{1}, m_{2}, \ldots, m_{T}\right)$

# bats model
bats_m <- bats(train)
bats_m

## BATS(0.354, {1,0}, -, {52})
## 
## Call: bats(y = train)
## 
## Parameters
##   Lambda: 0.353777
##   Alpha: 0.06931721
##   Gamma Values: -0.2472086
##   AR coefficients: 0.353693
## 
## Seed States:
##              [,1]
##  [1,] 236.2172862
##  [2,]  48.6153298
##  [3,]  25.6318979
##  [4,]  65.9402978
##  [5,]  48.5353739
##  [6,]  34.5740092
##  [7,]  32.2362991
##  [8,]  17.0571974
##  [9,]  18.2687693
## [10,]  12.1147339
## [11,]  17.6131427
## [12,]  -0.8316901
## [13,]  -1.5229147
## [14,]  12.8711144
## [15,] -14.7505548
## [16,] -19.1669848
## [17,] -19.8997841
## [18,] -21.1926193
## [19,] -23.4601100
## [20,] -20.8589143
## [21,] -22.8231038
## [22,] -28.4857736
## [23,] -28.1033726
## [24,] -32.0859166
## [25,] -26.6846780
## [26,] -22.8513308
## [27,] -35.7280360
## [28,] -42.0967584
## [29,] -13.6621107
## [30,]  18.7225542
## [31,]   3.6108027
## [32,] -18.6830362
## [33,]  -9.1773408
## [34,]  -1.2862120
## [35,]  -0.1671799
## [36,]  20.2180037
## [37,]   5.0885815
## [38,] -11.7724907
## [39,] -41.3620492
## [40,] -46.7859929
## [41,] -49.6365415
## [42,] -57.6488590
## [43,] -24.3202110
## [44,] -37.3709292
## [45,] -28.7887699
## [46,] -25.1689822
## [47,]   5.6126602
## [48,]   5.6972456
## [49,]  48.6994849
## [50,]  93.1440007
## [51,]  77.8410675
## [52,]  60.1611764
## [53,]  59.6051217
## [54,]   0.0000000
## attr(,"lambda")
## [1] 0.3537771
## 
## Sigma: 30.79541
## AIC: 8780.338

For TBATS Model,

$\operatorname{TBATS}\left(\omega, \phi, p, q,\left\{m_{1}, k_{1}\right\},\left\{m_{2}, k_{2}\right\}, \ldots,\left\{m_{T}, k_{T}\right\}\right)$

# tbats model
tbats_m <- tbats(train)
tbats_m

## TBATS(0.346, {0,1}, -, {<52.19,5>})
## 
## Call: tbats(y = train)
## 
## Parameters
##   Lambda: 0.345914
##   Alpha: 0.01716498
##   Gamma-1 Values: -0.009315079
##   Gamma-2 Values: 0.005982234
##   MA coefficients: 0.343452
## 
## Seed States:
##              [,1]
##  [1,] 177.5373756
##  [2,]  29.6240539
##  [3,]  17.7008739
##  [4,]  16.6389904
##  [5,]   0.5296389
##  [6,]  -0.3902185
##  [7,] -11.3794443
##  [8,] -17.1882094
##  [9,]   7.1278290
## [10,]   8.5281331
## [11,]   2.5042501
## [12,]   0.0000000
## attr(,"lambda")
## [1] 0.3459141
## 
## Sigma: 31.59406
## AIC: 8771.571

Check the KEY Difference!

Note that:

TBATS: The seed state for seasonal is $2 \cdot k_1 = 2 \cdot 5 = 10$.
BATS: The seed state for seasonal is $52$! Much larger than TBATS

Performance

Compare Accuracy:

bats_fc <- forecast(bats_m, h = length(test))
tbats_fc <- forecast(tbats_m, h = length(test))

accuracy(bats_fc, test)

##                    ME      RMSE      MAE          MPE        MAPE      MASE
## Training set 1199.837  89752.11 46368.39 -28256.17129 28276.14207 0.6546733
## Test set     5425.741 151518.01 92546.58    -41.36154    58.91754 1.3066612
##                    ACF1 Theil's U
## Training set -0.1925466        NA
## Test set     -0.2937480  1.204268

accuracy(tbats_fc, test)

##                     ME      RMSE      MAE          MPE        MAPE      MASE
## Training set  6471.137  99574.55 51738.22 -50245.07151 50265.01625 0.7304897
## Test set     36719.289 136648.52 68088.57    -15.87087    36.99544 0.9613396
##                     ACF1 Theil's U
## Training set -0.09246037        NA
## Test set      0.03653672  1.050218

plots:

bats_cv <- tssv_direct(weeksale, test_horizon = 8, forecast_model = bats_forecast)

tbats_cv <- tssv_direct(weeksale, test_horizon = 8, forecast_model = tbats_forecast)

plot_tssv(bats_cv, max_hist = 52) +
  labs(title = "BSTS 8-step Forecast for weeksale data")

plot_tssv(tbats_cv, max_hist = 52) +
  labs(title = "TBSTS 8-step Forecast for weeksale data")

Here, the tbats model does a better job in terms of RMSE criteria.

Tidymodels Ecosystem Tutorial

Thu, 10 Feb 2022 00:00:00 +0000

Tidymodels Ecosystem

I also hold this post on this RPub tutorial.

Introduction

First we should get a feeling of tidymodels ecosystem, and understand what each package does!

Note that:

Data Resampling and Feature Engineering: rsample, recipes
Model Fitting and Tuning: parsnip, tune, dials
Model Evaluation: yardstick

Big Picture

We will focus on the following packages although there are many more in the tidymodels ecosystem:

rsamples - to split the data into training and testing sets (as well as cross validation sets - more on that later!)
recipes - to prepare the data with preprocessing (assign variables and preprocessing steps)
parsnip - to specify and fit the data to a model
yardstick and tune - to evaluate model performance
workflows - combining recipe and parsnip objects into a workflow (this makes it easier to keep track of what you have done and it makes it easier to modify specific steps)
tune and dials - model optimization (more on what hyperparameters are later too!)
broom - to make the output from fitting a model easier to read

Here you can see a visual of how these packages work together in the process of performing a machine learning analysis:

To illustrate how to use each of these packages, we will work through some examples.

These are the major steps that we will cover in addition to some more advanced methods:

Other tidymodels packages include:

[source]

applicable compares new data points with the training data to see how much the new data points appear to be an extrapolation of the training data
baguette is for speeding up bagging pipelines
butcher is for dealing with pipelines that create model objects that take up too much memory
discrim has more model options for classification
embed has extra preprocessing options for categorical predictors
hardhat helps you to make new modeling packages
corrr has more options for looking at correlation matrices
rules has more model options for prediction rule ensembles
text recipes has extra preprocessing options for using text data
tidypredict is for running predictions inside SQL databases
modeldb is also for working within SQL databases and it allows for dplyr and tidyeval use within a database
tidyposterior compares models using resampling statistics

Most of these packages offer advanced modeling options and we will not be covering how to use them.

Machine Learning with Tidymodels

Note that the data, code and other materials are from Chen Xing’s Modeling with Tidymodels in R lectureNotes.

library(tidymodels)
library(tidyverse)
zetaEDA::enable_zeta_ggplot_theme()

## this below is for copy button in html code chunk
## ignore it if not need

## remotes::install_github("rlesur/klippy")
klippy::klippy(position = "right")

rsample: Creating training and test datasets

The goal is to learn rsample package!

Thersamplepackage is designed to create training and test datasets.

Creating a test dataset is important for estimating how a trained model will likely perform on new data. It also guards against overfitting, where a model memorizes patterns that exist only in the training data and performs poorly on new data.

home_sales data contains information on homes sold in the Seattle, Washington area between 2015 and 2016.

The outcome variable in this data is selling_price.

home_sales <- read_rds("data/home_sales.rds")
head(home_sales)

## # A tibble: 6 × 8
##   selling_price home_age bedrooms bathrooms sqft_living sqft_lot sqft_basement
##           <dbl>    <dbl>    <dbl>     <dbl>       <dbl>    <dbl>         <dbl>
## 1        487000       10        4      2.5         2540     5001             0
## 2        465000       10        3      2.25        1530     1245           480
## 3        411000       18        2      2           1130     1148           330
## 4        635000        4        3      2.5         3350     4007           800
## 5        380000       24        5      2.5         2130     8428             0
## 6        495000       21        3      3.5         1650     1577           550
## # … with 1 more variable: floors <dbl>

Generate training and testing data set.

Note that:

strata arg in initial_split() ensure that the random split with similar distribution of response variable.

?initial_split()

## Create a data split object
home_split <- initial_split(
  home_sales,
  prop = 0.7,
  ## stratification by outcome variable
  strata = selling_price
)

## Create the training data
home_training <- home_split %>%
  training()

## Create the test data
home_test <- home_split %>%
  testing()

## Check number of rows in each dataset
nrow(home_training)

## [1] 1042

nrow(home_test)

## [1] 450

Random sampling 得到的 training and testing set, distribution 是否一致呢？

## for training
home_training %>%
  summarize(across(selling_price, list(
    min = min,
    max = max,
    mean = mean,
    sd = sd
  )))

## # A tibble: 1 × 4
##   selling_price_min selling_price_max selling_price_mean selling_price_sd
##               <dbl>             <dbl>              <dbl>            <dbl>
## 1            350000            650000            478948.           80507.

## for testing
home_test %>%
  summarize(across(selling_price, list(
    min = min,
    max = max,
    mean = mean,
    sd = sd
  )))

## # A tibble: 1 × 4
##   selling_price_min selling_price_max selling_price_mean selling_price_sd
##               <dbl>             <dbl>              <dbl>            <dbl>
## 1            350000            650000            479401.           82151.

## note:
## Stratifying by the outcome variable ensures the model fitting process is performed on a representative sample of the original data.

distribution are quite similar!

parsnip: Fitting a linear regression model

The goal is to learn parsnip package!

Using parsnip to fit the model

## model setup
lm_mod <- linear_reg() %>%
  set_engine("lm") %>%
  set_mode("regression")

## check
lm_mod

## Linear Regression Model Specification (regression)
## 
## Computational engine: lm

model specification

{tidymodels}/{parsnip} Philosophy is to unify & make interfaces more predictable.
- Specify model type (e.g. linear regression, random forest …)
  - linear_reg()
  - rand_forest()
- Specify engine (i.e. package implementation of algorithm)
  - set_engine("some package's implementation")
- declare mode (e.g. classification vs linear regression)
  - use this when model can do both classification & regression
  - set_mode("regression")
  - set_mode("classification")

Modeling functions in parsnip separate model arguments into two categories:

Main arguments are more commonly used and tend to be available across engines.
Engine arguments are either specific to a particular engine or used more rarely.

Note:

一般来说，set_engine() 里面的参数就是包的名字，或者说给定包对应的主函数，例如："lm"。如果要修改这个主函数的默认参数，也在set_engine()里修改，使用的是set_engine(...)的功能。
如果要修改某个方法的通用参数，（所谓通用参数，就是说这个参数和你用哪个包是无关的），那么在specify model type 的函数里给定，例如：rand_forest(trees = 1000, min_n = 5)

Fitting model

lm_fit <- lm_mod %>%
  fit(selling_price ~ home_age + sqft_living, data = home_training)

## check
lm_fit

## parsnip model object
## 
## 
## Call:
## stats::lm(formula = selling_price ~ home_age + sqft_living, data = data)
## 
## Coefficients:
## (Intercept)     home_age  sqft_living  
##    290498.6      -1609.4        103.9

Obtain the estimated parameters

tidy(lm_fit)

## # A tibble: 3 × 5
##   term        estimate std.error statistic   p.value
##   <chr>          <dbl>     <dbl>     <dbl>     <dbl>
## 1 (Intercept)  290868.   7592.       38.3  6.10e-201
## 2 home_age      -1452.    173.       -8.40 1.49e- 16
## 3 sqft_living     103.      2.74     37.7  1.75e-196

Generate predictions

## using fitting model to get prediction
lm_pred <- lm_fit %>%
  predict(new_data = home_test)

head(lm_pred)

## # A tibble: 6 × 1
##     .pred
##     <dbl>
## 1 538413.
## 2 433433.
## 3 378981.
## 4 428202.
## 5 408554.
## 6 530970.

bind result

home_test_res <- home_test %>%
  select(selling_price, sqft_living, home_age) %>%
  bind_cols(lm_pred)

head(home_test_res)

## # A tibble: 6 × 4
##   selling_price sqft_living home_age   .pred
##           <dbl>       <dbl>    <dbl>   <dbl>
## 1        495000        1650       21 430724.
## 2        425000        1920       11 473124.
## 3        495000        2140        3 507457.
## 4        559900        2930       20 564331.
## 5        552321        1960       29 451111.
## 6        525000        2450       28 503154.

yardstick: Evaluate Model Performance

The goal is to learn yardstick package!

Input to yardstick functions:

all yardstick functions requires a tibble with columns:

true outcome (实际值)
model predictions, .pred, (预测值)

For RMSE and R^2,

## calculate RMSE
home_test_res %>%
  rmse(truth = selling_price, estimate = .pred)

## # A tibble: 1 × 3
##   .metric .estimator .estimate
##   <chr>   <chr>          <dbl>
## 1 rmse    standard      47624.

## calculate R^2
home_test_res %>%
  rsq(truth = selling_price, estimate = .pred)

## # A tibble: 1 × 3
##   .metric .estimator .estimate
##   <chr>   <chr>          <dbl>
## 1 rsq     standard       0.646

Visualize $R^2$ plot:

coord_obs_pred() can be used in a ggplot to make the x- and y-axes have the same exact scale along with an aspect ratio of one.

?coord_obs_pred
## note that the `coord_obs_pred` is from `tune` package!

home_test_res %>%
  ggplot(aes(x = selling_price, y = .pred)) +
  geom_point(alpha = .5) +
  geom_abline(color = "blue", linetype = 2) +
  coord_obs_pred() +
  labs(x = "Actual Home Selling Price", y = "Predicted Selling Price")

Streamlining Model Fitting

?last_fit()

last_fit() takes a model specification, model formula, and data split object Performs the following:

Creates training and test datasets
Fits the model to the training data
Calculates metrics and predictions on the test data
Returns an object with all results

## define linear regression model
lm_mod <- linear_reg() %>%
  set_engine("lm") %>%
  set_mode("regression")

## Train linear_model with last_fit()
lm_fit <- lm_mod %>%
  last_fit(selling_price ~ ., split = home_split)

## check
lm_fit

## # Resampling results
## # Manual resampling 
## # A tibble: 1 × 6
##   splits             id               .metrics .notes   .predictions .workflow 
##   <list>             <chr>            <list>   <list>   <list>       <list>    
## 1 <split [1042/450]> train/test split <tibble> <tibble> <tibble>     <workflow>

Collect predictions and view results

?tune::collect_predictions()

## Collect predictions and view results
predictions_df <- lm_fit %>% collect_predictions()

## check
head(predictions_df)

## # A tibble: 6 × 5
##   id                 .pred  .row selling_price .config             
##   <chr>              <dbl> <int>         <dbl> <chr>               
## 1 train/test split 527930.     1        487000 Preprocessor1_Model1
## 2 train/test split 424149.     2        465000 Preprocessor1_Model1
## 3 train/test split 399071.     3        411000 Preprocessor1_Model1
## 4 train/test split 438494.     6        495000 Preprocessor1_Model1
## 5 train/test split 400976.     7        355000 Preprocessor1_Model1
## 6 train/test split 478599.    10        535000 Preprocessor1_Model1

Question: when should I use last_fit() function?

After carefully reading the help file of the last_fit() function, the answer is obvious.

当我们尝试拟合了不同的模型，以及完成了hyper-parameter tuning，从而找到了最满意的模型。那么最后一步就是将这个模型重新在training set上拟合一遍，然后看看它在testing set 上的表现情况。

last_fit() 的目的就是帮助我们完成这最后的一步！

Classification Models

We will be working with the telecom_df dataset which contains information on customers of a telecommunications company. The outcome variable is canceled_service and it records whether a customer canceled their contract with the company. The predictor variables contain information about customers’ cell phone and internet usage as well as their contract type and monthly charges.

telecom_df <- read_rds("data/telecom_df.rds")

## check
head(telecom_df)

## # A tibble: 6 × 9
##   canceled_service cellular_service avg_data_gb avg_call_mins avg_intl_mins
##   <fct>            <fct>                  <dbl>         <dbl>         <dbl>
## 1 yes              single_line             7.78           497           127
## 2 yes              single_line             9.04           336            88
## 3 no               single_line            10.3            262            55
## 4 yes              multiple_lines          5.08           250           107
## 5 no               multiple_lines          8.05           328           122
## 6 no               single_line             9.3            326           114
## # … with 4 more variables: internet_service <fct>, contract <fct>,
## #   months_with_company <dbl>, monthly_charges <dbl>

Train/Test Set

Using rsample package again to create training and testing sets.

telecom_split <- initial_split(
  data = telecom_df,
  prop = .75,
  strata = canceled_service
)

## for training
telecom_training <- telecom_split %>%
  training()

## for testing
telecom_test <- telecom_split %>%
  testing()

## check
nrow(telecom_training)

## [1] 731

nrow(telecom_test)

## [1] 244

Model Fitting

Using parsnip package to fit a logistic regression model,

## Specify a logistic regression model
logistic_model <- logistic_reg() %>%
  ## Set the engine
  set_engine("glm") %>%
  ## Set the mode
  set_mode("classification")

## Fit to training data
logistic_fit <- logistic_model %>%
  fit(
    canceled_service ~ avg_call_mins + avg_intl_mins + monthly_charges,
    data = telecom_training
  )

## Print model fit object
logistic_fit

## parsnip model object
## 
## 
## Call:  stats::glm(formula = canceled_service ~ avg_call_mins + avg_intl_mins + 
##     monthly_charges, family = stats::binomial, data = data)
## 
## Coefficients:
##     (Intercept)    avg_call_mins    avg_intl_mins  monthly_charges  
##        2.607512        -0.011708         0.023125        -0.002036  
## 
## Degrees of Freedom: 730 Total (i.e. Null);  727 Residual
## Null Deviance:       932.4 
## Residual Deviance: 794.2     AIC: 802.2

Generate Prediction

## Predict outcome categories
class_preds <- predict(
  logistic_fit,
  new_data = telecom_test,
  type = "class"
)

## Obtain estimated probabilities for each outcome value
prob_preds <- predict(
  logistic_fit,
  new_data = telecom_test,
  type = "prob"
)

## Combine test set results
telecom_results <- telecom_test %>%
  select(canceled_service) %>%
  bind_cols(class_preds, prob_preds)

## View results tibble
telecom_results %>%
  head()

## # A tibble: 6 × 4
##   canceled_service .pred_class .pred_yes .pred_no
##   <fct>            <fct>           <dbl>    <dbl>
## 1 no               no              0.354    0.646
## 2 no               no              0.226    0.774
## 3 yes              yes             0.813    0.187
## 4 yes              yes             0.539    0.461
## 5 yes              yes             0.583    0.417
## 6 yes              no              0.152    0.848

Assessing Model Fitting

Confusion Matrix: Matrix with counts of all combinations of actual and predicted outcome values.

Correct Predictions

True Positive (TP)
True Negative (TN)

Classification Errors

False Positive (FP)
False Negative (FN)

Confusion Matrix

## Calculate the confusion matrix
yardstick::conf_mat(
  telecom_results,
  truth = canceled_service,
  estimate = .pred_class
)

##           Truth
## Prediction yes  no
##        yes  37  25
##        no   45 137

Some Metrics:

$accuracy = \frac{TP + TN}{TP + FP + TN + FN}$, is classification accuracy.
$sensitivity = \frac{TP}{TP + FP}$, is the proportion of all positive cases that were correctly classified.
$specificity = \frac{TN}{TN + FN}$, is the proportion of all negative cases that were correctly classified.
$false \ positive \ rate\ (FPR) \ = 1 - specificity$, is the proportion of false positives among true negatives.

## Calculate the accuracy
yardstick::accuracy(
  telecom_results,
  truth = canceled_service,
  estimate = .pred_class
)

## # A tibble: 1 × 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy binary         0.713

## Calculate the sensitivity
yardstick::sens(
  telecom_results,
  truth = canceled_service,
  estimate = .pred_class
)

## # A tibble: 1 × 3
##   .metric .estimator .estimate
##   <chr>   <chr>          <dbl>
## 1 sens    binary         0.451

## Calculate the specificity
yardstick::spec(
  telecom_results,
  truth = canceled_service,
  estimate = .pred_class
)

## # A tibble: 1 × 3
##   .metric .estimator .estimate
##   <chr>   <chr>          <dbl>
## 1 spec    binary         0.846

Instead of calculating accuracy, sensitivity, and specificity separately, you can create your own metric function that calculates all three at the same time.

## Create a custom metric function
telecom_metrics <- metric_set(
  yardstick::accuracy,
  yardstick::sens,
  yardstick::spec
)

## Calculate metrics using model results tibble
telecom_metrics(
  telecom_results,
  truth = canceled_service,
  estimate = .pred_class
)

## # A tibble: 3 × 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy binary         0.713
## 2 sens     binary         0.451
## 3 spec     binary         0.846

Calculate Many metrics all together,

## Create a confusion matrix
conf_mat(
  telecom_results,
  truth = canceled_service,
  estimate = .pred_class
) %>%
  ## Pass to the summary() function
  summary()

## # A tibble: 13 × 3
##    .metric              .estimator .estimate
##    <chr>                <chr>          <dbl>
##  1 accuracy             binary         0.713
##  2 kap                  binary         0.316
##  3 sens                 binary         0.451
##  4 spec                 binary         0.846
##  5 ppv                  binary         0.597
##  6 npv                  binary         0.753
##  7 mcc                  binary         0.322
##  8 j_index              binary         0.297
##  9 bal_accuracy         binary         0.648
## 10 detection_prevalence binary         0.254
## 11 precision            binary         0.597
## 12 recall               binary         0.451
## 13 f_meas               binary         0.514

Visualize Model Performance

Plotting a confusion matrix,

conf_mat(
  telecom_results,
  truth = canceled_service,
  estimate = .pred_class
) %>%
  ## Create a heat map
  autoplot(type = "heatmap")

conf_mat(
  telecom_results,
  truth = canceled_service,
  estimate = .pred_class
) %>%
  ## Create a mosaic map
  autoplot(type = "mosaic")

ROC curves and area under the ROC curve

roc auc plot

Summarizing the ROC curve

The area under the ROC curve (ROC AUC) captures the ROC curve information of a classification model in a single number

Useful interpretation as a le2er grade of classification performance

A - [0.9, 1
B - [0.8, 0.9)
C - [0.7, 0.8)
D - [0.6, 0.7)
F - [0.5, 0.6)

## Plot ROC curve
telecom_results %>%
  ## Calculate metrics across thresholds
  roc_curve(
    truth = canceled_service,
    estimate = .pred_yes
  ) %>%
  autoplot()

## Calculate the ROC AUC
telecom_results %>%
  roc_auc(
    truth = canceled_service,
    estimate = .pred_yes
  )

## # A tibble: 1 × 3
##   .metric .estimator .estimate
##   <chr>   <chr>          <dbl>
## 1 roc_auc binary         0.713

Automatic Tidymodels Workflow

## Train model with last_fit()
telecom_last_fit <- logistic_model %>%
  last_fit(
    ## formula
    canceled_service ~ avg_call_mins + avg_intl_mins + monthly_charges,
    split = telecom_split
  )

## View test set metrics
telecom_last_fit %>%
  collect_metrics()

## # A tibble: 2 × 4
##   .metric  .estimator .estimate .config             
##   <chr>    <chr>          <dbl> <chr>               
## 1 accuracy binary         0.713 Preprocessor1_Model1
## 2 roc_auc  binary         0.713 Preprocessor1_Model1

## Collect predictions
last_fit_results <- telecom_last_fit %>%
  collect_predictions()

## View results
last_fit_results %>%
  head()

## # A tibble: 6 × 7
##   id               .pred_yes .pred_no  .row .pred_class canceled_service .config
##   <chr>                <dbl>    <dbl> <int> <fct>       <fct>            <chr>  
## 1 train/test split     0.354    0.646     3 no          no               Prepro…
## 2 train/test split     0.226    0.774     6 no          no               Prepro…
## 3 train/test split     0.813    0.187     7 yes         yes              Prepro…
## 4 train/test split     0.539    0.461     9 yes         yes              Prepro…
## 5 train/test split     0.583    0.417    12 yes         yes              Prepro…
## 6 train/test split     0.152    0.848    14 no          yes              Prepro…

## Custom metrics function
last_fit_metrics <- metric_set(
  yardstick::accuracy,
  yardstick::sens,
  yardstick::spec,
  yardstick::roc_auc
)

## Calculate metrics
last_fit_metrics(
  last_fit_results,
  truth = canceled_service,
  estimate = .pred_class,
  ## note that roc_auc needs predicted probability
  .pred_yes
)

## # A tibble: 4 × 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy binary         0.713
## 2 sens     binary         0.451
## 3 spec     binary         0.846
## 4 roc_auc  binary         0.713

Now let’s try if we can improved the model by add one more regressor.

## new fitted model
new_fit <- logistic_model %>%
  last_fit(
    canceled_service ~ avg_call_mins + avg_intl_mins + monthly_charges + months_with_company,
    split = telecom_split
  )

## collection metrics, the accuracy and roc_auc indeed improved
new_fit %>%
  collect_metrics()

## # A tibble: 2 × 4
##   .metric  .estimator .estimate .config             
##   <chr>    <chr>          <dbl> <chr>               
## 1 accuracy binary         0.803 Preprocessor1_Model1
## 2 roc_auc  binary         0.827 Preprocessor1_Model1

## collection predictions
new_pred <- new_fit %>%
  collect_predictions()

## check ROC curve
new_pred %>%
  roc_curve(truth = canceled_service, estimate = .pred_yes) %>%
  autoplot()

Feature Engineering

Goal is to learn recipe package.

step 1: Specify variables, recipe(y ~ a+ b + …, data = dat)
step 2: Define pre-precessing steps, step_*()
step 3: Provide dataset(s) for recipe steps, prep()
step 4: Apply Pre-precessing, bake()

Creating recipe objects

The first step in feature engineering is to specify a recipeobject with the recipe() function and add data preprocessing steps with one or more step_*() functions. Storing all of this information in a single recipe object makes it easier to manage complex feature engineering pipelines and transform new data sources.

## Specify feature engineering recipe
telecom_log_rec <- recipe(
  ## define the response variable
  canceled_service ~ .,
  data = telecom_training
) %>%
  ## Add log transformation step
  step_log(avg_call_mins, avg_intl_mins, base = 10)

## View variable roles and data types
telecom_log_rec %>%
  summary()

## # A tibble: 9 × 4
##   variable            type    role      source  
##   <chr>               <chr>   <chr>     <chr>   
## 1 cellular_service    nominal predictor original
## 2 avg_data_gb         numeric predictor original
## 3 avg_call_mins       numeric predictor original
## 4 avg_intl_mins       numeric predictor original
## 5 internet_service    nominal predictor original
## 6 contract            nominal predictor original
## 7 months_with_company numeric predictor original
## 8 monthly_charges     numeric predictor original
## 9 canceled_service    nominal outcome   original

The next step in the feature engineering process is to train your recipe object using the training data. Then you will be able to apply your trained recipe to both the training and test datasets in order to prepare them for use in model fitting and model evaluation.

## Train the telecom_log_rec object
telecom_log_rec_prep <- telecom_log_rec %>%
  prep(training = telecom_training)

## View results
telecom_log_rec_prep

## Recipe
## 
## Inputs:
## 
##       role #variables
##    outcome          1
##  predictor          8
## 
## Training data contained 731 data points and no missing data.
## 
## Operations:
## 
## Log transformation on avg_call_mins, avg_intl_mins [trained]

## apply pre-processing on training set
telecom_log_rec_prep %>%
  bake(new_data = NULL)

## # A tibble: 731 × 9
##    cellular_service avg_data_gb avg_call_mins avg_intl_mins internet_service
##    <fct>                  <dbl>         <dbl>         <dbl> <fct>           
##  1 multiple_lines          8.05          2.52          2.09 digital         
##  2 multiple_lines          9.4           2.49          2.17 fiber_optic     
##  3 multiple_lines          9.96          2.53          2.13 fiber_optic     
##  4 multiple_lines         10.2           2.60          2.06 fiber_optic     
##  5 single_line             6.69          2.55          1.96 digital         
##  6 multiple_lines          4.11          2.57          1.81 digital         
##  7 multiple_lines          5.17          2.53          2.08 digital         
##  8 single_line             8.67          1.97          2.12 fiber_optic     
##  9 multiple_lines          9.24          2.59          2.15 fiber_optic     
## 10 multiple_lines         11.0           2.59          1.89 fiber_optic     
## # … with 721 more rows, and 4 more variables: contract <fct>,
## #   months_with_company <dbl>, monthly_charges <dbl>, canceled_service <fct>

## apply pre-processing on testing set
telecom_log_rec_prep %>%
  bake(new_data = telecom_test)

## # A tibble: 244 × 9
##    cellular_service avg_data_gb avg_call_mins avg_intl_mins internet_service
##    <fct>                  <dbl>         <dbl>         <dbl> <fct>           
##  1 single_line            10.3           2.42          1.74 fiber_optic     
##  2 single_line             9.3           2.51          2.06 fiber_optic     
##  3 multiple_lines          8.01          2.72          1.99 fiber_optic     
##  4 single_line             5.29          2.62          1.98 digital         
##  5 single_line             6.23          2.63          1.98 fiber_optic     
##  6 single_line             7.07          2.40          1.97 fiber_optic     
##  7 single_line             9.37          2.58          1.94 fiber_optic     
##  8 multiple_lines         10.6           2.45          2.17 fiber_optic     
##  9 multiple_lines          7.86          2.58          2.21 digital         
## 10 single_line             8.04          2.46          1.67 fiber_optic     
## # … with 234 more rows, and 4 more variables: contract <fct>,
## #   months_with_company <dbl>, monthly_charges <dbl>, canceled_service <fct>

Numeric Predictors

Fix multicolinearity problems caused by highly correlated predictors.

telecom_training %>%
  select(where(is.numeric)) %>%
  cor()

##                     avg_data_gb avg_call_mins avg_intl_mins months_with_company
## avg_data_gb           1.0000000   0.195307838     0.1565105         0.386500807
## avg_call_mins         0.1953078   1.000000000     0.0737045         0.006322915
## avg_intl_mins         0.1565105   0.073704496     1.0000000         0.222895844
## months_with_company   0.3865008   0.006322915     0.2228958         1.000000000
## monthly_charges       0.9561583   0.189384250     0.1665923         0.407137668
##                     monthly_charges
## avg_data_gb               0.9561583
## avg_call_mins             0.1893842
## avg_intl_mins             0.1665923
## months_with_company       0.4071377
## monthly_charges           1.0000000

telecom_training %>%
  ggplot(aes(x = avg_data_gb, y = monthly_charges)) +
  geom_point() +
  labs(title = "Muticolinearity Example")

use step_corr() and its argument threshold

Selecting predictors by type using:

all_outcomes(): Selects the outcome variable
all_numeric(): Selects all numeric variables (could include outcome variable if it is numeric)

## Specify a recipe object
telecom_cor_rec <- recipe(
  ## define outcome variable
  canceled_service ~ .,
  data = telecom_training
) %>%
  ## Remove correlated variables
  step_corr(all_numeric_predictors(), threshold = 0.8)

## Train the recipe
telecom_cor_rec_prep <- telecom_cor_rec %>%
  prep(training = telecom_training)

## Apply to training data
telecom_cor_rec_prep %>%
  bake(new_data = NULL)

## # A tibble: 731 × 8
##    cellular_service avg_data_gb avg_call_mins avg_intl_mins internet_service
##    <fct>                  <dbl>         <dbl>         <dbl> <fct>           
##  1 multiple_lines          8.05           328           122 digital         
##  2 multiple_lines          9.4            312           147 fiber_optic     
##  3 multiple_lines          9.96           340           136 fiber_optic     
##  4 multiple_lines         10.2            402           116 fiber_optic     
##  5 single_line             6.69           352            91 digital         
##  6 multiple_lines          4.11           371            64 digital         
##  7 multiple_lines          5.17           341           119 digital         
##  8 single_line             8.67            93           131 fiber_optic     
##  9 multiple_lines          9.24           392           142 fiber_optic     
## 10 multiple_lines         11.0            390            78 fiber_optic     
## # … with 721 more rows, and 3 more variables: contract <fct>,
## #   months_with_company <dbl>, canceled_service <fct>

## Apply to test data
telecom_cor_rec_prep %>%
  bake(new_data = telecom_test)

## # A tibble: 244 × 8
##    cellular_service avg_data_gb avg_call_mins avg_intl_mins internet_service
##    <fct>                  <dbl>         <dbl>         <dbl> <fct>           
##  1 single_line            10.3            262            55 fiber_optic     
##  2 single_line             9.3            326           114 fiber_optic     
##  3 multiple_lines          8.01           525            97 fiber_optic     
##  4 single_line             5.29           417            96 digital         
##  5 single_line             6.23           429            96 fiber_optic     
##  6 single_line             7.07           249            94 fiber_optic     
##  7 single_line             9.37           382            87 fiber_optic     
##  8 multiple_lines         10.6            281           147 fiber_optic     
##  9 multiple_lines          7.86           378           164 digital         
## 10 single_line             8.04           290            47 fiber_optic     
## # … with 234 more rows, and 3 more variables: contract <fct>,
## #   months_with_company <dbl>, canceled_service <fct>

For Normalization calling step_normalize

Normalization will center and scale numeric variable, i.e. subtract mean and divide by the standard deviation.

## Specify a recipe object
telecom_norm_rec <- recipe(
  canceled_service ~ .,
  data = telecom_training
) %>%
  ## Remove correlated variables
  step_corr(all_numeric(), threshold = 0.8) %>%
  ## Normalize numeric predictors
  step_normalize(all_numeric())

## Train the recipe
telecom_norm_rec_prep <- telecom_norm_rec %>%
  prep(training = telecom_training)

## Apply to test data
telecom_norm_rec_prep %>%
  bake(new_data = telecom_test)

## # A tibble: 244 × 8
##    cellular_service avg_data_gb avg_call_mins avg_intl_mins internet_service
##    <fct>                  <dbl>         <dbl>         <dbl> <fct>           
##  1 single_line            1.08         -1.15         -1.73  fiber_optic     
##  2 single_line            0.550        -0.300         0.206 fiber_optic     
##  3 multiple_lines        -0.126         2.34         -0.352 fiber_optic     
##  4 single_line           -1.55          0.906        -0.385 digital         
##  5 single_line           -1.06          1.07         -0.385 fiber_optic     
##  6 single_line           -0.619        -1.32         -0.450 fiber_optic     
##  7 single_line            0.587         0.442        -0.680 fiber_optic     
##  8 multiple_lines         1.26         -0.896         1.29  fiber_optic     
##  9 multiple_lines        -0.205         0.389         1.84  digital         
## 10 single_line           -0.111        -0.777        -1.99  fiber_optic     
## # … with 234 more rows, and 3 more variables: contract <fct>,
## #   months_with_company <dbl>, canceled_service <fct>

Nominal Predictors

Dummy variable Encoding: Excludes one value from original set of data values, i.e. n distinct values produce (n-1) indicator variables.

use step_dummy() function

Selecting columns by type: all_nominal()

## Create a recipe that predicts canceled_service using the training data
telecom_recipe <- recipe(
  canceled_service ~ .,
  data = telecom_training
) %>%
  ## Remove correlated predictors
  step_corr(all_numeric(), threshold = .8) %>%
  ## Normalize numeric predictors
  step_normalize(all_numeric()) %>%
  ## Create dummy variables
  step_dummy(all_nominal(), -all_outcomes())

## Train your recipe and apply it to the test data
telecom_recipe %>%
  prep(train = telecom_training) %>%
  bake(new_data = telecom_test)

## # A tibble: 244 × 9
##    avg_data_gb avg_call_mins avg_intl_mins months_with_company canceled_service
##          <dbl>         <dbl>         <dbl>               <dbl> <fct>           
##  1       1.08         -1.15         -1.73               0.654  no              
##  2       0.550        -0.300         0.206             -0.342  no              
##  3      -0.126         2.34         -0.352             -0.581  yes             
##  4      -1.55          0.906        -0.385             -1.02   yes             
##  5      -1.06          1.07         -0.385             -0.0231 yes             
##  6      -0.619        -1.32         -0.450             -0.581  yes             
##  7       0.587         0.442        -0.680             -1.18   no              
##  8       1.26         -0.896         1.29               0.813  no              
##  9      -0.205         0.389         1.84              -0.421  no              
## 10      -0.111        -0.777        -1.99               0.534  yes             
## # … with 234 more rows, and 4 more variables:
## #   cellular_service_single_line <dbl>, internet_service_digital <dbl>,
## #   contract_one_year <dbl>, contract_two_year <dbl>

Complete Model Workflow

## 1. feature engineering-----------------------------------
telecom_recipe <- recipe(
  canceled_service ~ .,
  data = telecom_training
) %>%
  ## Removed correlated predictors
  step_corr(all_numeric(), threshold = 0.8) %>%
  ## Log transform numeric predictors
  step_log(all_numeric(), base = 10) %>%
  ## Normalize numeric predictors
  step_normalize(all_numeric()) %>%
  ## Create dummy variables
  step_dummy(all_nominal(), -all_outcomes())

## Train recipe
telecom_recipe_prep <- telecom_recipe %>%
  prep(training = telecom_training)

## Apply pre-processing recipe on training data
telecom_training_prep <- telecom_recipe_prep %>%
  bake(new_data = NULL)

## Apply pre-processing recipe on test data
telecom_test_prep <- telecom_recipe_prep %>%
  bake(new_data = telecom_test)

telecom_test_prep

## # A tibble: 244 × 9
##    avg_data_gb avg_call_mins avg_intl_mins months_with_company canceled_service
##          <dbl>         <dbl>         <dbl>               <dbl> <fct>           
##  1     1.00           -1.06         -1.99              0.717   no              
##  2     0.591          -0.165         0.323             0.207   no              
##  3     0.00120         1.79         -0.190             0.00453 yes             
##  4    -1.64            0.845        -0.223            -0.633   yes             
##  5    -0.991           0.961        -0.223             0.411   yes             
##  6    -0.492          -1.27         -0.290             0.00453 yes             
##  7     0.620           0.485        -0.536            -1.14    no              
##  8     1.13           -0.775         1.13              0.774   no              
##  9    -0.0734          0.442         1.48              0.145   no              
## 10     0.0160         -0.645        -2.49              0.672   yes             
## # … with 234 more rows, and 4 more variables:
## #   cellular_service_single_line <dbl>, internet_service_digital <dbl>,
## #   contract_one_year <dbl>, contract_two_year <dbl>

## 2. Model Fitting-----------------------------------

## Train logistic model
logistic_fit <- logistic_model %>%
  fit(canceled_service ~ ., data = telecom_training_prep)

## Obtain class predictions
class_preds <- predict(
  logistic_fit,
  new_data = telecom_test_prep,
  type = "class"
)

## Obtain estimated probabilities
prob_preds <- predict(
  logistic_fit,
  new_data = telecom_test_prep,
  type = "prob"
)

## Combine test set results
telecom_results <- telecom_test_prep %>%
  select(canceled_service) %>%
  bind_cols(class_preds, prob_preds)

telecom_results

## # A tibble: 244 × 4
##    canceled_service .pred_class .pred_yes .pred_no
##    <fct>            <fct>           <dbl>    <dbl>
##  1 no               no             0.169     0.831
##  2 no               no             0.233     0.767
##  3 yes              yes            0.844     0.156
##  4 yes              no             0.379     0.621
##  5 yes              no             0.440     0.560
##  6 yes              no             0.153     0.847
##  7 no               yes            0.753     0.247
##  8 no               no             0.0197    0.980
##  9 no               no             0.0877    0.912
## 10 yes              no             0.422     0.578
## # … with 234 more rows

## 3. Model Performance Metric-----------------------------------

## Create a confusion matrix
telecom_results %>%
  yardstick::conf_mat(truth = canceled_service, estimate = .pred_class)

##           Truth
## Prediction yes  no
##        yes  45  19
##        no   37 143

## Calculate sensitivity
telecom_results %>%
  yardstick::sens(truth = canceled_service, estimate = .pred_class)

## # A tibble: 1 × 3
##   .metric .estimator .estimate
##   <chr>   <chr>          <dbl>
## 1 sens    binary         0.549

## Calculate specificity
telecom_results %>%
  yardstick::spec(truth = canceled_service, estimate = .pred_class)

## # A tibble: 1 × 3
##   .metric .estimator .estimate
##   <chr>   <chr>          <dbl>
## 1 spec    binary         0.883

## Plot ROC curve
telecom_results %>%
  yardstick::roc_curve(truth = canceled_service, .pred_yes) %>%
  autoplot()

Comparing with the results from Assessing Model Fitting, we can find that after feature engineering, the model perform better!

Additional Note

当学完 workflow 包之后，我们在回看 recipe 包的时候，可能会存在疑问：

prep() 和 bake() 到底具有什么意义，什么时候用？这个问题tmwr书进行里说明，Max Kuhn 也讲过。一般来说，recipe object 是被直接用在 workflow 里面的，i.e.

wflow_obj <- 
  workflow() %>% 
  add_model(...) %>% 
  add_recipe(...)

当需要debug的时候，使用prep() 和 bake()。

From Max Kuhn’s lecture notes, we learned that:

90% of the time, you will want to use a workflow to estimate and apply a recipe.
If you have an error, the original recipe object can be estimated manually with a function called prep() (analogous to fit()). This returns the fitted recipe. This can help debug any issues.
Another function (bake()) is analogous to predict() and gives you the processed data back.

The tidymodels book has more details on debugging.

Machine Learning Workflows

Recall that:

parsnip pkg is for Model Specification.
recipe pkg is for feature engineering.

We can combine models and recipes together! This is also the motivation for workflows package.

Combining Models and Recipes

Goal: learn workflows package.

The workflows package is designed for streamlining the model process. That is, combines a parsnip model and recipe object into a single workflow object.

Initialized with the workflow() function:

Add model object with add_model()
Add recipe object with add_recipe()
- Here must be specification, not a trained recipe.

Example code will use the new dataset, loans_df

loans_df <- read_rds("data/loan_df.rds")

set.seed("123")
## Create data split object
loans_split <- initial_split(
  data = loans_df,
  strata = loan_default
)

## Build training data
loans_training <- loans_split %>%
  training()

## Build test data
loans_test <- loans_split %>%
  testing()

## Check for correlated predictors
loans_training %>%
  ## Select numeric columns
  select_if(is.numeric) %>%
  ## Calculate correlation matrix
  cor()

##                 loan_amount interest_rate installment annual_income
## loan_amount     1.000000000  -0.009750748   0.9384548    0.36894798
## interest_rate  -0.009750748   1.000000000   0.0329387   -0.08849002
## installment     0.938454822   0.032938704   1.0000000    0.30850322
## annual_income   0.368947977  -0.088490024   0.3085032    1.00000000
## debt_to_income  0.138955303   0.133949453   0.1987249   -0.21357794
##                debt_to_income
## loan_amount         0.1389553
## interest_rate       0.1339495
## installment         0.1987249
## annual_income      -0.2135779
## debt_to_income      1.0000000

This time, we’ll use decision tree model,

dt_model <- decision_tree() %>%
  ## Specify the engine
  set_engine("rpart") %>%
  ## Specify the mode
  set_mode("classification")

## Build feature engineering pipeline
loans_recipe <- recipe(
  loan_default ~ .,
  data = loans_training
) %>%
  ## Correlation filter
  step_corr(all_numeric(), threshold = .85) %>%
  ## Normalize numeric predictors
  step_normalize(all_numeric()) %>%
  ## Create dummy variables
  step_dummy(all_nominal(), -all_outcomes())

loans_recipe

## Recipe
## 
## Inputs:
## 
##       role #variables
##    outcome          1
##  predictor          7
## 
## Operations:
## 
## Correlation filter on all_numeric()
## Centering and scaling for all_numeric()
## Dummy variables from all_nominal(), -all_outcomes()

workflow objects simplify the modeling process in tidymodels. With workflows, it’s possible to train a parsnip model and recipe object at the same time.

## Create a workflow
loans_dt_wkfl <- workflow() %>%
  ## Include the model object
  add_model(dt_model) %>%
  ## Include the recipe object
  add_recipe(loans_recipe)

## check
loans_dt_wkfl

## ══ Workflow ════════════════════════════════════════════════════════════════════
## Preprocessor: Recipe
## Model: decision_tree()
## 
## ── Preprocessor ────────────────────────────────────────────────────────────────
## 3 Recipe Steps
## 
## • step_corr()
## • step_normalize()
## • step_dummy()
## 
## ── Model ───────────────────────────────────────────────────────────────────────
## Decision Tree Model Specification (classification)
## 
## Computational engine: rpart

Model fitting with workflows

Training a workflow object

Pass workflow to last_fit() and provide data split object
View model evaluation results with collect_metrics()

Behind the scenes:

Training and test datasets created
recipe trained and applied
Decision tree trained with training data
Predictions and metrics on test data

## Train the workflow
loans_dt_wkfl_fit <- loans_dt_wkfl %>%
  last_fit(split = loans_split)

## check
loans_dt_wkfl_fit

## # Resampling results
## # Manual resampling 
## # A tibble: 1 × 6
##   splits            id               .metrics .notes   .predictions .workflow 
##   <list>            <chr>            <list>   <list>   <list>       <list>    
## 1 <split [653/219]> train/test split <tibble> <tibble> <tibble>     <workflow>

## Calculate performance metrics on test data
loans_dt_wkfl_fit %>%
  collect_metrics()

## # A tibble: 2 × 4
##   .metric  .estimator .estimate .config             
##   <chr>    <chr>          <dbl> <chr>               
## 1 accuracy binary         0.804 Preprocessor1_Model1
## 2 roc_auc  binary         0.857 Preprocessor1_Model1

Cross Validation

The vfold_cv() function
- Trining data
- Number of folds, v
- Stratification variable, strata
- Execute set.seed() before vfold_cv() for reporducibility
- splits is a list column with data split objects fore creating fold
The fit_resamplies() function
- Train a parsnip model or workflow object
- Provide cross validation folds, resamples
- Optional custom metric function, metrics
- Models trained with fit_resamples() are not able to provide predictions on new data sources
  - predict() function does not accept resample objects
- Purpose of fit_resample()
  - Explore and compare the performance profile of different model types
  - Select best performing model type and focus on model fitting efforts

Example is,

## Create cross validation folds
set.seed(1234)
loans_folds <- vfold_cv(
  loans_training,
  v = 5,
  strata = loan_default
)

## check
loans_folds

## #  5-fold cross-validation using stratification 
## # A tibble: 5 × 2
##   splits            id   
##   <list>            <chr>
## 1 <split [521/132]> Fold1
## 2 <split [522/131]> Fold2
## 3 <split [523/130]> Fold3
## 4 <split [523/130]> Fold4
## 5 <split [523/130]> Fold5

## Create custom metrics function
loans_metrics <- metric_set(
  yardstick::roc_auc,
  yardstick::sens,
  yardstick::spec
)

## Fit resamples
loans_dt_rs <- loans_dt_wkfl %>%
  fit_resamples(
    resamples = loans_folds,
    metrics = loans_metrics
  )

## View performance metrics
loans_dt_rs %>%
  collect_metrics()

## # A tibble: 3 × 6
##   .metric .estimator  mean     n std_err .config             
##   <chr>   <chr>      <dbl> <int>   <dbl> <chr>               
## 1 roc_auc binary     0.835     5 0.00982 Preprocessor1_Model1
## 2 sens    binary     0.675     5 0.0434  Preprocessor1_Model1
## 3 spec    binary     0.860     5 0.0262  Preprocessor1_Model1

Now, let’s try cross validation using logistic regression model,

logistic_model <- logistic_reg() %>%
  ## Specify the engine
  set_engine("glm") %>%
  ## Specify the mode
  set_mode("classification")

## Create workflow
loans_logistic_wkfl <- workflow() %>%
  ## Add model
  add_model(logistic_model) %>%
  ## Add recipe
  add_recipe(loans_recipe)

## Fit resamples
loans_logistic_rs <- loans_logistic_wkfl %>%
  fit_resamples(
    resamples = loans_folds,
    metrics = loans_metrics
  )

## View performance metrics
loans_logistic_rs %>%
  collect_metrics()

## # A tibble: 3 × 6
##   .metric .estimator  mean     n std_err .config             
##   <chr>   <chr>      <dbl> <int>   <dbl> <chr>               
## 1 roc_auc binary     0.848     5  0.0112 Preprocessor1_Model1
## 2 sens    binary     0.643     5  0.0284 Preprocessor1_Model1
## 3 spec    binary     0.863     5  0.0186 Preprocessor1_Model1

Comparing model performance profiles

The benefit of the collect_metrics() function is that it returns a tibble of cross validation results. This makes it easy to calculate custom summary statistics with the dplyr package.

## Detailed cross validation results
dt_rs_results <- loans_dt_rs %>%
  ## summarize = FALSE will provide all metric estimates for every cross validation fold
  collect_metrics(summarize = FALSE)

## Explore model performance for decision tree
dt_rs_results %>%
  group_by(.metric) %>%
  summarize(
    across(.estimate, list(
      min = min,
      median = median,
      max = max
    ), .names = "{.fn}")
  )

## # A tibble: 3 × 4
##   .metric   min median   max
##   <chr>   <dbl>  <dbl> <dbl>
## 1 roc_auc 0.812  0.826 0.867
## 2 sens    0.549  0.706 0.78 
## 3 spec    0.775  0.862 0.938

## Detailed cross validation results
logistic_rs_results <- loans_logistic_rs %>%
  collect_metrics(summarize = FALSE)

## Explore model performance for logistic regression
logistic_rs_results %>%
  group_by(.metric) %>%
  summarize(
    across(.estimate, list(
      min = min,
      median = median,
      max = max
    ), .names = "{.fn}")
  )

## # A tibble: 3 × 4
##   .metric   min median   max
##   <chr>   <dbl>  <dbl> <dbl>
## 1 roc_auc 0.828  0.834 0.888
## 2 sens    0.549  0.66  0.72 
## 3 spec    0.812  0.877 0.9

Hyperparameter tuning

Hyperparameters: Model parameters whose values are set prior to model training and control model complexity.

Hyperparameter tuning: Process of using cross validation to find the optimal set of hyperparameter values.

Goal: learn tune package.

To label hyperparameters for for tuning, set them equal to tune() in parsnip model specification
Create model object with tuning parameters will let other functions know that they need to be optimized

需要调整哪个参数，就tune()哪个，在model specification 中设置：

## Set tuning hyperparameters
dt_tune_model <- decision_tree(
  cost_complexity = tune(),
  tree_depth = tune(),
  min_n = tune()
) %>%
  ## Specify engine
  set_engine("rpart") %>%
  ## Specify mode
  set_mode("classification")

## check
dt_tune_model

## Decision Tree Model Specification (classification)
## 
## Main Arguments:
##   cost_complexity = tune()
##   tree_depth = tune()
##   min_n = tune()
## 
## Computational engine: rpart

workflow objects can be easily updated. For example, update_model provide new decision tree model with tuning parameters.

## Create a tuning workflow
loans_tune_wkfl <- loans_dt_wkfl %>%
  ## Replace model
  update_model(dt_tune_model)

loans_tune_wkfl

## ══ Workflow ════════════════════════════════════════════════════════════════════
## Preprocessor: Recipe
## Model: decision_tree()
## 
## ── Preprocessor ────────────────────────────────────────────────────────────────
## 3 Recipe Steps
## 
## • step_corr()
## • step_normalize()
## • step_dummy()
## 
## ── Model ───────────────────────────────────────────────────────────────────────
## Decision Tree Model Specification (classification)
## 
## Main Arguments:
##   cost_complexity = tune()
##   tree_depth = tune()
##   min_n = tune()
## 
## Computational engine: rpart

Random grid search

The most common method of hyperparameter tuning is grid search. This method creates a tuning grid with unique combinations of hyperparameter values and uses cross validation to evaluate their performance. The goal of hyperparameter tuning is to find the optimal combination of values for maximizing model performance.

Goal: learn dials package.

use parameters()to identify hyperparameters
use grid_random() to generate random combinations
- first arg is the results of parameters() function
- size arg sets the number of random combinatitions to generate
use tune_gird() to perform hyperparameter tuning, need args:
- workflow or parsnip model
- resamples, cross validation object
- grid, tuning grid
- metrics function is Optional

## Hyperparameter tuning with grid search
set.seed(214)
dt_grid <- grid_random(
  parameters(dt_tune_model),
  size = 5
)

## Hyperparameter tuning
dt_tuning <- loans_tune_wkfl %>%
  tune_grid(
    ## cv resample data
    resamples = loans_folds,
    grid = dt_grid,
    metrics = loans_metrics
  )

## View results
dt_tuning %>%
  collect_metrics()

## # A tibble: 15 × 9
##    cost_complexity tree_depth min_n .metric .estimator  mean     n std_err
##              <dbl>      <int> <int> <chr>   <chr>      <dbl> <int>   <dbl>
##  1    0.0000000758         14    39 roc_auc binary     0.832     5 0.00832
##  2    0.0000000758         14    39 sens    binary     0.699     5 0.0485 
##  3    0.0000000758         14    39 spec    binary     0.820     5 0.0329 
##  4    0.0243                5    34 roc_auc binary     0.765     5 0.00824
##  5    0.0243                5    34 sens    binary     0.619     5 0.0195 
##  6    0.0243                5    34 spec    binary     0.910     5 0.00842
##  7    0.00000443           11     8 roc_auc binary     0.808     5 0.0119 
##  8    0.00000443           11     8 sens    binary     0.683     5 0.0246 
##  9    0.00000443           11     8 spec    binary     0.810     5 0.0120 
## 10    0.000000600           3     5 roc_auc binary     0.765     5 0.00563
## 11    0.000000600           3     5 sens    binary     0.572     5 0.0273 
## 12    0.000000600           3     5 spec    binary     0.925     5 0.00692
## 13    0.00380               5    36 roc_auc binary     0.825     5 0.0109 
## 14    0.00380               5    36 sens    binary     0.623     5 0.0331 
## 15    0.00380               5    36 spec    binary     0.875     5 0.0228 
## # … with 1 more variable: .config <chr>

## Collect detailed tuning results
dt_tuning_results <- dt_tuning %>%
  collect_metrics(summarize = FALSE)

## Explore detailed ROC AUC results for each fold
dt_tuning_results %>%
  filter(.metric == "roc_auc") %>%
  group_by(id) %>%
  summarize(
    min_roc_auc = min(.estimate),
    median_roc_auc = median(.estimate),
    max_roc_auc = max(.estimate)
  )

## # A tibble: 5 × 4
##   id    min_roc_auc median_roc_auc max_roc_auc
##   <chr>       <dbl>          <dbl>       <dbl>
## 1 Fold1       0.744          0.787       0.818
## 2 Fold2       0.748          0.802       0.819
## 3 Fold3       0.776          0.807       0.858
## 4 Fold4       0.77           0.789       0.849
## 5 Fold5       0.759          0.805       0.853

Selecting the best model

show_best() function displays the top n models based on average value of metric
select_best() function will select the metric on which to evaluate performance, and returns a tibble with the best performing model and hyperparameter values
finalize_workflow() function will finalize a workflow that contains a model object with tuning parameters. It will return a workflow object with set hyperparameter values
last_filt() will
- Train and test datasets created
- recipe trained and applied
- Tuned model trained with entire triaining dataset
- Predictions and metrics on test data

## Display 5 best performing models
dt_tuning %>%
  show_best(metric = "roc_auc", n = 5)

## # A tibble: 5 × 9
##   cost_complexity tree_depth min_n .metric .estimator  mean     n std_err
##             <dbl>      <int> <int> <chr>   <chr>      <dbl> <int>   <dbl>
## 1    0.0000000758         14    39 roc_auc binary     0.832     5 0.00832
## 2    0.00380               5    36 roc_auc binary     0.825     5 0.0109 
## 3    0.00000443           11     8 roc_auc binary     0.808     5 0.0119 
## 4    0.0243                5    34 roc_auc binary     0.765     5 0.00824
## 5    0.000000600           3     5 roc_auc binary     0.765     5 0.00563
## # … with 1 more variable: .config <chr>

## Select based on best performance
best_dt_model <- dt_tuning %>%
  ## Choose the best model based on roc_auc
  select_best(metric = "roc_auc")

## check
best_dt_model

## # A tibble: 1 × 4
##   cost_complexity tree_depth min_n .config             
##             <dbl>      <int> <int> <chr>               
## 1    0.0000000758         14    39 Preprocessor1_Model1

## Finalize your workflow
final_loans_wkfl <- loans_tune_wkfl %>%
  finalize_workflow(best_dt_model)

final_loans_wkfl

## ══ Workflow ════════════════════════════════════════════════════════════════════
## Preprocessor: Recipe
## Model: decision_tree()
## 
## ── Preprocessor ────────────────────────────────────────────────────────────────
## 3 Recipe Steps
## 
## • step_corr()
## • step_normalize()
## • step_dummy()
## 
## ── Model ───────────────────────────────────────────────────────────────────────
## Decision Tree Model Specification (classification)
## 
## Main Arguments:
##   cost_complexity = 7.58290839567418e-08
##   tree_depth = 14
##   min_n = 39
## 
## Computational engine: rpart

## Train finalized decision tree workflow
loans_final_fit <- final_loans_wkfl %>%
  last_fit(split = loans_split)

## View performance metrics
loans_final_fit %>%
  collect_metrics()

## # A tibble: 2 × 4
##   .metric  .estimator .estimate .config             
##   <chr>    <chr>          <dbl> <chr>               
## 1 accuracy binary         0.763 Preprocessor1_Model1
## 2 roc_auc  binary         0.850 Preprocessor1_Model1

## Create an ROC curve
loans_final_fit %>%
  ## Collect predictions
  collect_predictions() %>%
  ## Calculate ROC curve metrics
  roc_curve(truth = loan_default, .pred_yes) %>%
  ## Plot the ROC curve
  autoplot()

Reference

Tidy Modeling with R books free online.
- Tidy Modeling with R Book Club，这个是tidy modeling with r 书的辅导材料，加入了总结，很不错
ISLR tidymodels Labs code example using tidymodels.
An Introduction to Statistical Learning 2nd version newest book
Tidyverse Skills for Data Science-Chapter 5.13

XGBoost in R Tutorial

Wed, 12 Jan 2022 00:00:00 +0000

Note that the original material is from Beginners Tutorial on XGBoost and Parameter Tuning in R.

Introduction

Last week, we learned about Random Forest Algorithm. Now we know it helps us reduce a model’s variance by building models on resampled data and thereby increases its generalization capability. Good!

Now, you might be wondering, what to do next for increasing a model’s prediction accuracy ? After all, an ideal model is one which is good at both generalization and prediction accuracy. This brings us to Boosting Algorithms.

Developed in 1989, the family of boosting algorithms has been improved over the years. In this article, we’ll learn about XGBoost algorithm.

XGBoost is the most popular machine learning algorithm these days. Regardless of the data type (regression or classification), it is well known to provide better solutions than other ML algorithms. In fact, since its inception (early 2014), it has become the “true love” of kaggle users to deal with structured data. So, if you are planning to compete on Kaggle, xgboost is one algorithm you need to master.

In this article, you’ll learn about core concepts of the XGBoost algorithm. In addition, we’ll look into its practical side, i.e., improving the xgboost model using parameter tuning in R.

What is XGBoost? Why is it so good?
How does XGBoost work?
Understanding XGBoost Tuning Parameters
Practical - Tuning XGBoost using R

What is XGBoost ? Why is it so good ?

XGBoost (Extreme Gradient Boosting) is an optimized distributed gradient boosting library. Yes, it uses gradient boosting (GBM) framework at core. Yet, does better than GBM framework alone. XGBoost was created by Tianqi Chen, PhD Student, University of Washington. It is used for supervised ML problems. Let’s look at what makes it so good:

Parallel Computing: It is enabled with parallel processing (using OpenMP); i.e., when you run xgboost, by default, it would use all the cores of your laptop/machine.
Regularization: I believe this is the biggest advantage of xgboost. GBM has no provision for regularization. Regularization is a technique used to avoid overfitting in linear and tree-based models.
Enabled Cross Validation: In R, we usually use external packages such as caret and mlr to obtain CV results. But, xgboost is enabled with internal CV function (we’ll see below).
Missing Values: XGBoost is designed to handle missing values internally. The missing values are treated in such a manner that if there exists any trend in missing values, it is captured by the model.
Flexibility: In addition to regression, classification, and ranking problems, it supports user-defined objective functions also. An objective function is used to measure the performance of the model given a certain set of parameters. Furthermore, it supports user defined evaluation metrics as well.
Availability: Currently, it is available for programming languages such as R, Python, Java, Julia, and Scala.
Save and Reload: XGBoost gives us a feature to save our data matrix and model and reload it later. Suppose, we have a large data set, we can simply save the model and use it in future instead of wasting time redoing the computation.
Tree Pruning: Unlike GBM, where tree pruning stops once a negative loss is encountered, XGBoost grows the tree upto max_depth and then prune backward until the improvement in loss function is below a threshold.

I’m sure now you are excited to master this algorithm. But remember, with great power comes great difficulties too. You might learn to use this algorithm in a few minutes, but optimizing it is a challenge. Don’t worry, we shall look into it in following sections.

How does XGBoost work ?

XGBoost belongs to a family of boosting algorithms that convert weak learners into strong learners. A weak learner is one which is slightly better than random guessing. Let’s understand boosting first (in general).

Boosting is a sequential process; i.e., trees are grown using the information from a previously grown tree one after the other. This process slowly learns from data and tries to improve its prediction in subsequent iterations. Let’s look at a classic classification example:

Four classifiers (in 4 boxes), shown above, are trying hard to classify + and - classes as homogeneously as possible. Let’s understand this picture well.

Box 1: The first classifier creates a vertical line (split) at D1. It says anything to the left of D1 is + and anything to the right of D1 is -. However, this classifier misclassifies three + points.
Box 2: The next classifier says don’t worry I will correct your mistakes. Therefore, it gives more weight to the three + misclassified points (see bigger size of +) and creates a vertical line at D2. Again it says, anything to right of D2 is - and left is +. Still, it makes mistakes by incorrectly classifying three - points.
Box 3: The next classifier continues to bestow support. Again, it gives more weight to the three - misclassified points and creates a horizontal line at D3. Still, this classifier fails to classify the points (in circle) correctly.
Remember that each of these classifiers has a misclassification error associated with them.
Boxes 1,2, and 3 are weak classifiers. These classifiers will now be used to create a strong classifier Box 4.
Box 4: It is a weighted combination of the weak classifiers. As you can see, it does good job at classifying all the points correctly.

That’s the basic idea behind boosting algorithms. The very next model capitalizes on the misclassification/error of previous model and tries to reduce it. Now, let’s come to XGBoost.

As we know, XGBoost can used to solve both regression and classification problems. It is enabled with separate methods to solve respective problems. Let’s see:

Classification Problems: To solve such problems, it uses booster = gbtree parameter; i.e., a tree is grown one after other and attempts to reduce misclassification rate in subsequent iterations. In this, the next tree is built by giving a higher weight to misclassified points by the previous tree (as explained above).

Regression Problems: To solve such problems, we have two methods: booster = gbtree and booster = gblinear. You already know gbtree. In gblinear, it builds generalized linear model and optimizes it using regularization (L1,L2) and gradient descent. In this, the subsequent models are built on residuals (actual - predicted) generated by previous iterations. Are you wondering what is gradient descent? Understanding gradient descent requires math, however, let me try and explain it in simple words:

Gradient Descent: It is a method which comprises a vector of weights (or coefficients) where we calculate their partial derivative with respective to zero. The motive behind calculating their partial derivative is to find the local minima of the loss function (RSS), which is convex in nature. In simple words, gradient descent tries to optimize the loss function by tuning different values of coefficients to minimize the error.

Hopefully, up till now, you have developed a basic intuition around how boosting and xgboost works. Let’s proceed to understand its parameters. After all, using xgboost without parameter tuning is like driving a car without changing its gears; you can never up your speed.

Note: In R, xgboost package uses a matrix of input data instead of a data frame.

Understanding XGBoost Tuning Parameters

Every parameter has a significant role to play in the model’s performance. Before hypertuning, let’s first understand about these parameters and their importance. In this article, I’ve only explained the most frequently used and tunable parameters. To look at all the parameters, you can refer to its official documentation.

XGBoost parameters can be divided into three categories (as suggested by its authors):

General Parameters: Controls the booster type in the model which eventually drives overall functioning
Booster Parameters: Controls the performance of the selected booster
Learning Task Parameters: Sets and evaluates the learning process of the booster from the given data

1. General Parameters

Booster[default=gbtree]
- Sets the booster type (gbtree, gblinear or dart) to use. For classification problems, you can use gbtree, dart. For regression, you can use any.
nthread[default=maximum cores available]
- Activates parallel computation. Generally, people don’t change it as using maximum cores leads to the fastest computation.
silent[default=0]
- If you set it to 1, your R console will get flooded with running messages. Better not to change it.

2. Booster Parameters

As mentioned above, parameters for tree and linear boosters are different. Let’s understand each one of them:

Parameters for Tree Booster

nrounds[default=100]
- It controls the maximum number of iterations. For classification, it is similar to the number of trees to grow.
- Should be tuned using CV
eta[default=0.3][range: (0,1)]
- It controls the learning rate, i.e., the rate at which our model learns patterns in data. After every round, it shrinks the feature weights to reach the best optimum.
- Lower eta leads to slower computation. It must be supported by increase in nrounds.
- Typically, it lies between 0.01 - 0.3
gamma[default=0][range: (0,Inf)]
- It controls regularization (or prevents overfitting). The optimal value of gamma depends on the data set and other parameter values.
- Higher the value, higher the regularization. Regularization means penalizing large coefficients which don’t improve the model’s performance. default = 0 means no regularization.
- Tune trick: Start with 0 and check CV error rate. If you see train error >>> test error, bring gamma into action. Higher the gamma, lower the difference in train and test CV. If you have no clue what value to use, use gamma=5 and see the performance. Remember that gamma brings improvement when you want to use shallow (low max_depth) trees.
max_depth[default=6][range: (0,Inf)]
- It controls the depth of the tree.
- Larger the depth, more complex the model; higher chances of overfitting. There is no standard value for max_depth. Larger data sets require deep trees to learn the rules from data.
- Should be tuned using CV
min_child_weight[default=1][range:(0,Inf)]
- In regression, it refers to the minimum number of instances required in a child node. In classification, if the leaf node has a minimum sum of instance weight (calculated by second order partial derivative) lower than min_child_weight, the tree splitting stops.
- In simple words, it blocks the potential feature interactions to prevent overfitting. Should be tuned using CV.
subsample[default=1][range: (0,1)]
- It controls the number of samples (observations) supplied to a tree.
- Typically, its values lie between (0.5-0.8)
colsample_bytree[default=1][range: (0,1)]
- It control the number of features (variables) supplied to a tree
- Typically, its values lie between (0.5,0.9)
lambda[default=0]
- It controls L2 regularization (equivalent to Ridge regression) on weights. It is used to avoid overfitting.
alpha[default=1]
- It controls L1 regularization (equivalent to Lasso regression) on weights. In addition to shrinkage, enabling alpha also results in feature selection. Hence, it’s more useful on high dimensional data sets.

Parameters for Linear Booster

Using linear booster has relatively lesser parameters to tune, hence it computes much faster than gbtree booster.

nrounds[default=100]
- It controls the maximum number of iterations (steps) required for gradient descent to converge.
- Should be tuned using CV
lambda[default=0]
- It enables Ridge Regression. Same as above
alpha[default=1]
- It enables Lasso Regression. Same as above

3. Learning Task Parameters

These parameters specify methods for the loss function and model evaluation. In addition to the parameters listed below, you are free to use a customized objective / evaluation function.

Objective[default=reg:linear]
- reg:linear - for linear regression
- binary:logistic - logistic regression for binary classification. It returns class probabilities
- multi:softmax - multiclassification using softmax objective. It returns predicted class labels. It requires setting num_classparameter denoting number of unique prediction classes.
- multi:softprob - multiclassification using softmax objective. It returns predicted class probabilities.
eval_metric [no default, depends on objective selected]
- These metrics are used to evaluate a model’s accuracy on validation data. For regression, default metric is RMSE. For classification, default metric is error.
- Available error functions are as follows:
  - mae - Mean Absolute Error (used in regression)
  - Logloss - Negative loglikelihood (used in classification)
  - AUC - Area under curve (used in classification)
  - RMSE - Root mean square error (used in regression)
  - error - Binary classification error rate [#wrong cases/#all cases]
  - mlogloss - multiclass logloss (used in classification)

We’ve looked at how xgboost works, the significance of each of its tuning parameter, and how it affects the model’s performance. Let’s bolster our newly acquired knowledge by solving a practical problem in R.

Practical - Tuning XGBoost in R

In this practical section, we’ll learn to tune xgboost in two ways: using the xgboost package and MLR package. I don’t see the xgboost R package having any inbuilt feature for doing grid/random search. To overcome this bottleneck, we’ll use MLR to perform the extensive parametric search and try to obtain optimal accuracy.

I’ll use the adult data set from my previous random forest tutorial. This data set poses a classification problem where our job is to predict if the given user will have a salary <=50K or >50K.

Using random forest, we achieved an accuracy of 85.8%. Theoretically, xgboost should be able to surpass random forest’s accuracy. Let’s see if we can do it. I’ll follow the most common but effective steps in parameter tuning:

First, you build the xgboost model using default parameters. You might be surprised to see that default parameters sometimes give impressive accuracy.
If you get a depressing model accuracy, do this: fix eta = 0.1, leave the rest of the parameters at default value, using xgb.cv function get best n_rounds. Now, build a model with these parameters and check the accuracy.
Otherwise, you can perform a grid search on rest of the parameters (max_depth, gamma, subsample, colsample_bytree etc) by fixing eta and nrounds. Note: If using gbtree, don’t introduce gamma until you see a significant difference in your train and test error.
Using the best parameters from grid search, tune the regularization parameters(alpha,lambda) if required.
At last, increase/decrease eta and follow the procedure. But remember, excessively lower eta values would allow the model to learn deep interactions in the data and in this process, it might capture noise. So be careful!

This process might sound a bit complicated, but it’s quite easy to code in R. Don’t worry, I’ve demonstrated all the steps below. Let’s get into actions now and quickly prepare our data for modeling (if you don’t understand any line of code, ask me in comments):

#set working directory
> path <- "~/December 2016/XGBoost_Tutorial"
> setwd(path)

#load libraries
> library(data.table)
> library(mlr)

#set variable names
setcol <- c("age",` `"workclass",` `"fnlwgt",` `"education",` `"education-num",` `"marital-status",` `"occupation",` `"relationship",` `"race",` `"sex",` `"capital-gain",` `"capital-loss",` `"hours-per-week",` `"native-country",` `"target")

#load data
> train <- read.table("adultdata.txt", header = F, sep = ",", col.names = setcol, na.strings = c(" ?"), stringsAsFactors = F)
> test <- read.table("adulttest.txt",header = F,sep = ",",col.names = setcol,skip = 1, na.strings = c(" ?"),stringsAsFactors = F)

#convert data frame to data table
> setDT(train) 
> setDT(test)

#check missing values 
> table(is.na(train))
> sapply(train, function(x) sum(is.na(x))/length(x))*100

> table(is.na(test))
> sapply(test, function(x) sum(is.na(x))/length(x))*100

#quick data cleaning
#remove extra character from target variable
> library(stringr)
> test [,target := substr(target,start = 1,stop = nchar(target)-1)]

#remove leading whitespaces
> char_col <- colnames(train)[ sapply (test,is.character)]
> for(i in char_col) set(train,j=i,value = str_trim(train[[i]],side = "left"))

> for(i in char_col) set(test,j=i,value = str_trim(test[[i]],side = "left"))

#set all missing value as "Missing" 
> train[is.na(train)] <- "Missing" 
> test[is.na(test)] <- "Missing"

Up to this point, we dealt with basic data cleaning and data inconsistencies. To use xgboost package, keep these things in mind:

Convert the categorical variables into numeric using one hot encoding
For classification, if the dependent variable belongs to class factor, convert it to numeric

R’s base function model.matrix is quick enough to implement one hot encoding. In the code below, ~.+0 leads to encoding of all categorical variables without producing an intercept. Alternatively, you can use the dummies package to accomplish the same task. Since xgboost package accepts target variable separately, we’ll do the encoding keeping this in mind:

#using one hot encoding 
> labels <- train$target 
> ts_label <- test$target
> new_tr <- model.matrix(~.+0,data = train[,-c("target"),with=F]) 
> new_ts <- model.matrix(~.+0,data = test[,-c("target"),with=F])

#convert factor to numeric 
> labels <- as.numeric(labels)-1
> ts_label <- as.numeric(ts_label)-1

For xgboost, we’ll use xgb.DMatrix to convert data table into a matrix (most recommended):

#preparing matrix 
> dtrain <- xgb.DMatrix(data = new_tr,label = labels) 
> dtest <- xgb.DMatrix(data = new_ts,label=ts_label)

As mentioned above, we’ll first build our model using default parameters, keeping random forest’s accuracy 85.8% in mind. I’ll capture the default parameters from above (written against every parameter):

#default parameters
> params <- list(booster = "gbtree", objective = "binary:logistic", eta=0.3, gamma=0, max_depth=6, min_child_weight=1, subsample=1, colsample_bytree=1)

Using the inbuilt xgb.cv function, let’s calculate the best nround for this model. In addition, this function also returns CV error, which is an estimate of test error.

> xgbcv <- xgb.cv( params = params, data = dtrain, nrounds = 100, nfold = 5, showsd = T, stratified = T, print.every.n = 10, early.stop.round = 20, maximize = F)
##best iteration = 79

The model returned lowest error at the 79th (nround) iteration. Also, if you noticed the running messages in your console, you would have understood that train and test error are following each other. We’ll use this insight in the following code. Now, we’ll see our CV error:

> min(xgbcv$test.error.mean)
[1]#0.1263

As compared to my previous random forest model, this CV accuracy (100-12.63)=87.37% looks better already. However, I believe cross-validation accuracy is usually more optimistic than true test accuracy. Let’s calculate our test set accuracy and determine if this default model makes sense:

#first default - model training
> xgb1 <- xgb.train (params = params, data = dtrain, nrounds = 79, watchlist = list(val=dtest,train=dtrain), print.every.n = 10, early.stop.round = 10, maximize = F , eval_metric = "error")
#model prediction
> xgbpred <- predict (xgb1,dtest)
> xgbpred <- ifelse (xgbpred > 0.5,1,0)

The objective function binary:logistic returns output probabilities rather than labels. To convert it, we need to manually use a cutoff value. As seen above, I’ve used 0.5 as my cutoff value for predictions. We can calculate our model’s accuracy using confusionMatrix()function from caret package.

#confusion matrix
> library(caret)
> confusionMatrix (xgbpred, ts_label)
#Accuracy - 86.54%` 

#view variable importance plot
> mat <- xgb.importance (feature_names = colnames(new_tr),model = xgb1)
> xgb.plot.importance (importance_matrix = mat[1:20])

As you can see, we’ve achieved better accuracy than a random forest model using default parameters in xgboost. Can we still improve it? Let’s proceed to the random / grid search procedure and attempt to find better accuracy. From here on, we’ll be using the MLR package for model building. A quick reminder, the MLR package creates its own frame of data, learner as shown below. Also, keep in mind that task functions in mlr doesn’t accept character variables. Hence, we need to convert them to factors before creating task:

#convert characters to factors
> fact_col <- colnames(train)[sapply(train,is.character)]

> for(i in fact_col) set(train,j=i,value = factor(train[[i]]))
> for (i in fact_col) set(test,j=i,value = factor(test[[i]]))

#create tasks
> traintask <- makeClassifTask (data = train,target = "target")
> testtask <- makeClassifTask (data = test,target = "target")

#do one hot encoding`<br/> 
> traintask <- createDummyFeatures (obj = traintask,target = "target") 
> testtask <- createDummyFeatures (obj = testtask,target = "target")

Now, we’ll set the learner and fix the number of rounds and eta as discussed above.

#create learner
> lrn <- makeLearner("classif.xgboost",predict.type = "response")
> lrn$par.vals <- list( objective="binary:logistic", eval_metric="error", nrounds=100L, eta=0.1)

#set parameter space
> params <- makeParamSet( makeDiscreteParam("booster",values = c("gbtree","gblinear")), makeIntegerParam("max_depth",lower = 3L,upper = 10L), makeNumericParam("min_child_weight",lower = 1L,upper = 10L), makeNumericParam("subsample",lower = 0.5,upper = 1), makeNumericParam("colsample_bytree",lower = 0.5,upper = 1))

#set resampling strategy
> rdesc <- makeResampleDesc("CV",stratify = T,iters=5L)

With stratify=T, we’ll ensure that distribution of target class is maintained in the resampled data sets. If you’ve noticed above, in the parameter set, I didn’t consider gamma for tuning. Simply because during cross validation, we saw that train and test error are in sync with each other. Had either one of them been dragging or rushing, we could have brought this parameter into action.

Now, we’ll set the search optimization strategy. Though, xgboost is fast, instead of grid search, we’ll use random search to find the best parameters. In random search, we’ll build 10 models with different parameters, and choose the one with the least error. You are free to build any number of models.

#search strategy
> ctrl <- makeTuneControlRandom(maxit = 10L)

We’ll also set a parallel backend to ensure faster computation. Make sure you’ve not opened several applications in backend. We’ll use all the cores in your machine.

#set parallel backend
> library(parallel)
> library(parallelMap) 
> parallelStartSocket(cpus = detectCores())

#parameter tuning
mytune <- tuneParams(learner = lrn, task = traintask, resampling = rdesc, measures = acc, par.set = params, control = ctrl, show.info = T)
> mytune$y 
#0.873069

This newly obtained tuned CV accuracy is better than our default xgboost model. To check the tuning result, write mytune in your R console and press Enter. Let’s build a model using tuned parameters and check the final test accuracy.

#set hyperparameters
> lrn_tune <- setHyperPars(lrn,par.vals = mytune$x)

#train model
> xgmodel <- train(learner = lrn_tune,task = traintask)

#predict model
> xgpred <- predict(xgmodel,testtask)

We’ve made our predictions on the test set. Let’s check our model’s accuracy.

> confusionMatrix(xgpred$data$response,xgpred$data$truth)
#Accuracy : 0.8747

As you see, we’ve achieved a better accuracy than our default xgboost model (86.45%). For tuning the xgboost model, always remember that simple tuning leads to better predictions. I’ll leave you here.

What’s next? If you are still curious to improve the model’s accuracy, update eta, find the best parameters using random search and build the model. Tell me in comments if you’ve achieved better accuracy. Here is the complete github script for code shared above.

Summary

This article is meant to help beginners in machine learning quickly learn the xgboost algorithm. If this article makes you want to learn more, I suggest you to read this paper published by its author. I would like to thank kaggler laurae whose valuable discussion helped me a lot in understanding xgboost tuning.

As I said in the beginning, learning how to run xgboost is easy. The real challenge lies in understanding what happens behind the code. I hope this article gave you enough information to help you build your next xgboost model better. In this article, I discussed the basics of the boosting algorithm and how xgboost implements it in an efficient manner. Also, we learned how to build models using xgboost with parameter tuning in R.

Feel free to drop in your comments, experiences, and knowledge gathered while building models using xgboost.

Change Point Detection in R

Fri, 31 Dec 2021 00:00:00 +0000

What is the difference between change point and outlier?

To answer this question, we should really understand what is a change point for a time series.

Changepoints are also known as:

breakpoints
segmentation
structural breaks
regime switching
detecting disorder

and can be found in a wide range of literature including

quality control
economics
medicine
environment
linguistics
$\cdots$

For data $y_1, \ldots, y_n$, if a changepoint exists at $\tau$, then $y_1,\ldots,y_{\tau}$ differ from $y_{\tau+1},\ldots,y_n$ in some way.

There are many different types of change.

Thus a changepoint model for a change in mean has the following formulation:

\[ y_t = \left\{ \begin{array}{lcl} \mu_1 & \mbox{if} & 1\leq t \leq \tau_1 \\ \mu_2 & \mbox{if} & \tau_1 < t \leq \tau_2 \\ \vdots & & \vdots \\ \mu_{m+1} & \mbox{if} & \tau_m < t \leq \tau_{m+1}=n \end{array} \right. \]

What is the goal?

Has a change occurred?
If yes, where is the change?
What is the difference between the pre and post change data?
- Maybe this is the type of change
- Maybe it is the parameter values before and after the change
What is the probability that a change has occurred?
How certain are we of the changepoint location?
How many changes have occurred (+ all the above for each change)?
Why has there been a change?

Online vs Offline

Online
- Processes data as it arrives or in batches
- Goal is quickest detection of a change
- Often used in processing control, intrusion detection
Offline
- Processes all the data in one go
- Goal is accurate detection of a change
- Often used in genome analysis, audiology

change point detection function using `ecp` package

library(zeta.forecast)
zetaEDA::enable_zeta_ggplot_theme()

plot_ts_change_point(eg_diamond_ts)

## # A tibble: 2 × 3
##   time        value cpt  
##   <date>      <dbl> <fct>
## 1 2013-12-01 652769 yes  
## 2 2019-09-01 305745 yes

M4 Forecast Competition Summary

Fri, 10 Dec 2021 00:00:00 +0000

Last Updated

last update date: 2021-12-20

Details Table

Note that the user_id in the table is the folder number in M4-methods URL: https://github.com/Mcompetitions/M4-methods. Find the paper of top methods in Winning methods and approaches.

The M4 Forecasting Competition

Conclusions

“The M4 competition: Conclusions” by Spyros Makridakis & Fotios Petropoulos