Dr. O. Aly
Computer Science

Abstract

The purpose of this project is to analyze the online radio dataset called (lastfm.csv). The project is divided into two main Parts.  Part-I evaluates and examines the dataset for understanding the Dataset using the RStudio.  Part-I involves three major tasks to review and understand the Dataset variables.  Part-II discusses the Pre-Data Analysis, by converting the Dataset to Data Frame, involving three major tasks to analyze the Data Frame. The Association Rule data mining technique is used in this project.  The support for each of the 1004 artists is calculated, and the support is displayed for all artists with support larger than 8% indicating that artists shown on the graph (Figure 4) are played by more than 8% of the users.  The construction of the association rules is also implemented using the function of “apriori” in R package arules.  The search was implemented for artists or groups of artists who have support larger than 1% and who give confidence to another artist that is larger than 50%.  These requirements rule out rare artists.  The calculation and the list of antecedents (LHS) are also implemented which involve more than one artist.  The list is further narrowed down by requiring that the lift is larger than 5 and the resulting list is ordered according to the decreasing confidence as illustrated in Figure 6. 

Keywords: Online Radio, Association Rule Data Mining Analysis

Introduction

This project examines and analyzes the Dataset of (lastfm.csv).  The dataset is downloaded from CTU course materials.  The lastfm.csv dataset reflect online radio which keeps track of every thing the user plays.  It has 289,955 observations with four variables.  The focus of this analysis is Association Rule.  The information in the dataset is used for recommending music the user is likely to enjoy and supports focused on marketing which sends the user advertisements for music the user is likely to buy.  From the available information such as demographic information (such as age, sex and location) the support for the frequencies of listeninig to various individual artists can be determined as well as the joint support for pairs or larger groupings of artists.  Thus, to calculate such support, the count of the incidences (0/1) (frequency) is implemented across all memebers of the network and divide those frequencies by the number of the members.  From the support, the confidence and the lift is calculated.

This project addresses two major Parts.  Part-I covers the following key Tasks to understand and examine the Dataset of “lastfm.csv.” 

• Task-1:  Review the Variables of the Dataset.
• Task-2:  Load and Understand the Dataset Using names(), head(), dim() Functions.
• Task-3:  Examine the Dataset, Summary of the Descriptive Statistics, and Visualization of the Variables.

Part-II covers the following three primary key Tasks to the plot, discuss and analyze the result.

• Task-1: Required Computations for Association Rules and Frequent Items.
• Task-2: Association Rules.
• Task-3: Discussion and Analysis.

Various resources were utilized to develop the required code using R. These resources include (Ahlemeyer-Stubbe & Coleman, 2014; Fischetti, Mayor, & Forte, 2017; Ledolter, 2013; r-project.org, 2018).

Part-I:  Understand and Examine the Dataset “lastfm.csv”

Task-1:  Review the Variables of the Dataset

The purpose of this task is to understand the variables of the dataset.  The Dataset is “lastfm.csv” dataset.  The Dataset describes the artists and the users who listens to the music. From the available information such as demographic information (such as age, sex and location) the support for the frequencies of listeninig to various individual artists can be determined as well as the joint support for pairs or larger groupings of artists.  There are 4 variables.  Table 1 summarizes the selected variables for this project.  

Table 1:  LastFm Dataset Variables

Task-2:  Load and Understand the Dataset Using names(), head(), dim() Functions.

            The purpose of this task is to load and understand the Dataset using names(), head(), dim() function.  The task also displays the first three observations.

• ## reading the data
• lf <-read.csv(“C:/CS871/Data/lastfm.csv”)
• lf
• dim(lf)
• length(lf$user)
• names(lf)
• head(lf)
• lf <- data.frame(lf)
• head(lf)
• str(lf)
• lf[1:20,]
• lfsmallset <- lf[1:1000,]
• lfsmallset
• plot(lfsmallset, col=”blue”, main=”Small Set of Online Radio”)

Figure 1.  First Sixteen Observations for User (1) – Woman from Germany.

Figure 2. The plot of Small Set of Last FM Variables.

 Task-3:  Examine the Dataset, Summary of the Descriptive Statistics and Visualization of the Variables.

            The purpose of this task is to examine the dataset.  This task also factor the user and levels users and artist variables.  It also displays the summary of the variables and the visualization of each variable.

• ### Factor user and levels user and artist
• lf$user <- factor(lf$user)
• levels(lf$user)    ## 15,000 users
• levels(lf$artist)  ## 1,004 artists        
• ## Summary of the Variables
• summary(lf)
• summary(lf$user)
• summary(lf$artist)
• ## Plot for Visualization of the variables.
• plot(lf$user, col=”blue”)
• plot(lf$artist, col=”blue”)
• plot(lf$sex, col=”orange”)
• plot(lf$country, col=”orange”)

Figure 3.  Plots of LastFM Variables.

Part-II:  Association Rules Data Mining, Discussion and Analysis

 Task-1:  Required Computations for Association Rules and Frequent Items

The purpose of this task is to first implement computations which are required for the association rules.  The required package arules is first installed.  This task visualizes the frequency of items in Figure 4.

• ## Install arules library for association rules
• install.packages(“arules”)
• library(arules)
• ### computational environment for mining association rules and frequent item sets
• playlist <- split(x=lf[,”artist”], f=lf$user)
• playlist[1:2]
• ## Remove Artist Duplicates.
• playlist <- lapply(playlist,unique)
• playlist <- as(playlist,”transactions”)
• ## view this as a list of “transaction”
• ## transactions is a data class defined in arules
• itemFrequency(playlist)
• ## lists the support of the 1,004 bands
• ## number of times band is listed to on the playlist of 15,000 users
• ## computes relative frequency of artist mentioned by the 15,000 users
• ## plots the item frequencies.
• itemFrequencyPlot(playlist,support=0.08, cex.names=1.5, col=”blue”, main=”Item Frequency”)

Figure 4.  Plot of Item Frequency.

Task-2:  Association Rules Data Mining

The purpose of this task is to implement the data mining for the music list (lastfm.csv) using Association Rules technique.  First, the code builds the Association Rules, followed by the implementation of the associations with support > 0.01 and confidence > 0.50. Rule out rare bands and ordering the result by confidence for better understanding of the association rules result.

• ## Build the Association Rules
• ## Only associations with support > 0.01 and confidence > 0.50
• ## Rule out rare bands
• music.association.rules <- apriori(playlist, parameter=list(support=0.01, confidence=0.50))
• inspect(music.association.rules)
• ## Filter by lift > 5
• ## Show only those with lift > 5, among those association with support > 0.01 and confidence > 0.50.
• inspect(subset(music.association.rules, subset=lift > 5))
• ## Order by confidence for better understanding of the association rules result.
• inspect(sort(subset(music.association.rules, subset=lift>5), by=”confidence”))

Figure 5.  Example of Listening to both “Muse” and “Beatles” with a Confidence of 0.507 for Radiohead.

Figure 6.  Narrow the List by increasing the Lift to > 5 and Decreasing Confidence.

 Task-3: Discussion and Analysis

The association rules are used to explore the relationship between items and sets of items (Fischetti et al., 2017; Giudici, 2005).  Each transaction is composed of one or more items.  The interest is in transactions of at least two items because there cannot be relationships between several items in the purchase of a single item (Fischetti et al., 2017). The association rule is the explicit mention in a relationship in the data, in the form of X >= Y, where X (the antecedent) can be composed of one or several items and is called itemset, and Y (the consequent) is always one single item.  In this project, the interest is in the antecedents of music since the interest is in promoting the purchase of music.  The frequent “itemsets” are the items or collections of items which frequently occur in transactions.  The “itemsets” are considered frequent if they occur more frequently than a specified threshold (Fischetti et al., 2017).  The threshold is called minimal support (Fischetti et al., 2017).  The omission of “itemsets” with support less than the minimum support is called support pruning (Fischetti et al., 2017). The support for an itemset is the proportion among all cases where the itemset of interest is present, which allows estimation of how interesting an itemset or a rule is when support is low, the interest is limited (Fischetti et al., 2017). The confidence is the proportion of cases of X where X >= Y, which can be computed as the number of cases featuring X and Y divided by the number of cases featuring X (Fischetti et al., 2017).  Lift is a measure of the improvement of the rule support over what can be expected by chance, which is computed as support(X>=Y)/support(X)*support(Y) (Fischetti et al., 2017).  If the lift value is not higher than 1, the rule does not explain the relationship between the items better than could be expected by chance.  The goal of “apriori” is to compute the frequent “itemsets” and the association rules efficiently and to compute support and confidence.  

In this project, the large dataset of lastfm (289,955 observations and four variables) is used.  The descriptive analysis shows that the number of males (N=211823) exceeds the number of female users (N=78132) as illustrated in Figure 3.  The top artist has a value of 2704, followed by “Beatles” of 2668 and “Coldplay” of 2378.  The top country has the value of 59558 followed by the United Kingdom of 27638 and German of 24251 as illustrated in Task-3 of Part-I.

As illustrated in Figure 1, the first sixteen observations are for the user (1) for a woman from Germany, resulting in the first sixteen rows of the data matrix.  The R package arules was used for mining the association rules and for identifying frequent “itemsets.”  The data is transformed into an incidence matrix where each listener represents a row, with 0 and 1s across the columns indicating whether or not the user has played a particular artist.  The incidence matrix is stored in the R object “playlist.”  The support for each of the 1004 artists is calculated, and the support is displayed for all artists with support larger than 8% indicating that artists shown on the graph (Figure 4) are played by more than 8% of the users. 

The construction of the association rules is also implemented using the function of “apriori” in R package arules.  The search was implemented for artists or groups of artists who have support larger than 1% and who give confidence to another artist that is larger than 50%.  These requirements rule out rare artists.  The calculation and the list of antecedents (LHS) are also implemented which involve more than one artist.  For instance, listening both to “Muse” and “Beatles” has support larger than 1%, and the confidence for “Radiohead,” given that someone listens to both “Muse” and “Beatles” is 0.507 with a lift of 2.82 as illustrated in Figure 5.  This result exceeded the two requirements as antecedents involving three artists do not come up in the list because they do not meet both requirements.  The list is further narrowed down by requiring that the lift is larger than 5 and the resulting list is ordered according to the decreasing confidence as illustrated in Figure 6.  The result shows that listening to both “Led Zeppelin” and “the Doors” has a support of 1%, the confidence of 0.597 (60%) and lift of 5.69 and is quite predictive of listening to “Pink Floyd” as shown in Figure 6. Another example of the association rule result is listening to “Judas Priest” lifts the chance of listening to the “Iron Maiden” by a factor of 8.56 as illustrated in Figure 6.  Thus, if the user listens to “Judas Priest,” the recommendation for that user to also to listen to “Iron Maiden.”  The same association rules results apply to all of the six items listed in Figure 6.

References

Ahlemeyer-Stubbe, A., & Coleman, S. (2014). A practical guide to data mining for business and industry: John Wiley & Sons.

Fischetti, T., Mayor, E., & Forte, R. M. (2017). R: Predictive Analysis: Packt Publishing.

Giudici, P. (2005). Applied data mining: statistical methods for business and industry: John Wiley & Sons.

Ledolter, J. (2013). Data mining and business analytics with R: John Wiley & Sons.

r-project.org. (2018). R: A Language and Environment for Statistical Computing. Retrieved from https://cran.r-project.org/doc/manuals/r-release/fullrefman.pdf.

Dr. O. Aly
Computer Science

Introduction

The purpose of this discussion is to discuss and analyze creating ensembles from different methods such as logistic regression, nearest neighbor methods, classification trees, Bayesian, or discriminant analysis. This discussion also addresses the use of the Random Forest to do the analysis.

Ensembles

There are two useful techniques which combine methods for improving predictive power: ensembles and uplift modeling.  Ensembles are the focus of this discussion. Thus, uplift modeling is not discussed in this discussion.  An ensemble combines multiple “supervised” models into a “super-model” (Shmueli, Bruce, Patel, Yahav, & Lichtendahl Jr, 2017)).  An ensemble is based on the dominant notion of combining models (EMC, 2015; Shmueli et al., 2017). Thus, several models can be combined to achieve improved predictive accuracy (Shmueli et al., 2017). 

Ensembles played a significant role in the million-dollar Netflix Prize contest which started in 2006 to improve their movie recommendation system (Shmueli et al., 2017).  The principle of combining methods is known for reducing risk because the variation is smaller than each of the individual components (Shmueli et al., 2017).  The risk is equivalent to a variation in prediction error in predictive modeling.  The more the prediction errors vary, the more volatile the predictive model (Shmueli et al., 2017).  Using an average of two predictions can potentially result in smaller error variance, and therefore, better predictive power (Shmueli et al., 2017).  Thus, results can be combined from multiple prediction methods or classifiers (Shmueli et al., 2017).  The combination can be implemented for predictions, classifications, and propensities as discussed below. 

Ensembles Combining Prediction Using Average Method

When combining prediction, the predictions can be combined with different methods by taking an average.  One alternative to a simple average is taking the median prediction, which would be less affected by extreme predictions (Shmueli et al., 2017). Computing a weighted average is another possibility where the weights are proportional to a quantity of interest such as quality or accuracy (Shmueli et al., 2017).  Ensembles for prediction are useful not only in cross-sectional prediction but also in time series forecasting (Shmueli et al., 2017).

Ensembles Combining Classification Using Voting Method

When combining classification, combining the results from multiple classifiers can be implemented using “voting,” for each record, multiple classifications are available.  A simple rule would be to choose the most popular class among these classifications (Shmueli et al., 2017). For instance, Classification Tree, a Naïve Bayes classifier, and discriminant analysis can be used for classifying a binary outcome (Shmueli et al., 2017).  For each record, three predicted classes are generated (Shmueli et al., 2017). Simple voting would choose the most common class of the three (Shmueli et al., 2017). Similar to the prediction, heavier weights can be assigned to scores from some models, based on considerations such as model accuracy or data quality, which can be implemented by setting a “majority rule” which is different from 50% (Shmueli et al., 2017).   Concerning the nearest neighbor (K-NN), an ensemble learning such as bagging can be performed with K-NN (Dubitzky, 2008). The individual decisions are combined to classify new examples.  Combining of individual results is performed by weighted or unweighted voting (Dubitzky, 2008).

Ensembles Combining Propensities Using Average Method

Similar to prediction, propensities can be combined by taking a simple or weighted average.  Some algorithms such as Naïve Bayes produce biased propensities and should not, therefore, be averaged with propensities from other methods (Shmueli et al., 2017).

Other Forms of Ensembles

Various methods are commonly used for classification, including bagging, boosting, random forest, and support vector machines (SVM).  The bagging, boosting, and random forest is all examples of ensemble methods which use multiple models to obtain better predictive performance than can be obtained from any of the constituent models (EMC, 2015; Ledolter, 2013; Shmueli et al., 2017).

• Bagging: It is short for “bootstrap aggregating” (Ledolter, 2013; Shmueli et al., 2017). It was proposed by Leo Breiman in 1994, which is a model aggregation technique to reduce model variance (Swamynathan, 2017).  It is another form of Ensembles which is based on averaging across multiple random data samples (Shmueli et al., 2017).  There are two steps to implement bagging.  Figure 1illustrates the bagging process flow.
  • Generate multiple random samples by sampling “with replacement from the original data.”  This method is called “bootstrap sampling.”
  • Running an algorithm on each sample and producing scores (Shmueli et al., 2017).

Figure 1.  Bagging Process Flow (Swamynathan, 2017).

Bagging improves the performance stability of a model and helps avoid overfitting by separately modeling different data samples and then combining the result.  Thus, it is especially useful for algorithms such as Trees and Neural Networks.  Figure 2 illustrates an example of the bootstrap sample that has the same size as the original sample size, with ¾ of the original values plus replacement result in repetition of values.

Figure 2:  Bagging Example (Swamynathan, 2017).

Boosting: It is a slightly different method of creating ensembles.  It was introduced by Freud and Schapire in 1995 using the well-known AdaBoost algorithm (adaptive boosting) (Swamynathan, 2017).  The underlying concept of boosting is that rather than an independent individual hypothesis, combining hypotheses in a sequential order increases the accuracy (Swamynathan, 2017).  The boosting algorithms convert the “weak learners” into “strong learners” (Swamynathan, 2017).  Boosting algorithms are well designed to address the bias problems (Swamynathan, 2017). Boosting tends to increase the accuracy (Ledolter, 2013). The “AdaBoosting” process involves three steps. Figure 3 illustrates the “AdaBoosting” process:

1. Assign uniform weight for all data points W₀(x)=1/N, where N is the total number of training data points.
   1. At each iteration fit a classifier y_m(x_n) to the training data and update weights to minimize the weighted error function.

• The final model is given by the following equation:

Figure 3.  “AdaBoosting” Process (Swamynathan, 2017).

            As an example illustration of AdaBoost, there is a sample dataset with 10 data points, with an assumption that all data points will have equal weights giving by, 1/10 as illustrated in Figure 4. 

Figure 4.  An Example Illustration of AdaBoost. Final Model After Three Iteration (Swamynathan, 2017).

• Random Forest: It is another class of ensemble method using decision tree classifiers.  It is a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest. A particular case of random forest uses bagging on decision trees, where samples are randomly chosen with replacement from the original training set (EMC, 2015).
• SVM: Itis another common classification method which combines linear models with instance-based learning techniques. The SVM select a small number of critical boundary instances called support vectors from each class and build a linear decision function which separates them as widely as possible.  SVM can efficiently perform, by default linear classifications and can also be configured to perform non-linear classifications (EMC, 2015).

Advantages and Limitations of Ensembles

Combining scores from multiple models is aimed at generating more precise predictions by lowering the prediction error variance (Shmueli et al., 2017).  The ensemble method is most useful when the combined models generate prediction error which is negatively associated or correlated, but it can also be useful when the correlation is low (Ledolter, 2013; Shmueli et al., 2017).  Ensembles can use simple averaging, weighted averaging, voting, and median (Ledolter, 2013; Shmueli et al., 2017).  Models can be based on the same algorithm or different algorithms, using the same sample or different sample (Ledolter, 2013; Shmueli et al., 2017).  Ensembles have become an important strategy for participants in data mining contests, where the goal is to optimize some predictive measure (Ledolter, 2013; Shmueli et al., 2017).  Ensembles which are based on different data samples help avoid overfitting. However, overfit can also happen with an ensemble in instances such as the choice of best weights when using a weighted average (Shmueli et al., 2017).   

The primary limitation of the ensemble is the resources which it requires such as computationally, and the skills and time investments (Shmueli et al., 2017).  Ensembles which combine results from different algorithms require the development of each model and their evaluation.  The boosting-type ensembles and bagging-type ensembles do not require much effort. However, they do have a computational cost.  Furthermore, ensembles which rely on multiple data sources require the collection and the maintenance of the multiple data sources (Shmueli et al., 2017).  Ensembles are regarded to be “black box” methods, where the relationship between the predictors and the outcome variable usually becomes non-transparent (Shmueli et al., 2017). 

The Use of Random Forests for Analysis

The decision tree is based on a set of True/False decision rules. The prediction is based on the tree rules for each terminal node.  A decision tree for a small set of sample training data encounters the overfitting problem. Random forest model, in contrast, is well suited to handle small sample size problems.  The random forest contains multiple decision trees as the more trees, the better.  Randomness is in selecting the random training subset from the training dataset, using bootstrap aggregating or bagging method to reduce the overfitting by stabilizing the predictions. This method is utilized in many other machine-learning algorithms, not only in the Random Forests (Hodeghatta & Nayak, 2016). There is another type of randomness which occurs when selecting variables randomly from the set of variables, resulting in different trees which are based on different sets of variables.  In a forest, all the trees would still influence the overall prediction by the random forest (Hodeghatta & Nayak, 2016).

The programming logic for Random Forest includes seven steps as follows (Azhad & Rao, 2011).

1. Input the number of training set N.
2. Compute the number of attributes M.
3. For (m) input attributes used to form the decision at a node m<M.
4. Choose training set by sampling with replacement.
5. For each node of the tree, use one of the (m) variables as the decision node.
6. Grow each tree without pruning.
7. Select the classification with maximum votes.

Random Forests have a low bias (Hodeghatta & Nayak, 2016).  The variance is reduced, and thus, overfitting, by adding more trees, which is one of the advantages of the Random Forests, and hence gaining popularity.  The models of Random Forests are relatively robust to the set of input variables and often do not care about pre-processing of data.  Random Forests are described to be more efficient to build than other models such as SVM (Hodeghatta & Nayak, 2016).  Table 1 summarizes the Advantages and Disadvantages of Random Forests in a comparison with other Classification Algorithms such as Naïve Bayes, Decision Tree, Nearest Neighbor. 

Table 1.  Advantages and Disadvantages of Random Forest in comparison with other Classification Algorithms. Adapted from (Hodeghatta & Nayak, 2016).

References

Azhad, S., & Rao, M. S. (2011). Ensuring data storage security in cloud computing.

Dubitzky, W. (2008). Data Mining in Grid Computing Environments: John Wiley & Sons.

EMC. (2015). Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data. (1st ed.): Wiley.

Hodeghatta, U. R., & Nayak, U. (2016). Business Analytics Using R-A Practical Approach: Springer.

Ledolter, J. (2013). Data mining and business analytics with R: John Wiley & Sons.

Shmueli, G., Bruce, P. C., Patel, N. R., Yahav, I., & Lichtendahl Jr, K. C. (2017). Data mining for business analytics: concepts, techniques, and applications in R: John Wiley & Sons.

Swamynathan, M. (2017). Mastering Machine Learning with Python in Six Steps: A Practical Implementation Guide to Predictive Data Analytics Using Python: Apress.

Dr. O. Aly
Computer Science

Introduction

The purpose of this discussion is to discuss and analyze Decision Trees, with a comparison of Classification and Regression Decision Trees.  The discussion also addresses the advantages and disadvantages of the Decision Trees.  The focus of this discussion is on the Classification and Regression Tree (CART) algorithm as one of the statistical criteria. The discussion begins with a brief overview of the Classification, followed by additional related topics.  It will end with a sample Decision Tree for a decision whether or not to take an umbrella.

Classification

Classification is a fundamental data mining technique (EMC, 2015).  Most classification methods are supervised, in which they start with a training set of pre-labeled observations to learn how likely the attributes of these observations may contribute to the classification of future unlabeled observations (EMC, 2015).  For instance, marketing, sales, and customer demographic data can be used to develop a classifier to assign a “purchase” or “no purchase” label to potential future customers (EMC, 2015).  Classification is widely used for prediction purposes (EMC, 2015).  Logistic Regression is one of the popular classification methods (EMC, 2015).  Classification can be used for health care professionals to diagnose diseases such as heart disease (EMC, 2015).   There are two fundamental classification methods:  Decision Trees and Naïve Bayes.  In this discussion, the focus is on the Decision Trees. 

The Tree Models vs. Linear & Logistic Regression Models

The tree models are distinguished from the Linear and Logistic Regression models.  The tree models produce a classification of observations into groups first and then obtain a score for each group, while the Linear and Logistic Regression methods produce a score and then possibly a classification based on a discriminant rule (Giudici, 2005). 

Regression Trees vs. Classification Trees

The tree models are divided into regression trees and classification trees (Giudici, 2005).  The regression trees are used when the response variable is continuous, while the classification trees are used when the response variable is quantitative discrete or qualitative (categorical) (Giudici, 2005).  The tree models can be defined as a recursive process, through which a set of (n) statistical units are divided into groups progressively, based on a division rule aiming to increase a homogeneity or purity measure of the response variable in each of the obtained group (Giudici, 2005). An explanatory variable specifies a division rule at each step of the procedure, to split and establish splitting rules to partition the observations (Giudici, 2005). The final partition of the observation is the main result of a tree model (Giudici, 2005).  It is critical to specify a “stopping criteria” for the division process to achieve such a result (Giudici, 2005). 

Concerning the classification tree, fitted values are given regarding the fitted probabilities of affiliation to a single group (Giudici, 2005). A discriminant rule for the classification trees can be derived at each leaf of the tree (Giudici, 2005).  The classification of all observations belonging to a terminal node in the class corresponding to the most frequent level is a commonly used rule, called “majority rule” (Giudici, 2005).  While other “voting” schemes can also be implemented, in the absence of other consideration, this rule is the most reasonable (Giudici, 2005).  Thus, each of the leaves points out a clear allocation rule of the observation, which is read using the path that connects the initial node to each of them.  Therefore, every path in the tree model represents a classification rule (Giudici, 2005).  

With comparison to other discriminant models, the tree models produce rules which are less explicit analytically, and easier to understand graphically (Giudici, 2005). The tree models can be regarded as nonparametric predictive models as they do not require assumptions about the probability distribution of the response variable (Giudici, 2005).  This flexibility indicates that the tree models are generally applicable, whatever the nature of the dependent variable and the explanatory variables  (Giudici, 2005).  However, the disadvantages of this flexibility of a higher demand of computational resources, and their sequential nature and the complexity of their algorithm can make them dependent on the observed data, and even a small change might alter the structure of the tree  (Giudici, 2005). Thus, it is difficult to take a tree structure designed for one context and generalize it to other contexts  (Giudici, 2005).

The Classification Tree Analysis vs. The Hierarchical Cluster Analysis

The classification tree analysis is distinguished from the hierarchical cluster analysis despite their graphical similarities  (Giudici, 2005).  The classification trees are predictive rather than descriptive. While the hierarchical cluster analysis performs an unsupervised classification of the observations based on all available variables, the classification trees perform a classification of the observations based on all explanatory variables and supervised by the presence of the response variable (target variable) (Giudici, 2005).  The second critical difference between the hierarchical cluster analysis and the classification tree analysis is related to the partition rule.  While in the classification trees the segmentation is typically carried out using only one explanatory variable at a time, in the hierarchical clustering the divisive or agglomerative rule between groups is established based on the considerations on the distance between them, calculated using all the available variables  (Giudici, 2005).

Decision Trees Algorithms

The goal of Decision Trees is to extract from the training data the succession of decisions about the attributes that explain the best class, that is, group membership (Fischetti, Mayor, & Forte, 2017).  Decision Trees have a root, which is the best attribute to split the data upon, about the outcome (Fischetti et al., 2017).  The dataset is partitioned into branches by this attribute (Fischetti et al., 2017).  The branches lead to other nodes which correspond to the next best partition for the considered branch (Fischetti et al., 2017).  The process continues until the terminal nodes are reached, where no more partitioning is required (Fischetti et al., 2017).   Decision Trees allow class predictions (group membership) of previously unseen observations (testing datasets or prediction datasets) using statistical criteria applied on the seen data (training dataset) (Fischetti et al., 2017).  There are six statistical criteria of six algorithms:

• ID3
• C4.5
• Random Forest.
• Conditional Inference Trees.
• Classification and Regress Trees (CART)

The most used algorithm in the statistical community is the CART algorithm, while C4.5 and its latest version C5.0 are widely used by computer scientists (Giudici, 2005).  The first versions of C4.5 and 5.0 were limited to categorical predictors, but the most recent versions are similar to CART (Giudici, 2005).  

Classification and Regression Trees (CART)

CART is often used as a generic acronym for the decision tree, although it is a specific implementation of tree models (EMC, 2015).  CART, similar to C4.5, can handle continuous attributes (EMC, 2015).  While C4.5 uses entropy-based criteria to rank tests, CART uses the Gini diversity index defined in equation (1) (EMC, 2015; Fischetti et al., 2017).

Moreover, while C4.5 uses stopping rules, CART construct a sequence of subtrees, uses cross-validation to estimate the misclassification cost of each subtree, and chooses the one with the lowest cost, (EMC, 2015; Hand, Mannila, & Smyth, 2001). CART represents a powerful nonparametric technique which generalizes parametric regression models (Ledolter, 2013).  It allows nonlinearity and variable interactions without having to specify the structure in advance (Ledolter, 2013).  It operates by choosing the best variable for splitting the data into two groups at the root node (Hand et al., 2001).  It builds the tree using a single variable at a time, and can readily deal with large numbers of variables (Hand et al., 2001).  It uses different statistical criteria to decide on tree splits (Fischetti et al., 2017).  There are some differences between CART used for classification and the family of algorithms.  In CART, the attribute to be partition is selected with the Gini index as a decision criterion (Fischetti et al., 2017).  This method is described as more efficient compared to the information gain and information ratio (Fischetti et al., 2017).  CART implements the necessary partitioning on the modalities of the attribute and merges modalities for the partition, such as modality A versus modalities B and C (Fischetti et al., 2017).  The CART can predict a numeric outcome (Fischetti et al., 2017).  In the case of regression trees, CART performs regression and builds the tree in a way which minimizes the squared residuals (Fischetti et al., 2017). 

CART Algorithms of Division Criteria and Pruning

There are two critical aspects of the CART algorithm:  Division Criteria, and Pruning, which can be employed to reduce the complexity of a tree (Giudici, 2005).  Concerning the division criteria algorithm, the primary essential element of a tree model is to choose the division rule for the units belonging to a group, corresponding to a node of the tree (Giudici, 2005).  The decision rule selection means a predictor selection from those available, and the selection of the best partition of its levels (Giudici, 2005).  The selection is generally made using a goodness measure of the corresponding division rule, which allows the determination of the rule to maximize the goodness measure at each stage of the procedure (Giudici, 2005). 

The impurity concept refers to a measure of variability of the response values of the observations (Giudici, 2005).  In a regression tree, a node will be pure if it has null variance as all observations are equal, and it will be impure if the variance of the observation is high (Giudici, 2005).  For the regression trees, the impurity corresponds to the variance, while for the classification trees alternative measures for the impurity are considered such as Misclassification impurity, Gini impurity, Entropy impurity, and Tree assessments (Giudici, 2005).  

When there is no “stopping criterion,” a tree model can grow until each node contains identical observation regarding the values or levels of the dependent variable  (Giudici, 2005). This approach does not contain a parsimonious segmentation  (Giudici, 2005). Thus, it is critical to stop the growth of the tree at a reasonable dimension  (Giudici, 2005). The tree configuration becomes ideal when it is parsimonious and accurate  (Giudici, 2005). The parsimonious attribute indicates that the tree has a small number of leaves, and therefore, the predictive rule can be easily interpreted  (Giudici, 2005). The accurate attribute indicates a large number of leaves which are pure to a maximum extent  (Giudici, 2005). There are two opposing techniques for the final choice which tree algorithms can employ. The first technique uses stopping rules based on the thresholds on the number of the leaves, or on the maximum number of steps in the process, whereas the other algorithm technique introduces probabilistic assumptions on the variables, allowing the use of suitable statistical tests  (Giudici, 2005). The growth is stopped when the decrease in impurity is too small, in the absence of the probabilistic assumptions  (Giudici, 2005). The result of a tree model can be influenced by the choice of the stopping rule (Giudici, 2005). 

The CART method utilizes a strategy different from the stepwise stopping criteria. The method is based on the pruning concept.  The tree, first, is built to its greatest size, and it then gets “trimmed” or “pruned” according to a cost-complexity criterion  (Giudici, 2005). The concept of pruning is to find a subtree optimally, to minimize a loss function, which is used by CART algorithm and depends on the total impurity of the tree and the tree complexity  (Giudici, 2005). The misclassification impurity is usually chosen to be used for the pruning, although the other impurity methods can also be used.  The minimization of the loss function results in a compromise between choosing a complex model with low impurity but high complexity cost and choosing a simple model with a high impurity with low complexity cost  (Giudici, 2005).   The loss function is assessed by measuring the complexity of the model fitted on the training dataset, whose misclassification errors are measured in the validation data set  (Giudici, 2005).  This method partitions the training data into a subset for building the tree and then estimates the misclassification rate on the remaining validation subset (Hand et al., 2001).

The CART has been widely used for several years by marketing applications and others (Hodeghatta & Nayak, 2016).   The CART is described as a flexible model as the violations of constant variance which is very critical in regression, is permissible in the CART (Ledolter, 2013).  However, the biggest challenge in the CART is the avoidance of the “overfitting” (Ledolter, 2013).

Advantages and Disadvantages of the Trees

Decision trees for regression and classification have advantages and disadvantages.  Trees are regarded to be easier than linear regression and can be displayed graphically and interpreted easily (Cristina, 2010; Tibshirani, James, Witten, & Hastie, 2013).  Decision trees are self-explanatory and easy to understand even for non-technical users (Cristina, 2010; Tibshirani et al., 2013). They can handle qualitative predictors without the need to create dummy variables (Tibshirani et al., 2013).  Decision trees are efficient. Complex alternatives can be expressed quickly and precisely. A decision tree can easily be modified as new information becomes available.  Standard decision tree notation is easy to adopt (Cristina, 2010).  They can be used in conjunction with other management tools.  Decision trees can handle both nominal and numerical attributes (Cristina, 2010).  They are capable of handling datasets which may have errors or missing values.  Decision trees are considered to be a non-parametric method, which means that they have no assumption about the spatial distribution and the classifier structure.  Their representations are rich enough to represent any discrete-value classifier.

However, trees have limitations as well.  They do not have the same level of predictive accuracy as some of the other regression and classification models (Tibshirani et al., 2013). Most of the algorithms, like ID3 and C4.5, require that the target attribute will have only discrete values. Decision trees are over-sensitive to the training set, to irrelevant attributes and noise. Decision trees tend to perform less if many complex interactions are present, and well if a few highly relevant attributes exist as they use the “divide and conquer” method (Cristina, 2010).  Table 1 summarizes the advantages and disadvantages of the trees.

Table 1.  Summary of the Advantages and Disadvantages of Trees.
Note:  Constructed by the researcher based on the literature.

Take an Umbrella Decision Tree Example:

• If input field value < n
  • Then target = Y%
• If input field value > n
  • Then target = X%

Figure 1.  Decision Tree for Taking an Umbrella

• The decision depends on the weather, on the predicted rain probability, and whether it is sunny or cloudy.
• The forecast predicts rain with a probability between 70% and 30%.
  • If it is >70% rain probability, take an umbrella, else use >30% and <30% probability for further predictions.
  • If it is >30% rain probability and cloudy, take an umbrella, else no umbrella.
  • If it is <30% rain probability, no umbrella.

References

Cristina, P. (2010). Decision Trees. Retrieved from http://www.cs.ubbcluj.ro/~gabis/DocDiplome/DT/DecisionTrees.pdf.

EMC. (2015). Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data. (1st ed.): Wiley.

Fischetti, T., Mayor, E., & Forte, R. M. (2017). R: Predictive Analysis: Packt Publishing.

Giudici, P. (2005). Applied data mining: statistical methods for business and industry: John Wiley & Sons.

Hand, D. J., Mannila, H., & Smyth, P. (2001). Principles of data mining.

Hodeghatta, U. R., & Nayak, U. (2016). Business Analytics Using R-A Practical Approach: Springer.

Ledolter, J. (2013). Data mining and business analytics with R: John Wiley & Sons.

Tibshirani, R., James, G., Witten, D., & Hastie, T. (2013). An introduction to statistical learning-with applications in R: New York, NY: Springer.

Dr. O. Aly
Computer Science

Abstract

The purpose of this project is to analyze the flight delays Dataset. The project is divided into two main Parts.  Part-I evaluates and examines the Dataset for understanding the Dataset using the RStudio.  Part-I involves five major tasks to review and understand the Dataset variables.  Part-II discusses the Pre-Data Analysis, by converting the Dataset to Data Frame, involving three major tasks to analyze the Data Frame using logistic regression first, followed by the naïve Bayesian method.   The naïve Bayesian method used probabilities from the training set consisting of 60% randomly selected flights, and the remaining 40% of the 2201 flights serve as the holdout period.  The misclassification proportion of the naïve Bayesian method shows 19.52%, which is a little higher than the logistic regression. The prediction has 30 delayed flight out of the 167 correctly but fails to identify 137/(137+30), or 73% of the delayed flights.  Moreover, the 35/(35+679), or 4.9% of on-time flights are predicted as delayed as illustrated in Task-2 of Part-II and Figure-19.

Keywords: Flight-Delays Dataset; Naïve Bays Prediction Analysis Using R.

Introduction

This project examines and analyzes the Dataset of (flight.delays.csv).  The Dataset is downloaded from CTU course materials.  There have been a couple of attempts to download the Dataset from the following link https://www.transtats.bts.gov/.  However, the attempts failed to continue with the Dataset analysis due to the size of the downloaded Datasets from that link and the limited resources of the student’s machine.  Thus, this project utilized the version of flight.delays.csv which is provided by the course in the course material.  The Dataset of (flight.delays.cvs) has 2201 observations on 14 variables.  The focus of this analysis is Naïve Bayes.  However, for a better understanding of the prediction and a comparison using two different models, the researcher has also implemented the Logistic Regression first, followed by the Naïve Bayesian Approach on the same Dataset of flight.delays.csv.  This project addresses two major Parts.  Part-I covers the following key Tasks to understand and examine the Dataset of “flight.delays.csv.” 

• Task-1:  Review the Variables of the Dataset.
• Task-2:  Load and Understand the Dataset Using names(), head(), dim() Functions.
• Task-3:  Examine the Dataset, Install the Required Packages, and Summary of the Descriptive Statistics.
• Task-4:  Create Data Frame and Histogram of the Delay (Response)
• Task-5:  Visualization of the Desired Variables Using Plot() Function.

Part-II covers the following three primary key Tasks to the plot, discuss and analyze the result.

• Task-1:  Logistic Regression Model for Flight Delays Prediction
• Task-2:  Naïve Bayesian Model for Flight Delays Prediction.
• Task-3:  Discussion and Analysis.

Various resources were utilized to develop the required code using R. These resources include (Ahlemeyer-Stubbe & Coleman, 2014; Fischetti, Mayor, & Forte, 2017; Ledolter, 2013; r-project.org, 2018).

Part-I:  Understand and Examine the Dataset “flight.delays.csv”

Task-1:  Review the Variables of the Dataset

The purpose of this task is to understand the variables of the Dataset.  The Dataset is “flight. Delays” Dataset.  The Dataset describes the clients who can default on a loan.  There are 14 variables.  Table 1 summarizes the selected variables for this project.  

Table 1:  Flight Delays Variables

Task-2:  Load and Understand the Dataset Using names(), head(), dim() Functions.

            The purpose of this task is to load and understand the Dataset using names(), head(), dim() function.  The task also displays the first three observations.

• ## reading the data
• fd <-read.csv(“C:/CS871/Data/flight.delays.csv”)
• names(fd[1:5,])
• head(fd)
• dim(fd)
• fd[1:3,]

Task-3:  Examine the Dataset, Install the Required Packages, and Summary of the Descriptive Statistics.

            The purpose of this task is to examine the dawta set, install the requried package (car).  This task also displays the descriptive statistics for analysis.

• ### set seed
• set.seed(1)
• ##Required Library(car) to recode a variable
• install.packages(“car”)
• library(car)
• summary(fd)
• plot(fd, col=”blue”)

Figure 1. The plot of the Identified Variables for the Flight Delays Dataset.

Task-5:  Visualization of the Desired Variables Using Plot() Function.

            The purpose of this task is to visualize the selected variables using the Plot() Function for a good understanding of these variables and the current trend for each variable.

• plot(fd$schedf, col=”blue”, main=”Histogram of the Scheduled Time”)
• plot(fd$carrier, col=”blue”, main=”Histogram of the Carrier”)
• plot(fd$dest, col=”blue”, main=”Histogram of the Destination”)
• plot(fd$origin, col=”blue”, main=”Histogram of the  Origin”)
• plot(fd$weather, col=”blue”, main=”Histogram of the Weather”)
• plot(fd$dayweek, col=”blue”, main=”Histogram of the Day of Week”)

Figure 2.  Histogram of the Schedule Time and Carrier.

Figure 3.  Histogram of the Destination and Origin.

Figure 4.  Histogram of the Weather and Day of Week.

Part-II:  Plot, Discuss and Analyze

 Task-1: Logistic Regression Model for Flight Delays

The purpose of this task is to first use the logistic regression model for predicting the on-time and delayed flights more than 15 minutes. The Dataset consists of 2201 flights for the year of 2004 from Washington DC into the NYC.  The characteristic of the response is whether or not a flight has been delayed by more than 15 minutes and coded as 0=no delay, and 1=delay by more than 15 minutes.  The explanatory variables include:

• Three arrival airports (Kennedy, Newark, and LaGuardia).
• Three different departure airports (Reagan, Dulles, and Baltimore.
• Eight carriers a categorical variable for 16 different hours of departure (6:00 AM to 10:00 PM).
• Weather conditions (0=good, 1=bad).
• Day of week (1 for Sunday and Monday; and 0 for all other days).

The code of R is shown below for the logistic regression model.

• ## Create a Data Frame and Understand the Dataset.
• fd <-data.frame(fd)
• names(fd)
• head(fd)
• fd[1:5,]
• dim(fd)
• summary(fd)
• plot(fd, co=”blue”)
• ## library car is needed to recode variables
• library(car)
• ##Define hours of Departure
• fd$sched=factor(floor(fd$schedtime/100))
• table(fd$sched)
• table(fd$carrier)
• table(fd$dest)
• table(fd$origin)
• table(fd$weather)
• table(fd$dayweek)
• table(fd$daymonth)
• table(fd$delay)
• fd$delay=recode(fd$delay,”‘delayed’=1;else=0″)
• fd$delay=as.numeric(levels(fd$delay)[fd$delay])
• table(fd$delay)
• ## Summary of the Major Variables
• summary(fd$sched)
• summary(fd$carrier)
• summary(fd$dest)
• summary(fd$origin)
• summary(fd$weather)
• summary(fd$dayweek)
• summary(fd$daymonth)
• summary(fd$delay)
• ## Plots and Histograms of the Major Variables
• plot(fd$sched, col=”blue”, main=”Schedule Departure Time”)
• plot(fd$carrier, col=”darkblue”, main=”Flight Carriers”)
• plot(fd$dest, col=”darkred”, main=”Destination of Flights”)
• plot(fd$origin, col=”green”, main=”Origin of Flights”)
• plot(fd$weather, col=”darkgreen”, main=”Weather During Flight Days”)
• hist(fd$dayweek, col=”darkblue”, main=”Flights Day of the Week”, xlab=”Day of Week”)
• hist(fd$daymonth, col=”yellow”, main=”Flights Day of the Month”)
• plot(fd$delay, col=”red”, main=”Plot of the Delay”)
• hist(fd$delay, col=”red”, main=”Histogram of the Delay”)
• ## Delay: 1=Monday and 7=Sunday coded as 1, else 0.
• fd$dayweek=recode(fd$dayweek,”c(1,7)=1;else=0″)
• table(fd$dayweek)
• summary(fd$dayweek)
• hist(fd$dayweek, col=”darkblue”, main=”Flights Day of the Week”, xlab=”Day of Week”)
• ## Omit unused variables
• fd=fd[,c(-1,-3,-5,-6,-7,-11,-12)]
• fd[1:5,]
• ## Create Sample Dataset
• delay.length=length(fd$delay)
• delay.length
• delay.length1=floor(delay.length*(0.6))
• delay.length1
• delay.length2=delay.length-delay.length1
• delay.length2
• train=sample(1:delay.length, delay.length1)
• train
• plot(train, col=”red”)
• ## Estimation of Logistic Regression Model
• ##Explanatory Variables: carrier, destination, origin, weather, day of week,
• ##(weekday/weekend), scheduled hour of departure.
• ## Create design matrix; indicators for categorical variables (factors)
• Xfd <- model.matrix(delay~., data=fd) [,-1]
• Xfd[1:5,]
• xtrain <- Xfd[train,]
• xtrain[1:2,]
• xtest <- Xfd[-train,]
• xtest[1:2,]
• ytrain <- fd$delay[train]
• ytrain[1:5]
• ytest <- fd$delay[-train]
• ytest[1:5]
• model1 = glm(delay~., family=binomial, data=data.frame(delay=ytrain,xtrain))
• summary(model1)
• ## Prediction: predicted default probabilities for cases in test set
• probability.test <- predict(model1, newdata=data.frame(xtest), type=”response”)
• data.frame(ytest,probability.test)[1:10,]
• ## The first column in list represents the case number of the test element
• plot(ytest~probability.test, col=”blue”)
• ## Coding as 1 if probability 0.5 larger
• ### using floor function
• probability.fifty = floor(probability.test+0.5)
• table.ytest = table(ytest,probability.fifty)
• table.ytest
• error = ((table.ytest[1,2]+table.ytest[2,1])/delay.length2)
• error

Figure 5.  The probability of the Delay Using Logistic Regression.

Task-2:  Naïve Bayesian Model for Predicting Delays and Ontime Flights

The purpose of this task is to use the Naïve Bayesian model for predicting a categorical response from most categorical predictor variables.  The Dataset consists of 2201 flights in 2004 from Washington, DC into NYC.  The characteristic of the response is whether or not a flight has been delayed by more than 15 minutes (0=no delay, 1=delay).  The explanatory variables include the following:

• Three arrival airports (Kennedy, Newark, and LaGuardia).
• Three different departure airports (Reagan, Dulles, and Baltimore.
• Eight carriers.
• A categorical variable for 16 different hours of departure (6:00 AM to 10:00 PM).
• Weather conditions (0=good, 1=bad).
• Day of week (7 days with Monday=1, …, Sunday=7).

The code of R is shown below for the logistic regression model, followed by the result of each code.

• fd=data.frame(fd)
• fd$schedf=factor(floor(fd$schedtime/100))
• fd$delay=recode(fd$delay,”‘delayed’=1;else=0″)
• response=as.numeric(levels(fd$delay)[fd$delay])
• hist(response, col=”orange”)
• fd.mean.response=mean(response)
• fd.mean.response
• ## Create Train Dataset 60/40
• n=length(fd$dayweek)
• n
• n1=floor(n*(0.6))
• n1
• n2=n-n1
• n2
• train=sample(1:n,n1)
• train
• plot(train, col=”blue”, main=”Train Data Plot”)
• ## Determine Marginal Probabilities
• td=cbind(fd$schedf[train],fd$carrier[train],fd$dest[train],fd$origin[train],fd$weather[train],fd$dayweek[train],response[train])
• td
• tdtrain0=td[td[,7]<0.5,]
• tdtrain1=td[td[,7]>0.5,]
• tdtrain0[1:3]
• tdtrain1[1:3]
• plot(td, col=”blue”, main=”Train Data”)
• plot(tdtrain0, col=”blue”, main=”Marginal Probability <0.5″ )
• plot(tdtrain1, col=”blue”, main=”Marginal Probability > 0.5″)
• ## Prior Probabilities for Delay for P( y = 0 ) and P(y = 1)
• tdel=table(response[train])
• tdel=tdel/sum(tdel)
• tdel
• ## P( y = 0)       P( y = 1)         1
• ##0.8022727    0.1977273
• ## Probabilities for Scheduled Time
• ### Probabilities for (y=0) for Scheduled Time
• ts0=table(tttrain0[,1])
• ts0=ts0/sum(ts0)
• ts0
• ### Probabilities for (y = 1) for Scheduled Time
• ts1=table(tttrain1[,1])
• ts1=ts1/sum(ts1)
• ts1
• ## Probabilities for Carrier
• ## Probabilities for (y = 0) for Carrier
• tc0=table(tttrain0[,2])
• tc0=tc0/sum(tc0)
• tc0
• tc1=table(tttrain1[,2])
• tc1=tc1/sum(tc1)
• tc1
• ## Probabilities for Destination
• ##Probabilities for (y=0) for Destination
• td0=table(tttrain0[,3])
• td0=td0/sum(td0)
• td0
• ##Probabilities for (y=1) for Destination
• td1=table(tttrain1[,3])
• td1=td1/sum(td1)
• td1
• ## Probabilities for Origin
• ##Probabilities for (y=0) for Origin
• to0=table(tttrain0[,4])
• to0=to0/sum(to0)
• to0
• ##Probabilities for (y=1) for Origin
• to1=table(tttrain1[,4])
• to1=to1/sum(to1)
• to1
• ## Probabilities for Weather
• ##Probabilities for (y=0) for Origin
• tw0=table(tttrain0[,5])
• tw0=tw0/sum(tw0)
• tw0
• ## bandaid as no observation in a cell
• tw0=tw1
• tw0[1]=1
• tw0[2]=0
• ##Probabilities for (y=1) for Weather
• tw1=table(tttrain1[,5])
• tw1=tw1/sum(tw1)
• tw1
• ## Probabilities for Day of Week
• #### Probabilities for (y-0) for Day of Week
• tdw0=table(tttrain0[,6])
• tdw0=tdw0/sum(tdw0)
• tdw0
• #### Probabilities for (y-1) for Day of Week
• tdw1=table(tttrain1[,6])
• tdw1=tdw1/sum(tdw1)
• tdw1
• ### Create Test Data
• testdata=cbind(fd$schedf[-train],fd$carrier[-train],fd$dest[-train],fd$origin[-train],fd$weather[-train],fd$dayweek[-train],response[-train])
• testdata[1:3]
• ## With these estimates, the following probabilities can be determined.
• ##P(y = 1|Carrier = 7,DOW = 7,DepTime = 9 AM−10 AM,Dest = LGA,Origin = DCA,Weather = 0)
• =[(0.015)(0.172)(0.027)(0.402)(0.490)(0.920)](0.198)
• [(0.015)(0.172)(0.027)(0.402)(0.490)(0.920)](0.198)
• +[(0.015)(0.099)(0.059)(0.545)(0.653)(1)](0.802)
• = 0.09.
• ## Creating Predictions, stored in prediction
• p0=ts0[tt[,1]]*tc0[tt[,2]]*td0[tt[,3]]*to0[tt[,4]]*tw0[tt[,5]+1]*tdw0[tt[,6]]
• p1=ts1[tt[,1]]*tc1[tt[,2]]*td1[tt[,3]]*to1[tt[,4]]*tw1[tt[,5]+1]*tdw1[tt[,6]]
• prediction=(p1*tdel[2])/(p1*tdel[2]+p0*tdel[1])
• hist(prediction, col=”blue”, main=”Histogram of Predictions”)
• plot(response[-train], prediction,col=”blue”)
• ###Coding as 1 if probability >=0.5 
• ## Calculate the Probability for at least 0.5 or more
• prob1=floor(prediction+0.5)
• tr=table(response[-train],prob1)
• tr
• error=(tr[1,2]+tr[2,1])/n2
• error
• ## Calculate the Probability for at least 0.3 or more
• prob2=floor(prediction+0.3)
• tr2=table(response[-train],prob2)
• tr2
• error=(tr[1,2]+tr[2,1])/n2
• error
• ## calculating the lift, cumulative, sorted by predicted values and average success.
• ## cumulative 1’s sorted by predicted values
• ## cumulative 1’s using the average success prob from training set
• axis=dim(n2)
• ax=dim(n2)
• ay=dim(n2)
• axis[1]=1
• ax[1]=xbar
• ay[1]=bb1[1,2]
• for (i in 2:n2) {
• axis[i]=i
• ax[i]=xbar*i
• ay[i]=ay[i-1]+bb1[i,2]
• }
• aaa=cbind(bb1[,1],bb1[,2],ay,ax)
• aaa[1:100,]
• plot(axis,ay,xlab=”Number of Cases”,ylab=”Number of Successes”,main=”Lift: Cum Successes Sorted Predicted Values Using Average Success Probabilitis”, col=”red”)
• points(axis,ax,type=”l”)

Figure 6. Pre and Post Factor and Level of History Categorical Variable.

Figure 7: Train Dataset Plot.

Figure 8.  Train Data, Marginal Probability of <0.5 and >0.5.

Figure 9.  Prior Probability for Delay (y=0) and (y-1).

Figure 10.  Prior Probability for Scheduled Time: Left (y=0) and Right (y-1).

Figure 11.  Prior Probability for Carrier: Left (y=0) and Right (y-1).

Figure 12.  Prior Probability for Destination: Left (y=0) and Right (y-1).

Figure 13.  Prior Probability for Origin: Left (y=0) and Right (y-1).

Figure 14.  Prior Probability for Weather: Left (y=0) and Right (y-1).

Figure 15.  Prior Probability for Day of Week: Left (y=0) and Right (y-1).

Figure 16.  Test Data Plot.

Figure 17.  Histogram of the Prediction Using Bayesian Method.

Figure 18.  Plot of Prediction to the Response Using the Test Data.

Figure 19.  Probability Calculation for at least 0.5 or larger (left), and at least 0.3 or larger (right).

Figure 20.  Lift: Cum Success Sorted by Predicted Values Using Average Success Probabilities.

Task-3: Discussion and Analysis

            The descriptive analysis shows that the average schedule time is 13:72 which is less than the median of 14:55 indicating a negatively skewed distribution, while the average for the departure time is 13:69 which is less than the median of 14:50 confirming the negatively skewed distribution.  The result of the carrier shows that the DH has the highest rank of 551, followed by RU of 408. The result of the destination shows that the LGA has the highest rank of 1150, followed by EWR of 665 and JFK of 386.   The result of the origin shows that DCA has the highest rank of 1370, followed by IAD of 686 and 145 for BWI.   The result shows that the weather is not the primary reason for the delays.  Few instances of weather instances are related to the delays.  The descriptive analysis shows the ontime has the highest frequency of 1773, followed by the delays of 428 frequency.  The average delay or response is 0.195.

The result shows the success probability which is the proportion of delayed planes in the training set if 0.198 as analyzed in Task 2 of Part-II; the failure probability which is the proportion of on-time flights is 0.802 as discussed and analyzed in Task-2 of Part-II.   The naïve rule which does not incorporate any covariate information classified every flight as being on-time as the estimated unconditional probability of a flight being on-time, 0.802, is larger than the cutoff of 0.5. Thus, this rule does not make an error predicting a flight which is on-time, but it makes a 100% error when the flight is delayed.   The naïve rule fails to identify the 167 delayed flights among the 881 flights of the evaluation Dataset as shown in Task-1 of Part-II; its misclassification error rate in the holdout sample is 167/881=0.189.  The logistic regression reduces the overall misclassification error in the holdout (evaluation/test) Dataset to 0.176, which is a modest improvement over the naïve rule of (0.189) as illustrated in Task-1 of Part-II.   The logistic regression identifies, among 167 delayed flights, correctly 14 delayed flights 8.4%, but it misses 153/167 delayed flights (92.6%).  Moreover, the logistic regression model predicts 2 of the 714 on-time flights as being delayed as illustrated in Task-1 of Part-II.

The naïve Bayesian method used probabilities from the training set consisting of 60% randomly selected flights, and the remaining 40% of the 2201 flights serve as the holdout period.  The misclassification proportion of the naïve Bayesian method shows 19.52%, which is a little higher than the logistic regression. The prediction has 30 delayed flight out of the 167 correctly but fails to identify 137/(137+30), or 73% of the delayed flights.  Moreover, the 35/(35+679), or 4.9% of on-time flights are predicted as delayed as illustrated in Task-2 of Part-II and Figure-19.

The lift charts (Figure 20) is constructed with the number of cases on the x-axis and the cumulative true-positive cases on the y-axis.  True positives are those observations which are classified correctly.  It measures the effectiveness of a classification model by comparing the true positives without a model (Hodeghatta & Nayak, 2016).  It also provides an indication of how well the model performs if the samples are selected randomly from a population (Hodeghatta & Nayak, 2016).  With the lift chart, a comparison of different models’ performance for a set of random cases (Hodeghatta & Nayak, 2016).  In Figure 20, the lift varies with the number of cases, and the black line is a reference line, meaning if a prediction of a positive case is made in case there was no model, then, this line provides a benchmark.  The lift curve graph in Figure 20, graphs the expected number of delayed flights, assuming that the probability of delay is estimated by the proportion of delayed flights in the evaluation sample, against the number of cases.  The reference line expresses the performance of the naïve model. With ten flights, for instance, the expected number of delayed flights is 10 p, where p is the proportion of delayed flights in the evaluation sample which is 0.189 in this case.  At the very end, the lift curve and the reference line meet. However, in the beginning, the logistic regression leads to a “lift.” For instance, when picking 10 cases with the largest estimated success probabilities, all the 10 case turn out to be delayed.  If the lift is close to the reference line, then there is not much point in using the estimated model for classification. The overall misclassification rate of the logistic regression is not that different from that of the naïve strategy which considers all flights as being on-time. However, as the lift curve shows, flights with the largest probabilities of being delayed are classified correctly. The logistic regression is quite successful in identifying those flight as being delayed.  The lift curve in Figure 10 shows that the model gives an advantage in detecting the most apparent flights which are going to be delayed or on-time.

References

Ahlemeyer-Stubbe, A., & Coleman, S. (2014). A practical guide to data mining for business and industry: John Wiley & Sons.

Fischetti, T., Mayor, E., & Forte, R. M. (2017). R: Predictive Analysis: Packt Publishing.

Hodeghatta, U. R., & Nayak, U. (2016). Business Analytics Using R-A Practical Approach: Springer.

Ledolter, J. (2013). Data mining and business analytics with R: John Wiley & Sons.

r-project.org. (2018). R: A Language and Environment for Statistical Computing. Retrieved from https://cran.r-project.org/doc/manuals/r-release/fullrefman.pdf.

Dr. O. Aly
Computer Science

Abstract

The purpose of this project is to analyze the German Credit dataset. The project is divided into two main Parts.  Part-I evaluates and examines the DataSet for understanding the Dataset using the RStudio.  Part-I involves six significant tasks.  Part-II discusses the Pre-Data Analysis, by converting the Dataset to Data Frame, involving four significant tasks to analyze the Data Frames.   Although the project analyzes three significant models including Standards Linear Regression, Multiple Linear Regression, the  Logistic Regression is the emphasis of this project.  The result shows that the duration, amount, installment and rent show positive coefficient values indicating that they have a positive impact on the probability of the dependent binary outcome (Default).  As the p-value is much less than 0.05 for duration, amount, installment, history, purpose for used car, goods, repair, and business, and rent, we reject the null hypotheses that there is no significance of the parameter to the model and accept the alternate hypotheses that there is significance of the parameter to the model.  The p-value for the age is (p=0.05) indicating that we accept the null hypothesis that there is no significance of the parameter to the model.  The p-value for (purpose) of education is > 0.05 indicating that we accept the null hypothesis that there is no significance of the parameter to the model.  The performance of the Logistic Regression model for the test dataset shows that the Logistic Regression recognizes 23 of the 28 Defaults (82%) and predicts the defaults 42 of the 72 good loans (58%). 

Keywords: German-Credit Dataset; Regression; Logistic Regression Analysis Using R.

Introduction

This project examines and analyze the dataset of (german.credit.csv).  The dataset is downloaded from the following archive site for machine learning repository:

https://archive.ics.uci.edu/ml/datasets/Statlog+(German+Credit+Data).

The dataset has 1000 obeservation on 22 variables. There are two datasets for german.credit.csv.  The original dataset, in the form provided by Professor Hofmann, contains categorical/symbolic attributes and is in the current german.credit.csv file which is used in this project.  The other dataset “german.data-numeric” is not used in this project which was developed by Strathclyde University for algorithms that need numerical attributes.  This project utilized the original version of german.credit.csv which has categorical variables, because these categorical variables will be transformed during the analysis process to generate various models including linear models, and logistic model.

There are two Parts for this project.  Part-I addresses five tasks to examine and understand the dataset using R before the analysis as follows:

• Task-1: Review the Variables of the Dataset
• Task-2: Understand the Dataset using names(), head(), and dim() functions
• Task-3: Pre and Post Factor and Level of Categorical Variables of the Dataset.
• Task-4: Summary and Plot the Continuous Variables: Duration, Amount, and Installment
• Task-5: Classify Amount into Groups.
• Task-6: Summary of all selected variables.

Part-II address the analysis using R. Part-II includes seven tasks include the following. These seven tasks are followed by the discussion and analysis of the results. 

• Task-1: Select and Plot Specific Variables for this Project.
• Task-2:  Model-1:  Linear Regression – single variable, and Diagnostic Analysis.
• Task-3:  Model-3:  Multiple Regression Analysis
• Task-4:  Discussion and Analysis.

There is an assumption that, on average, lending into default is five times as costly as not lending to a good debtor (default=success=0, no default=failure=1).  The default is defined as “success,” while no defaulting on a loan is defined as “failure.”   If a certain (p) for the probability of default is estimated, the expected costs are 5p, assuming the bank makes the loan, and 1(1-p) if the bank refuses to make the loan. Thus, if 5p < 1-p, the bank can expect to lose less by making the loan than by turning away business.  The following decision rule can be implied: make the loan if the probability of default p < 1/6.  The prediction of Default (Success) is to implemented whenever p > 1/6.

This project analyuze the German Credit dataset.  The two expected outcomes are success (defaulting on the loan), and failure (not defaulting on the loan).  This project implements three models, Linear Regression, Multiple Linear Regression.  However, the emphasis of this project is on the Logistic Regression (Model-3) to estimate the probability of default, using continuous variables (duration, amount, installment, age), and categorical variables (loan history, purpose, rent) as explanatory variables (independent variables).

Various resources were utilized to develop the required code using R. These resources include (Ahlemeyer-Stubbe & Coleman, 2014; Fischetti, Mayor, & Forte, 2017; Ledolter, 2013; r-project.org, 2018).

Part-I:  Understand and Examine the Dataset “german.credit.csv”

Task-1:  Review the Variables of the Data Sets

The purpose of this task is to understand the variables of the dataset.  The dataset is “german.credit” dataset.  The dataset describes the clients who can default on a loan.  There are selected variables out of the 22 variables which are target for this project.  Table 1 and Table 2 summarize these selected variables for this project.  Table 1 focuses on the variables with binary and numerical values, while Table 2 focuses on the variables with categorical values.

Table 1:  Binary (Default) and Continuous (Numerical) Variables

Table 2: Categorical Variables.

Task-2:  Load and Understand the Data Sets Using names(), head(), dim() functions.

            The purpose of this task is to load and understand the data set using names(), head(), and dim() functions as shown in the code below.

• gc <- read.csv(“C:/CS871/german.credit.csv”)
• names(gc)
• head(gc)
• dim(gc)
• gc[1:3,]

Task-3:  Pre and Post Factor and Level of Categorical Variables of the Data Sets

            The purpose of this task is to factor and level all categorical variables and show the result pre- and post this process as shown in the code and snapshots below.

• ### history categorical variable pre and post factor and level.
• summary(gc$history)
• plot(gc$history, col=”green”, xlab=”History Categorical Variable Pre Factor and Level”)
• gc$history = factor(gc$history, levels=c(“A30”, “A31”, “A32”, “A33”, “A34”))
• levels(gc$history)=c(“good-others”, “good-thisBank”, “current-paid-duly”, “bad-delayed”, “critical”)
• summary(gc$history)
• plot(gc$history, col=”green”, xlab=”History Categorical Variable Post Factor and Level”)

Figure 1. Pre and Post Factor and Level of History Categorical Variable.

• ##### purpose pre and post factor and level
• summary(gc$purpose)
• plot(gc$purpose, col=”darkgreen”)
• ###tranform purpose
• gc$purpose <- factor(gc$purpose, levels=c(“A40″,”A41″,”A42″,”A43″,”A44″,”A45″,”A46″,”A48″,”A49″,”A410”))
• levels(gc$purpose) <- c(“newcar”,”usedcar”,”furniture/equipment”,”radio/television”,”domestic appliances”,”repairs”, “edu”,”vacation-doesNotExist”, “retraining”, “business”, “others”)
• summary(gc$purpose)
• plot(gc$purpose, col=”darkgreen”)

Figure 2. Pre and Post Factor and Level of Purpose Categorical Variable.

• ### housing categorical variable pre and post factor and level.
• summary(gc$housing)
• gc$rent <- factor(gc$housing==”A151″)
• summary(gc$rent)

• ##### housing categorical variable pre and post factor and level
• summary(gc$housing)
• plot(gc$housing, col=”orange”)
• gc$housing <- factor(gc$housing, levels=c(“A151”, “A152”, “A153”))
• levels(gc$housing) <- c(“rent”,”own”,”for free”)
• summary(gc$housing)

Figure 3. Pre and Post Factor and Level of Housing Categorical Variable.

 Task-4:  Summary & Plot the Numerical Variables: Duration, Amount, Installment, & Age.

            The purpose of this task is to obtain the summary of the numerical variables: duration, amount, installment and age and plot the result.

• ##summary and plot of those numerical variables
• summary(gc$duration)

Figure 4. Duration Plot as a Numerical and Continous Variable.

• summary(gc$amount)
• plot(gc$amount, col=”blue”, main=”Amount Numerical Variable”)

Figure 5. Amount Plot as a Numerical and Continous Variable.

• summary(gc$installment)
• plot(gc$installment, col=”blue”, main=”Installment Numerical Variable”)

Figure 6. Installment Plot as a Numerical and Continous Variable.

• summary(gc$age)
• plot(gc$age, col=”blue”, main=”Age Numerical Variable”)

Figure 7.  Age Plot as a Numerical and Continous Variable.

Task-5:  Classify the Amount into Groups

            The purpose of this task is to classify the amount into groups as shown below.

• #### To classify the amount into groups
• gc$amount <-as.factor(ifelse(gc$amount <=2500, ‘0-2500′, ifelse(gc.df$amount<=5000,’2600-5000’, ‘5000+’)))
• summary(gc$amount)

Task-6:  Summary of all variables

The purpose of this task is to obtain the result of all required variables.

• summary(gc$duration)
• summary(gc$amount)
• summary(gc$installment)
• summary(gc$age)
• summary(gc$history)
• summary(gc$purpose)
• summary(gc$housing)
• summary(gc$rent)

Part-II: Plots, Discussion and Analysis

Task-1:  Select and Plot specific variables for this project

            The purpose of this task is to select the specific required variable and plot them into one single diagram.

• ##for demonstration, cut the dataset to these selected variables
• ##(duration, amount, installment, and age) which are numeric and
• ##(history, purpose and housing) which are categorical and
• ## Default (representing the risk) which is binary.
• gc.sv <- gc[,c(“Default”, “duration”, “amount”, “installment”, “age”, “history”, “purpose”, “foreign”, “housing”)]
• gc.sv[1:3,]
• summary(gc.sv)
• plot(gc.sv, col=”red”)
• ### Setting the Rent
• gc$rent <- factor(gc$housing==”A151″)
• summary(gc$rent)

Figure 8.  Plot of all Selected Variables.

Task-2: Model-1: Linear Regression – single variable, and Diagnostic Analysis

                The purpose of this task is to implement Model-1 for Linear Regression using one variable (History) as a factor for the Default.

• # to get a plot of the Default as dependent and history as independent
• plot(Default~history, data=gc, col=”darkred”)
• ##calculate the mean first – but no mean because of NA
• mean(gc$Default)
• # to check the NA values
• gc$Default
• ##we have to remove null values
• mean(gc$Default, na.rm=T)
• ##get the mean without NA values
• mean.Default=mean(gc$Default,na.rm=T)
• ## plot again after removing the NA value
• plot(Default~history, data=gc, col=”darkred”, main=”Default and History Effect”)
• #put horizontal line at the mean
• abline(h=mean.Default)
• #use lm to fit a regression line through these data:
• model1.Default.history = lm(Default ~ history, data=gc)
• model1.Default.history
• abline(model1.Default.history, col=”yellow”)
• plot(model1.Default.history)
• # to see the effect of solar
• termplot(model1.Default.history)
• #summary
• summary(model1.Default.history)

Figure 9.  Model-1: Linear Regression Using the Mean Line (black).

Figure 10.  Model-1: Linear Regression Using Regression Line (yellow).

Figure 11.  Model-1: Four Diagnostic Plots For Standard Linear Regression.

Task-3: Model-2: Multiple Regressions Analysis

            The purpose of this task is to implement Model-2 for Multiple Regressions including the Purpose, History, Installment, Housing, Amount, Duration, Age which can influence the Default as dependent variable.

• #Find relationship between multiple variables
• ##purpose on Default conditioned with history.
• coplot(Default~purpose|history, panel=panel.smooth, gc)  
• ##rent on Default conditioned with history.
• coplot(Default~rent|history, panel=panel.smooth, gc)  
• ##housing on Default conditioned with history.
• coplot(Default~housing|history, panel=panel.smooth, gc)  

Figure 12.  Model-2: Multiple Linear Regression Using Purpose, Housing, Rent, conditioned with History as factors influencing Default.

• ##amount on Default conditioned with duration.
• coplot(Default~amount|duration, panel=panel.smooth, gc)  
• ##installment on Default conditioned with duration.
• coplot(Default~installment|duration, panel=panel.smooth, gc)
• ##age on Default conditioned with duration.
• coplot(Default~age|duration, panel=panel.smooth, gc)  

Figure 13.  Model-2: Multiple Linear Regression Using Amount, Duration as Factors influencing Default.

Task-3: Model-3:  Logistic Regression Analysis

            The purpose of this task is to implement the Logistic Regression Analysis.

• summary(gc)

• ###Create a Design Matrix
• ##Factor variables are turned into indicator variables
• ##The first column of ones is ommitted
• Xgc <- model.matrix(Default~.,data=gc)[,-1]
• Xgc[1:3,]

• ## Create training and prediction datasets
• ## Select 900 rows for estimation and 100 for testing
• set.seed(1)
• train <- sample(1:1000,900)
• xtrain <- Xgc[train,]
• xnew <- Xgc[-train,]
• ytrain <- gc$Default[train]
• ynew <- gc$Default[-train]
• gcglm=glm(Default~.,family=binomial,data=data.frame(Default=ytrain,xtrain))
• summary(gcglm)
• plot(gcglm, col=”blue”)
• coef(gcglm)

Figure 14.  Model-3: Logistic Regression: Diagnostic Analysis: Residuals v. Fitted.

Figure 15.  Model-3: Logistic Regression: Diagnostic Analysis: Normal Q-Q.

Figure 16.  Model-3: Logistic Regression: Diagnostic Analysis: Scale-Location.

Figure 17.  Model-3: Logistic Regression: Diagnostic Analysis: Residuals vs. Leverage.

• ## Now to prediction: what are the underlying default probabilities
• ## for cases in the test set
• ptest <- predict(gcglm, newdata=data.frame(xnew),type=”response”)
• data.frame(ynew,ptest)

• ## Using probability cutoff 1/6
• ## coding as 1 (predicting default) if probability 1/6 or larger
• probability.one.sixth=floor(ptest+(5/6))
• probability1=table(ynew,probability.one.sixth)
• probability1
• error=(probability1[1,2]+probability1[2,1])/100
• error

• ### Binary Classification, Using Probabiillties to make decision, sensitivity and specificity,
• ### Cut 1/6
• cut=1/6
• true.positive <- ynew==1 & ptest >= cut
• true.positive
• true.negative <- ynew==0 & ptest < cut
• true.negative
• # Sensitivity (predict default when it does happen)
• sum(true.positive)/sum(ynew==1)
• # Specificity (predict no default when it does not happen)
• sum(true.negative)/sum(ynew==0)

• ## Using probability cutoff 1/2
• ## coding as 1 if probability 1/2 or larger
• cut=1/2
• probability.half=floor(ptest+(1/2))
• probability2=table(ynew,probability.half)
• probability2
• error=(probability2[1,2]+probability2[2,1])/100
• error
• true.positive <- ynew==1 & ptest >=cut
• true.negative <- ynew==0 & ptest < cut
• # Sensitivity (predict default when it does happen)
• sum(true.positive)/sum(ynew==1)
• # Specificity (predict no default when it does not happen)
• sum(true.negative)/sum(ynew==0)

• ## R macro for plotting the ROC curve
• ## plot the ROC curve for classification of y with p
• roc <- function(p,y){
• y <- factor(y)
• n <- length(p)
• p <- as.vector(p)
• Q <- p > matrix(rep(seq(0,1,length=500),n),ncol=500,byrow=TRUE)
• fp <- colSums((y==levels(y)[1])*Q)/sum(y==levels(y)[1])
• tp <- colSums((y==levels(y)[2])*Q)/sum(y==levels(y)[2])
• plot(fp, tp, xlab=”1-Specificity”, ylab=”Sensitivity”)
• abline(a=0,b=1,lty=2,col=8)
• }

• ###### ROC for hold-out period
• roc(p=ptest,y=ynew)
• ## ROC for all cases (in-sample)
• gcglmall <- glm(gc$Default ~ Xgc,family=binomial)
• roc(p=gcglmall$fitted, y=gcglmall$y)
• ## using the ROCR package to graph the ROC curves
• library(ROCR)

Figure 18.  Model-3: Logistic Regression: Sensitivity and Specificity for a sample of the data.

Figure 19.  Model-3: Logistic Regression: Sensitivity and Specificity for All the data.

• ## input is a data frame consisting of two columns
• ## predictions in first column and actual outcomes in the second
• ## ROC for hold-out period
• predictions=ptest
• labels=ynew
• data=data.frame(predictions,labels)
• data
• ## prediction1: function to create prediction objects
• prediction1 <- prediction(data$predictions,data$labels)
• prediction1

• ## perf: creates the input to be plotted
• ## sensitivity and one minus specificity (the false positive rate)
• performance <- performance(prediction1, “sens”, “fpr”)
• performance
• plot(performance)

Figure 20.  Model-3: Logistic Regression: ROC for specifc cases.

• ## ROC for all cases (in-sample)
• gcglmall <- glm(gc$Default ~ Xgc,family=binomial)
• predictions=gcglmall$fitted
• labels=gcglmall$y
• data=data.frame(predictions,labels)
• prediction2 <- prediction(data$predictions,data$labels)
• performance2 <- performance(prediction2, “sens”, “fpr”)
• plot(performance2)

Figure 21.  Model-3: Logistic Regression: ROC for all cases.

Task-4:  Discussion and Analysis

The descriptive analysis of the history (Figure 1) shows that the existing credits paid back duly till now are ranked number one of 530, followed by the critical account of 293, followed by the category of delay in paying off in the past of 88.  The category of all credits at this bank paid back duly include only 49 and no credits taken or all credits paid back duly has only 40.  The descriptive analysis of the purpose (Figure 2) shows that the radio/TV of 280 ranks number one in the loan, followed by the new car category of 234, followed by the furniture/equipment of 181, and used car of103.  The business and education have 97 and 50 respectively, while the repairs, appliances, retaining and others have the least rank for loans. The descriptive analysis of the housing (Figure 3) shows that the own (Own=713) ranks number one for those who receive loans, followed by the rent (Rent=179).  The category of “free” housing (Free=108) ranks the last for receiving loans.  The descriptive analysis of the duration (Figure 4) shows that the duration average (Mean=21) is higher than the Median duration in a month (Median=18), indicating a positively skewed distribution.  The maximum duration has the value of 72 months, while the minimum duration for the loan is four months.  Most of the duration periods are up to 60 months.  There is an outlier of the duration period of ~70 months as illustrated in Figure 4.  The descriptive analysis of the amount (Figure 5) shows that the average amount of the loan (Mean=3271), is higher than the Median amount of the loan (Median=2320) indicating a positively skewed distribution.  The maximum amount for a loan is 18424, while the minimum amount is 250.  Most loans have the amount of ~5000, followed by the amount of 10,000-15,000.  There is an outlier of a loan amount above 15,000 as illustrated in Figure 5.  The descriptive analysis of the installment (Figure 6) shows that average installment (Mean=2.98) is higher than the Median installment (Median=1.00) indicating a positively skewed distribution.  The maximum installment rate is 4.00, while the minimum installment has a rate of 1.00.  The installment rates as illustrated in (Figure 6) are categorized between 1.00, 2.00, 3.00 and 4.00. The descriptive analysis of the age (Figure 7) shows that the average age (Mean=36) which is higher than the Median age (Median=33) indicating a positively skewed distribution.  The maximum age is 75, while the minimum age is 19.  As illustrated in Figure 7, most of the age is from ~20 to 50, with few densities from 50 to 60 and the fewest density is above 60 and 70.  

The Linear Regression is the first model (Model-1) is implemented on the (history) explanatory variable over the default.  The mean (Mean=3) is plotted in Figure 9 of the Linear Regression of history over the default.  Another line is plotted in Figure 10 is illustrated the Linear Regression using the Regression Line.  The result shows that there is a negative relationship between the Default (success) and the history.  The bad history shows the success in defaulting on a loan. The diagnostic plots of the standard regression are also discussed in this project. Figure 11 illustrates four different diagnostic plots of the standard regression. This analysis also covers the residuals and fitted lines.  Figure 11 illustrated the Residuals vs. Fitted in Linear Regression Model for the History explanatory variable as a function of the Default.  The residuals depict the difference between the actual value of the response variable and the response variable predicted using the regression equation (Hodeghatta & Nayak, 2016). The principle behind the regression line and the regression equation is to reduce the error or this difference (Hodeghatta & Nayak, 2016).  The expectation is that the median value should be near zero (Hodeghatta & Nayak, 2016).  For the model to pass the test of linearity, no pattern in the distribution of the residuals should exist (Hodeghatta & Nayak, 2016).  Where there is no pattern in the distribution of the residuals, it passes the condition of linearity (Hodeghatta & Nayak, 2016). The plot of the fitted values against the residuals with the line shows the relationship between the two. The horizontal and straight line indicates that the “average residual” for all “fitted values” it is more or less the same (Navarro, 2015).  The result of the Linear Regression for the identified variables of History and Default shows that the residual has a curved pattern, indicating that a better model can be obtained using the quadratic term because ideally, this line should be a straight horizontal line.  Figure 11 also illustrates the Normal QQ plot, which is used to test the normality of the distribution (Hodeghatta & Nayak, 2016).  The residuals are not on the straight line, indicating that the residuals are not normally distributed. Hence, the normality test of the residuals did not pass, as it is supposed to be a straight line for the residuals to pass.  Figure 11 also illustrates the Scale-Location graph, which is one of the graphs generated as part of the plot. The points are spread in a random fashion around the horizontal line but not equally the line.  If the horizontal line with equally randomly spread points, the result could indicate that the assumption of constant variance of the errors or homoscedasticity is fulfilled (Hodeghatta & Nayak, 2016).  Thus, it is not fulfilled in this case.  Figure 11 also illustrates the Residuals vs. Leverage Plot generated for the Linear Regression Model. In this plot of Residuals vs. Leverage, the patterns are not as relevant as the case with the diagnostics plot of the Linear Regression. In this plot, the outlying values at the upper right corner or the lower right are watched (Bommae, 2015).  Those spots are the places where a case can be influential against a regression line (Bommae, 2015).  When cases are outside of the Cook’s distance, meaning they have high Cook’s distance scores, the cases are influential to the regression results (Bommae, 2015).  The Cook’s distance lines are (red dashed line) are far indicating there is no influential case.

The Multiple Linear Regression is the second Model.  Multiple variables such as purpose conditioned with history, rent conditioned with history, housing conditioned with history are used to implement the Multiple Regression models.  There are two most noticeable results (Figure 12).  The first noticeable result shows that the category of “good to other banks” that the default as success goes down when the loans are made for (purpose) of new car, furniture, radio/TV, domestic appliances, and repair, while the default as success goes up – meaning failure to default) with education, retraining, business and others.  The second noticeable result is illustrated in the category of “currently paid duly,” that the default as success goes down meaning the loan receiver default on the loan when the loan is made to the same categories as the first noticeable result, but until radio/TV, after which the default on loan will fail for (purpose) of domestic appliances, repair, education, retraining, business, and others.   The other Multiple Regression used the rent and free, the history shows a failure to default for the “good to others” category, while the own category shows success to default for the “current paid duly” category.   For the rent specific, the result shows that these specific categories, there no default on loan with the rent on both categories.  For the amount, installment and age conditioned with duration Multiple Regression models, the result shows that the loan 5000+, when the duration increases, the default failure increases, and when the duration decreases, the default failure decreases meaning there is a significant chance for defaulting on the loan. The same pattern shows when the installment increases, the default on a loan decreases.  The same pattern is also shown for the age, when the age increases, the default on the loan decreases.

The Logistic Regression Model (Model-3) is the emphasis of this project. The Logistic Regression is used to predict the probability of Default on loan based on other independent variables. In this case, the probability of the Default based on the identified required independent and explanatory variables of duration, amount, installment, age (continuous variables), history, purpose, and rent (categorical variables). 

A random selection of 900 of the 1000 cases is made for the Training set, and the remaining of 100 cases is made into the Test set.  The result shows the coefficient estimates, starting with a negative intercept (slope) of -2.705e-01 with a standard error of 4.833e-01, and the coefficient for each of these identified explanatory variables are also estimated.  The standard error of the coefficient estimates represents the accuracy of the coefficient. The larger the standard error, the less confident about the estimates.  The (z) value represents the z-statistic, which is the coefficient estimate divided by the standard error of the estimates.  The Pr(>|z|) is the last column in the result of the logistic analysis, where the p-value corresponding to the z-statistic.  The smaller the p-value, the more significant the estimates are.  One unit increase in duration (an increase in the number of duration by one) is associated with an increase of logarithm of the odds of the observation being Default.  As indicated in (Fischetti et al., 2017), if the coefficient is positive, it has a positive impact on the probability of the dependent variable, and if the coefficient is negative, it has a negative impact on the probability of the binary outcome (Default).  The age, history with poor and terrible, the purpose for used care, goods or repairs, education, and business, have negative coefficient values indicating that there is a negative impact on the probability of the binary outcome of the Default.  The duration, amount, installment and rent show positive coefficient values indicating that they have a positive impact on the probability of the dependent binary outcome (Default).  As the p-value is much less than 0.05 for duration, amount, installment, history, purpose for used car, goods, repair, and business, and rent, we reject the null hypotheses that there is no significance of the parameter to the model and accept the alternate hypotheses that there is significance of the parameter to the model.  The p-value for the age is (p=0.05) indicating that we accept the null hypothesis that there is no significance of the parameter to the model.  The p-value for (purpose) of education is > 0.05 indicating that we accept the null hypothesis that there is no significance of the parameter to the model.  The performance of the Logistic Regression model for the Test dataset shows that the Logistic Regression recognizes 23 of the 28 Defaults (82%) and predicts the defaults 42 of the 72 good loans (58%).  The coordinates of the sensitivity and specificity (sensitivity = 0.82, 1-specifictiy=0.58) define one point on the ROC curve (Figure 18).  The sensitivity = 8/28=0.29, and specificity =68/72=0.94 for another point on ROC curve (sensitivity = 0.29, 1-specificity=0.06) (Figure 19).  The ROC is calculated using various cutoff on the probability.  The ROC curves that assess the predictive quality of the classification rule on the holdout sample of 100 observations as shown in (Figure 18), and on the complete data set of all 1000 cases as shown in (Figure 19).  Specified values on sensitivity and specificity imply a certain value for the probability cutoff.  However, for certain data and models, no cutoff may achieve the given desires properties on sensitivity and specificity, implying that the desired sensitivity and specificity cannot be attained (Ledolter, 2013).

References

Ahlemeyer-Stubbe, A., & Coleman, S. (2014). A practical guide to data mining for business and industry: John Wiley & Sons.

Bommae, K. (2015). Understanding Diagnostic Plots of Linear Regression Analysis. Retrieved from https://data.library.virginia.edu/diagnostic-plots/.

Fischetti, T., Mayor, E., & Forte, R. M. (2017). R: Predictive Analysis: Packt Publishing.

Hodeghatta, U. R., & Nayak, U. (2016). Business Analytics Using R-A Practical Approach: Springer.

Ledolter, J. (2013). Data mining and business analytics with R: John Wiley & Sons.

Navarro, D. J. (2015). Learning statistics with R: A tutorial for psychology students and other beginners. R package version 0.5.

r-project.org. (2018). R: A Language and Environment for Statistical Computing. Retrieved from https://cran.r-project.org/doc/manuals/r-release/fullrefman.pdf.