## scikit-learn : Data Preprocessing I - Missing / categorical data

In real-world samples, it is not uncommon that there are missing one or more values such as the blank spaces in our data table.

Quite a few computational tools, however, are unable to handle such missing values and might produce unpredictable results. So, before we proceed with further analyses, it is critical that we take care of those missing values.

To get a better feel for the problem, let's create a simple example using CSV file:

to get a better grasp of the problem:The **StringIO()** function allows us to read the string assigned to csv_data into a
pandas **DataFrame** via the **read_csv()** function as if it was a regular CSV file on our hard drive.

Note that the two missing cells were replaced by **NaN**.

We can use **isnull()** method to check whether a cell contains a numeric value ( False ) or if data is missing ( True ):

For a larger DataFrame, we may want to use the **sum()** method which returns the number of missing values per column:

Note that we can always access the underlying **NumPy** array of the DataFrame via the values attribute before
we feed it into a scikit-learn estimator:

So, while scikit-learn was developed for working with NumPy
arrays, it can sometimes be more convenient to preprocess data
using pandas' **DataFrame**.

We can remove the corresponding features (columns) or samples (rows) from the dataset.

The rows with missing values can be dropped via the **pandas.DataFrame.dropna()** method:

We can drop columns that have at least one NaN in any row by setting the axis argument to 1:

where axis : {0 or 'index', 1 or 'columns'}.

The dropna() method has several additional parameters:

The removal of missing data appears to be a convenient approach, however, it also comes with certain disadvantages:

- There are chances that we may end up removing too many, which will make our analysis not reliable.
- By eliminating too many feature columns, we may run the risk of losing valuable information for our classifier.

In the next section, we will look into **interpolation techniques** which one of the most commonly used alternatives for dealing with missing data.

Mean imputation is a method replacing the missing values with the mean value of the entire feature column. While this method maintains the sample size and is easy to use, the variability in the data is reduced, so the standard deviations and the variance estimates tend to be underestimated.

We'll use the **sklearn.preprocessing.Imputer** class:

Here, we replaced each NaN value by the corresponding mean from each feature column.

We can use the row means if we change the setting axis=0 to axis=1:

As for the **strategy** parameter, there are other options such as **median** or **most_frequent**.

The **Imputer** class we used in the previous section belongs to the so-called **transformer** classes in scikit-learn that are used for data transformation.

There are two methods of estimators:

**fit**:

This method is used to learn the parameters from the training data.**transform**:

transform() method uses those parameters to transform the data.

The diagram below shows how a transformer fitted on the training data is used to transform a training dataset as well as a new test dataset:

In supervised learning tasks, we additionally provide the class labels for fitting the model, which can then be used to make predictions about new data samples via the **predict()** method:

picture source : Python machine learning by Sebastian Raschka

Not all data has numerical values. Here are examples of **categorical** data:

- The blood type of a person: A, B, AB or O.
- The state that a resident of the United States lives in.
- T-shirt size. XL > L > M
- T-shirt color.

Even among categorical data, we may want to distinguish further between
**nominal** and **ordinal** which can be sorted or ordered features. So, T-shirt size can be an ordinal feature, because we can define an order XL > L > M.

Let's create a new categorical data frame:

As we can see from the output, the DataFrame contains a nominal feature (color), an ordinal feature (size) as well as a numerical feature (price) column. In the last column, the class labels are created.

In order for out learning algorithm to interpret the ordinal features correctly, we should convert the categorical string values into integers.

However, since there is no convenient function that can automatically derive the correct order of the labels of our size feature, we have to define the mapping manually.

Let's assume that we know the difference between features such as XL = L + 1 = M + 2.

If we want to transform the integer values back to the original string representation, we can simply define a reverse-mapping dictionary "inv_size_mapping" that can then be used via the pandas' map method on the transformed feature column similar to the "size_mapping" dictionary that we used previously:

Usually, class labels are required to be encoded as integer values.

While most estimators for classification in scikit-learn convert class labels to integers internally, as a good practice, we may want to provide class labels as integer arrays to avoid any issues.

To encode the class labels, we can use an approach similar to the mapping of ordinal features discussed in the previous section.

Since class labels are not ordinal, it doesn't matter which integer number we assign to a particular string-label.

So, we can simply enumerate the class labels starting at 0:

Now we want mapping dictionary to transform the class labels into integers:

As we did for "size" in the previous section, we can reverse the key-value pairs in the mapping dictionary as follows to map the converted class labels back to the original string representation:

Though we encoded the class labels manually, luckily, there is a convenient **LabelEncoder** class directly implemented in scikit-learn to achieve the same:

Note that the **fit_transform()** method is just a shortcut for calling fit and
transform separately, and we can use the **inverse_transform()** method to
transform the integer class labels back into their original string representation:

So far, we used a simple dictionary-mapping approach to convert the ordinal size feature into integers.

Because scikit-learn's estimators treat class labels without any order, we used the convenient **LabelEncoder** class to encode the string
labels into integers.

We can use a similar approach to transform the nominal color column of our dataset as well:

We may want to create a new **dummy feature** for each
unique value in the nominal feature column. In other words, we would convert the **color**
feature into three new features: blue, green, and red. Binary values can then be
used to indicate the particular color of a sample; for example, a blue sample can be
encoded as blue=1, green=0, red=0. This technique is called **one-hot encoding**.

In order to perform this transformation, we can use the **scikit-learn.preprocessingOneHotEncoder**:

Note that when we initialized the **OneHotEncoder**, we defined the column position of the
variable that we want to transform via the categorical_features parameter which is the first column in the feature matrix X.

The **OneHotEncoder**, by default,
returns a sparse matrix when we use the **transform()** method, and we converted the
sparse matrix representation into a regular (**dense**) NumPy array for the purposes of
visualization via the **toarray()** method. Sparse matrices are simply a more efficient
way of storing large datasets, and one that is supported by many scikit-learn
functions. It is especially useful if it contains a lot of zeros.

If we want to omit the **toarray()** step, we may initialize the encoder as **OneHotEncoder(...,sparse=False)** to return
a regular NumPy array:

Another way which is more convenient is to create those dummy features via one-hot encoding
is to use the **pandas.get_dummies()** method. Applied on a **DataFrame**,
the **get_dummies()** method will only convert string columns and leave all other
columns unchanged:

Source is available from bogotobogo-Machine-Learning .

**Next:**

# Machine Learning with scikit-learn

scikit-learn installation

scikit-learn : Features and feature extraction - iris dataset

scikit-learn : Machine Learning Quick Preview

scikit-learn : Data Preprocessing I - Missing / Categorical data

scikit-learn : Data Preprocessing II - Partitioning a dataset / Feature scaling / Feature Selection / Regularization

scikit-learn : Data Preprocessing III - Dimensionality reduction vis Sequential feature selection / Assessing feature importance via random forests

Data Compression via Dimensionality Reduction I - Principal component analysis (PCA)

scikit-learn : Data Compression via Dimensionality Reduction II - Linear Discriminant Analysis (LDA)

scikit-learn : Data Compression via Dimensionality Reduction III - Nonlinear mappings via kernel principal component (KPCA) analysis

scikit-learn : Logistic Regression, Overfitting & regularization

scikit-learn : Supervised Learning & Unsupervised Learning - e.g. Unsupervised PCA dimensionality reduction with iris dataset

scikit-learn : Unsupervised_Learning - KMeans clustering with iris dataset

scikit-learn : Linearly Separable Data - Linear Model & (Gaussian) radial basis function kernel (RBF kernel)

scikit-learn : Decision Tree Learning I - Entropy, Gini, and Information Gain

scikit-learn : Decision Tree Learning II - Constructing the Decision Tree

scikit-learn : Random Decision Forests Classification

scikit-learn : Support Vector Machines (SVM)

scikit-learn : Support Vector Machines (SVM) II

Flask with Embedded Machine Learning I : Serializing with pickle and DB setup

Flask with Embedded Machine Learning II : Basic Flask App

Flask with Embedded Machine Learning III : Embedding Classifier

Flask with Embedded Machine Learning IV : Deploy

Flask with Embedded Machine Learning V : Updating the classifier

scikit-learn : Sample of a spam comment filter using SVM - classifying a good one or a bad one

### Machine learning algorithms and concepts

Batch gradient descent algorithmSingle Layer Neural Network - Perceptron model on the Iris dataset using Heaviside step activation function

Batch gradient descent versus stochastic gradient descent

Single Layer Neural Network - Adaptive Linear Neuron using linear (identity) activation function with batch gradient descent method

Single Layer Neural Network : Adaptive Linear Neuron using linear (identity) activation function with stochastic gradient descent (SGD)

Logistic Regression

VC (Vapnik-Chervonenkis) Dimension and Shatter

Bias-variance tradeoff

Maximum Likelihood Estimation (MLE)

Neural Networks with backpropagation for XOR using one hidden layer

minHash

tf-idf weight

Natural Language Processing (NLP): Sentiment Analysis I (IMDb & bag-of-words)

Natural Language Processing (NLP): Sentiment Analysis II (tokenization, stemming, and stop words)

Natural Language Processing (NLP): Sentiment Analysis III (training & cross validation)

Natural Language Processing (NLP): Sentiment Analysis IV (out-of-core)

Locality-Sensitive Hashing (LSH) using Cosine Distance (Cosine Similarity)

### Artificial Neural Networks (ANN)

[Note] Sources are available at Github - Jupyter notebook files1. Introduction

2. Forward Propagation

3. Gradient Descent

4. Backpropagation of Errors

5. Checking gradient

6. Training via BFGS

7. Overfitting & Regularization

8. Deep Learning I : Image Recognition (Image uploading)

9. Deep Learning II : Image Recognition (Image classification)

10 - Deep Learning III : Deep Learning III : Theano, TensorFlow, and Keras

Ph.D. / Golden Gate Ave, San Francisco / Seoul National Univ / Carnegie Mellon / UC Berkeley / DevOps / Deep Learning / Visualization