How, when and why should you normalize/standardize/resize your data? (2023)

Last updated on May 28, 2020 byeditorial team

Author(s):Swetha Lakshmanan

Before we dive into this topic, let's start with some definitions.

"Re-scale"A vector means adding or subtracting a constant and then multiplying or dividing by a constant, just as you would to change units of measurement for data, for example, to convert a temperature from Celsius to Fahrenheit.

"Normalization"A vector generally means dividing by a norm of the vector. It is also often referred to as the minimum and range scale of the vector to get all elements between 0 and 1, making all numeric values ​​in the column equal.recordfor a commonScale.

"Standardize"A vector usually means subtracting a measure of location and dividing by a measure ofScale. For example, if the vector contains random values ​​with a Gaussian distribution, you could subtract the mean and divide by the standard deviation, getting a "standard normal".random variablewith mean 0 and standard deviation 1.

After reading this post you will know:

• Why should you standardize/normalize/scale your data?
• How to standardize your numeric attributes to a mean of 0 and variance of unit using a standard scalar
• How to normalize your numeric attributes between the range of 0 and 1 using the min-max scalar
• How to normalize with a robust scalar
• When should you choose standardization or normalization?

Let's start.

Why you should standardize/normalize variables:

Standardization:

Standardizing features around the center and 0 with a standard deviation of 1 is important when comparing measures with different units. Variables measured on different scales do not contribute equally to the analysis and may end up forming a baseline.

Example: A variable that is between 0 and 1000 outperforms a variable that is between 0 and 1. Using these variables without standardization produces the variable with the highest rank weight of 1000 in the analysis. This problem can be avoided by transforming the data to comparable scales. Typical data standardization techniques compensate for the range and/or variability of the data.

Normalization:

Also, the purpose of normalization is to change the values ​​of the numeric columns torecordon a common scale without distorting differences in value ranges. Formachine learning, not all data sets require normalization. It is only needed when the resources have different ranks.

For example, consider a data set that contains two characteristics, age and income (x2). Where age ranges from 0 to 100 while income ranges from 0 to 100,000 and above. Income is about 1,000 times greater than old age. These two characteristics are therefore in very different realms. If we carry out a deeper analysis, e.g. B. multivariatelinear regressionFor example, the imputed income will have a greater impact on the result itself due to its higher value. However, this does not necessarily mean that it is more important as a predictor. So we normalize the data to bring all the variables to the same range.

When should you use normalization and standardization?

normalizationThis is a good technique when you don't know the distribution of your data, or when you know that the distribution is not Gaussian (a bell curve). Normalization is useful when your data have different scales and thealgorithmIt does not make assumptions about the disclosure of your data, such ask-nearest neighborsand artificialneural networks.

Standardizationit assumes that your data has a Gaussian distribution (bell curve). This does not necessarily have to be the case, but the technique is more efficient when the attribute distribution is Gaussian. Standardization is useful when your data have different scales and thealgorithmYou use assumptions about your data with a Gaussian distribution, such aslinear regression, logistic regression and linear discriminant analysis.

Record:

i used thoseLending Club Loan Recordby Kaggle to demonstrate examples in this article.

Import libraries:

matterpandasas pdimport numpy as np import seaborn as snsimport matplotlib.pyplot as plt

Import registration:

Let's import three columns: loan amount, int_rate, and rate, and the first 30,000 rows in the dataset (to reduce computation time)

cols = ['prestamo_amnt', 'int_rate', 'rate']data = pd.read_csv('loan.csv', nrows = 30000, usecols = cols)

If you import all the data, there will bemissing valuesin some columns. You can simply remove the lines withmissing valuesUse ofpandaspost-drop welding method.

Basic analysis:

Now let's look at the basic statistical values ​​of our data set.

data.describe()

Different variables have different ranges of values, that is, different orders of magnitude. Not only are the minimum and maximum values ​​different, but they are also spread over areas of different widths.

Standardization (standard scalar):

As we discussed earlier, standardization (or z-score normalization) means centering the variable at zero and normalizing the variance at 1. The procedure involves subtracting the mean from each observation and then dividing by the standard deviation:

The result of standardization is that the features are scaled to have the properties of a standard normal distribution.

μ=0 y σ=1

where μ is the mean (average) and σ is the standard deviation of the mean.

CODE:

sci-kit-learn's StandardScaler removes the mean and scales the data to unit variance. We can import the StandardScalar method from sci-kit learn and apply it to our dataset.

aus sklearn.preprocessing import StandardScalerscaler = StandardScaler() data_scaled = scaler.fit_transform(datos)

Now we are going to check the mean and standard deviation values

print(data_scaled.mean(axis=0))print(data_scaled.std(axis=0))

As expected, the mean of each variable is now approximately zero, and the standard deviation is set to 1. Therefore, all variable values ​​are in the same range.

print('Minimum Values ​​(Loan Amount, Int. Rate and Commission): ', data_scaled.min(axis=0))print('Maximum Amounts (Loan Amount, Int. Rate and Commission): ', data_scaled. max(axis=0))

However, the minimum and maximum values ​​vary according to the initial dispersion of the variable and are strongly influenced by the presence of outliers.

Normalization (scalar min-max):

With this approach, the data is scaled to a fixed range, typically 0 to 1.
Unlike standardization, the price of this narrow range is that we end up with smaller standard deviations that can suppress the effect of outliers. Therefore, MinMax Scalar is sensitive to outliers.

A min-max scale is normally done using the following equation:

CODE:

Let's import MinMaxScalar from scikit-learn and apply it to our dataset.

aus sklearn.preprocessing import MinMaxScalerscaler = MinMaxScaler() data_scaled = scaler.fit_transform(data)

Now we are going to check the mean and standard deviation values.

print('mean (loan amount, interest rate, and fee): ', data_scaled.mean(axis=0))print('std (loan amount, interest rate, and fee): ', data_scaled.std(axis =0))

After MinMaxScaling, the distributions are not centered at zero and the standard deviation is not 1.

print('Min (loan amount, internal rate and fee): ', data_scaled.min(axis=0))print('Max (loan amount, internal rate and fee): ', data_scaled.max(axis=0 ))

However, the minimum and maximum values ​​are standardized across all variables, as opposed to standardization.

Robust scaling (scaling to median and quantiles):

The scale with median and quantiles consists of subtracting the median from all the observations and then dividing by the interquartile difference. Scale features withthe statisticsrobust to outliers.

The interquartile difference is the difference between the 75th and 25th quantiles:

IQR = 75. Quantile — 25. Quantile

The equation to calculate the scaled values:

X_scaled = (X — X. Mediana) / IQR

CODE:

First, import RobustScalar from Scikit Learn.

aus sklearn.preprocessing import RobustScalerscaler = RobustScaler() data_scaled = scaler.fit_transform(datos)

Now check the mean and standard deviation values.

print('mean (loan amount, interest rate, and fee): ', data_scaled.mean(axis=0))print('std (loan amount, interest rate, and fee): ', data_scaled.std(axis =0))

As you can see, the distributions are not centered at zero and the standard deviation is not 1.

print('Min (loan amount, internal rate and fee): ', data_scaled.min(axis=0))print('Max (loan amount, internal rate and fee): ', data_scaled.max(axis=0 ))

Also, the min and max values ​​are not set to a specific upper and lower bound like in MinMaxScaler.

I hope you found this article useful. Have fun with your studies!

references

1. https://www.udemy.com/feature-engineering-for-machine-learning/
2. https://www.geeksforgeeks.org/python-how-and-where-to-apply-feature-scaling/

How, when and why should you normalize/standardize/resize your data?was originally posted onTowards AI - Multidisciplinary Scientific Journalon Medium, where people continue the conversation by starring and responding to that story.

Posted viaAI rumor