# 5 APPENDIX

Complementary reading.

## 5.1 The magic of percentiles

Percentile is such a crucial concept in data analysis that we are going to cover it extensively in this book. It considers each observation with respect to others. An isolated number may not be meaningful, but when it is compared with others the distribution concept appears.

Percentiles are used in profiling as well as evaluating the performance of a predictive model.

**The dataset, an advice before continue:**

This contains many indicators regarding world development. Regardless the profiling example, the idea is to provide a ready-to-use table for sociologists, researchers, etc. interested in analyzing this kind of data.

The original data source is: http://databank.worldbank.org. There you will find a data dictionary that explains all the variables.

In this section we’ll be using a table which is already prepared for analysis. The complete step-by-step data preparation is in Profiling chapter.

Any indicator meaning can be checked in data.worldbank.org. For example, if we want to know what `EN.POP.SLUM.UR.ZS`

means, then we type: http://data.worldbank.org/indicator/EN.POP.SLUM.UR.ZS

### 5.1.1 How to calculate percentiles

There are several methods to get the percentile. Based on interpolations, the easiest way is to order the variable ascendantly, selecting the percentile we want (for example, 75%), and then observing *what is the maximum value if we want to choose the 75% of the ordered population*.

Now we are going to use the technique of keeping a small sample so that we can have maximum control over *what is going on* behind the calculus.

We retain the random 10 countries and print the vector of `rural_poverty_headcount`

which is the variable we are going to use.

```
library(dplyr)
data_world_wide = read.delim(file = "https://goo.gl/NNYhCW",
header = T)
data_sample = filter(data_world_wide, Country.Name %in%
c("Kazakhstan", "Zambia", "Mauritania",
"Malaysia", "Sao Tome and Principe",
"Colombia", "Haiti", "Fiji", "Sierra Leone",
"Morocco")) %>% arrange(rural_poverty_headcount)
select(data_sample, Country.Name, rural_poverty_headcount)
```

```
## Country.Name rural_poverty_headcount
## 1 Malaysia 1.6
## 2 Kazakhstan 4.4
## 3 Morocco 14.4
## 4 Colombia 40.3
## 5 Fiji 44.0
## 6 Mauritania 59.4
## 7 Sao Tome and Principe 59.4
## 8 Sierra Leone 66.1
## 9 Haiti 74.9
## 10 Zambia 77.9
```

Please note that the vector is ordered only for didactic purposes. *As we said in the Profiling chapter, our eyes like order.*

Now we apply the `quantile`

function on the variable `rural_poverty_headcount`

(the percentage of the rural population living below the national poverty lines):

`quantile(data_sample$rural_poverty_headcount)`

```
## 0% 25% 50% 75% 100%
## 1.6 20.9 51.7 64.4 77.9
```

**Analysis**

**Percentile 50%**: 50% of the countries (five of them) have a`rural_poverty_headcount`

below`51.7`

We can check this in the last table: these countries are: Fiji, Colombia, Morocco, Kazakhstan, and Malaysia.**Percentile 25%**: 25% of the countries are below 20.87. Here we can see an interpolation because 25% represents ~2.5 countries. If we use this value to filter the countries, then we’ll get three countries: Morocco, Kazakhstan, and Malaysia.

More information about the different types of quantiles and their interpolations: `help("quantile")`

.

#### 5.1.1.1 Getting semantical descriptions

From the last example we can state that:

*“Half of the countries have as much as 51.7% of rural poverty”**“Three-quarters of the countries have a maximum of 64.4% regarding its rural poverty”*(based on the countries ordered ascendantly).

We can also think of **using the opposite**:

*“A quarter of the countries that exhibit the highest rural poverty values have a percentage of at least 64.4%.”*

### 5.1.2 Calculating custom quantiles

Typically, we want to calculate certain quantiles. The example variable will be the `gini_index`

**What is the Gini index?**

It is a measure of income or wealth inequality.

- A Gini coefficient of
**zero**expresses**perfect equality**where all values are the same (for example, where everyone has the same income). - A Gini coefficient of
**1**(or 100%) expresses**maximal inequality**among values (e.g., for a large number of people, where only one person has all the income or consumption while all others have none, the Gini coefficient will be very nearly one).

Source: https://en.wikipedia.org/wiki/Gini_coefficient

**Example in R**:

If we want to get the 20, 40, 60, and 80th quantiles of the Gini index variable, we use again the `quantile`

function.

The `na.rm=TRUE`

parameter is necessary if we have empty values like in this case:

```
# We also can get multiple quantiles at once
p_custom = quantile(data_world_wide$gini_index, probs = c(0.2,
0.4, 0.6, 0.8), na.rm = TRUE)
p_custom
```

```
## 20% 40% 60% 80%
## 32 35 41 46
```

### 5.1.3 Indicating where most of the values are

In descriptive statistics, we want to describe the population in general terms. We can speak about ranges using two percentiles. Let’s take the percentiles 10 and 90th to describe 80% of the population.

*The poverty ranges from 0.075% to 54.4% in 80% of the countries*. (80% because we did 90th–10th, focusing on the middle of the population.)

If we consider the 80% as the majority of the population, then we could say: *“Normally (or in general terms), poverty goes from 0.07% to 54.4%”*. This is a semantical description.

We looked at 80% of the population, which seems a good number to describe where most of the cases are. We also could have used the 90% range (percentile 95th - 0.5th).

### 5.1.4 Visualizing quantiles

Plotting a histogram alongisde the places where each percentile is can help us understand the concept:

```
quantiles_var = quantile(data_world_wide$poverty_headcount_1.9,
c(0.25, 0.5, 0.75), na.rm = T)
df_p = data.frame(value = quantiles_var, quantile = c("25th",
"50th", "75th"))
library(ggplot2)
ggplot(data_world_wide, aes(poverty_headcount_1.9)) +
geom_histogram() + geom_vline(data = df_p, aes(xintercept = value,
colour = quantile), show.legend = TRUE, linetype = "dashed") +
theme_light()
```

If we sum all the gray bars before the 25th percentile, then it will be around the height of the gray bars sum after the 75th percentile.

In the last plot, the IQR appears between the first and the last dashed lines and contains 50% of the population.

### 5.1.5 Rank and top/bottom ‘*X%*’ concepts

The ranking concept is the same as the one seen in competitions. It allows us to answer *what is the country with the highest rate in ?*

We’ll use the `dense_rank`

function from the `ggplot2`

package. It assigns the position (rank) to each country, but we need them in reverse order; that is, we assign the `rank = 1`

to the highest value.

Now the variable will be: *Population living in slums is the proportion of the urban population living in slum households. A slum household is defined as a group of individuals living under the same roof and lacking one or more of the following conditions: access to improved water, access to improved sanitation, sufficient living area, and durability of housing.*

The question to answer: *What are the top six countries with the highest rates of people living in slums?*

```
# Creating rank variable
data_world_wide$rank_pop_living_slums = dense_rank(-data_world_wide$pop_living_slums)
# Ordering data by rank
data_world_wide = arrange(data_world_wide,
rank_pop_living_slums)
# Printing the first six
# results
select(data_world_wide, Country.Name,
rank_pop_living_slums) %>%
head(.)
```

```
## Country.Name rank_pop_living_slums
## 1 South Sudan 1
## 2 Central African Republic 2
## 3 Sudan 3
## 4 Chad 4
## 5 Sao Tome and Principe 5
## 6 Guinea-Bissau 6
```

We can also ask: *In which position is Ecuador?*

```
filter(data_world_wide, Country.Name == "Ecuador") %>%
select(rank_pop_living_slums)
```

```
## rank_pop_living_slums
## 1 57
```

##### 5.1.5.0.1 Top and bottom ‘X%’ concepts

Other questions that we may be interested in answered: *What is the value for which I get the top 10% of lowest values?*

The 10th percentile is the answer:

```
quantile(data_world_wide$pop_living_slums, probs = 0.1,
na.rm = T)
```

```
## 10%
## 12
```

Working on the opposite: *What is the value for which I get the bottom 10% of highest values?*

The 90th percentile is the answer, `quantile(data_world_wide$pop_living_slums, probs=.9, na.rm = T)`

. We can filter all the cases above this value.

### 5.1.6 Percentile in scoring data

There are two chapters that use this concept:

The basic idea is to develop a predictive model that predicts a binary variable (`yes`

/`no`

). Suppose we need to score new cases, for example, to use in a marketing campaign. The question to answer is:

*What is the score value to suggest to sales people in order to capture 50% of potential new sales?* The answer comes from a combination of percentile analysis on the scoring value plus the cumulative analysis of the current target.

#### 5.1.6.1 Case study: distribution of wealth

The distribution of wealth is similar to the Gini index and is focused on inequality. It measures owner assets (which is different from income), making the comparison across countries more even to what people can acquire according to the place where they live. For a better definition, please go to the *Wikipedia* article and *Global Wealth Report 2013*. Refs. (Wikipedia 2017a) and (Suisse 2013) respectevely.

Quoting *Wikipedia* (Ref. (Wikipedia 2017a)):

half of the world’s wealth belongs to the top 1% of the population;

the top 10% of adults hold 85% while the bottom 90% hold the remaining 15% of the world’s total wealth; and

the top 30% of adults hold 97% of the total wealth.

Just as we did before, from the third sentence we can state that: *“3% of total wealth is distributed to 70% of adults.”*

The metrics `top 10%`

and `top 30%`

are the quantiles `0.1`

and `0.3.`

Wealth is the numeric variable.

Amatriain, Xavier. 2015. “In Machine Learning, What Is Better: More Data or Better Algorithms.” http://www.kdnuggets.com/2015/06/machine-learning-more-data-better-algorithms.html.

Caban, Jesus J., Ulas Bagci, Alem Mehari, Shoaib Alam, Joseph R. Fontana, Gregory J. Kato, and Daniel J. Mollura. 2012. “Characterizing Non-Linear Dependencies Among Pairs of Clinical Variables and Imaging Data.” *Conf Proc IEEE Eng Med Biol Soc* 2012 (August): 2700–2703. doi:10.1109/EMBC.2012.6346521.

Fernandez-Delgado, Manuel. 2014. “Do We Need Hundreds of Classifiers to Solve Real World Classification Problems?” http://jmlr.csail.mit.edu/papers/volume15/delgado14a/delgado14a.pdf.

Fortmann, Scott. 2012. “Understanding the Bias-Variance Tradeoff.” http://scott.fortmann-roe.com/docs/BiasVariance.html.

Handbook, Engineering Statistics. 2013. “Measures of Skewness and Kurtosis.” http://www.itl.nist.gov/div898/handbook/eda/section3/eda35b.htm.

Hyndman, Rob J. 2010. “Why Every Statistician Should Know About Cross-Validation?” https://robjhyndman.com/hyndsight/crossvalidation/.

———. 2017. “ARIMA Modelling in R.” https://www.otexts.org/fpp/8/.

Izbicki, Mike. 2011. “Converting Images into Time Series for Data Mining.” https://izbicki.me/blog/converting-images-into-time-series-for-data-mining.html.

Kuhn, Max. 2017. “Recursive Feature Elimination in R Package Caret.” https://topepo.github.io/caret/recursive-feature-elimination.html.

McNeese, Bill. 2016. “Are the Skewness and Kurtosis Useful Statistics?” https://www.spcforexcel.com/knowledge/basic-statistics/are-skewness-and-kurtosis-useful-statistics.

Raschka, Sebastian. 2017. “Machine Learning Faq.” http://sebastianraschka.com/faq/docs/evaluate-a-model.html.

Reshef, David N., Yakir A. Reshef, Hilary K. Finucane, Sharon R. Grossman, Gilean McVean, Peter J. Turnbaugh, Eric S. Lander, Michael Mitzenmacher, and Pardis C. Sabeti. 2011. “Detecting Novel Associations in Large Data Sets.” *Science* 334 (6062): 1518–24. doi:10.1126/science.1205438.

stackoverflow.com. 2017. “What Is Entropy and Information Gain?” http://stackoverflow.com/questions/1859554/what-is-entropy-and-information-gain.

stats.stackexchange.com. 2015. “Gradient Boosting Machine Vs Random Forest.” https://stats.stackexchange.com/questions/173390/gradient-boosting-tree-vs-random-forest.

———. 2017a. “How to Interpret Mean Decrease in Accuracy and Mean Decrease Gini in Random Forest Models.” http://stats.stackexchange.com/questions/197827/how-to-interpret-mean-decrease-in-accuracy-and-mean-decrease-gini-in-random-fore.

———. 2017b. “Percentile Vs Quantile Vs Quartile.” https://stats.stackexchange.com/questions/156778/percentile-vs-quantile-vs-quartile.

Suisse, Credit. 2013. “Global Wealth Report 2013.” https://publications.credit-suisse.com/tasks/render/file/?fileID=BCDB1364-A105-0560-1332EC9100FF5C83.

Wikipedia. 2017a. “Distribution of Wealth.” https://en.wikipedia.org/wiki/Distribution_of_wealth.

———. 2017b. “Monotonic Function.” https://en.wikipedia.org/wiki/Monotonic_function.

———. 2017c. “Occam’s Razor.” https://en.wikipedia.org/wiki/Occam's_razor#Probability_theory_and_statistics.

———. 2017d. “White Noise - Time Series Analysis and Regression.” https://en.wikipedia.org/wiki/White_noise.

### References

stats.stackexchange.com. 2015. “Gradient Boosting Machine Vs Random Forest.” https://stats.stackexchange.com/questions/173390/gradient-boosting-tree-vs-random-forest.

2017a. “How to Interpret Mean Decrease in Accuracy and Mean Decrease Gini in Random Forest Models.” http://stats.stackexchange.com/questions/197827/how-to-interpret-mean-decrease-in-accuracy-and-mean-decrease-gini-in-random-fore. 2017b. “Percentile Vs Quantile Vs Quartile.” https://stats.stackexchange.com/questions/156778/percentile-vs-quantile-vs-quartile.Wikipedia. 2017a. “Distribution of Wealth.” https://en.wikipedia.org/wiki/Distribution_of_wealth.

Suisse, Credit. 2013. “Global Wealth Report 2013.” https://publications.credit-suisse.com/tasks/render/file/?fileID=BCDB1364-A105-0560-1332EC9100FF5C83.