Contents

• Covariance and correlation coefficient
  1. Covariance
  2. Correlation analysis

Covariance and correlation coefficient

Covariance

If it is a continuous variable, you cannot create a cross tabulation that is subject to χ² test. Instead, you can apply correlation analysis. Correlation analysis is an analysis method that measures the relationship between two or more continuous variables.

Use a scatterplot to visually represent the correlation between the two variables. Figure 1(a) shows a clear direct proportion between y₁ and y₂. On the other hand, (c) shows an inverse relationship, but (b) cannot specify any proportional relationship between y₁ and y₂. These relationships can be quantitatively represented using statistics called correlation coefficients, which relate to covariance of two variables and their respective standard deviations.

Figure 1. (a) normal relationship (b) unrelated relationship (c) inverse relationship of the two variables.

Figure 1(a) measures each deviation of y₁-μ₁, y₂-μ₂ between the means of each variable and any point $y_1 and y_2$. In this case, an increase of $y_2$ with an increase of $y_1$ is observed, so the product (y₁-μ₁)(y₂-μ₂) of the two deviations will increase more than each and be positive. If the same process is applied to figure (c), the product of the two deviations will be negative. In Figure (b), you cannot specify the product sign of the two deviations. As a result,(y₁-μ₁)(y₂-μ₂) is an indicator of linear dependence of two variables y₁ and y₂ and the expected value of this deviation product E[(y₁-μ₁)(y₁-μ₁)] is called covariance (Equation 1).

$$\begin{align}\tag{1} \text{Cov}(Y_1, Y_2)&=E[(Y_1-\mu_1)(Y_2-\mu_2)]\\ &=E(Y_1Y_2-Y_1\mu_2-\mu_1 Y_2+\mu_1 \mu_2)\\&= E(Y_1Y_2)-E(Y_1)\mu_2-\mu_1E(Y_2)+\mu_1 \mu_2\\&=E(Y_1Y_2)-\mu_1 \mu_2\\\because\; E(Y_1)=\mu_1, & E(Y_2)=\mu_2\end{align}$$

As the absolute value of covariance between two variables increases, linear dependence increases, positive covariance means direct proposition, and negative value means inverse relationship. If the covariance is zero, there is no linear dependence between the two variables. However, using covariance as an absolute dependency scale is difficult because its value depends on the measurement scale. As a result, it is difficult to check whether the covariance is large or small at a glance. These problems can be solved by standardizing values and using the Pearson correlation coefficient (ρ), which is an amount related to covariance.(Equation 2)

$$\begin{equation}\tag{2} \begin{aligned}&\rho = \frac{\text{Cov}(Y_1, Y_2)}{\sigma_1 \sigma_2}\\ & -1 \le \rho \le 1\\ &\sigma_1, \sigma_2: \text{standard deviation of}\,Y_1, Y_2 \end{aligned} \end{equation}$$

The sign of the correlation coefficient is the same as the sign of covariance and is organized as follows:

Table 1. Correlation Coefficient

correlation coefficient	mean
ρ = 1	perfect direct relationship
0 < ρ < 1	direct relationship
ρ= 0	No correlation
-1< ρ <0	inverse relationship
ρ = -1	perfect inverse relationship

The lack of correlation between the two variables means covariance=0, as shown in Table 1. This means that the two variables are independent of each other. That is, if the two variables are independent, the following is established:

$$E(Y_1Y_2)=E(Y_1)E(Y_2)$$

This result is equal to μ₁, μ₂ so the covariance, which is the difference between the two, is zero.

Example 1)
  Determine covariance and correlation coefficients from data on daily change rates between Apple and Google's beginning and closing price.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import FinanceDataReader as fdr
st=pd.Timestamp(2020,1,1)
et=pd.Timestamp(2021, 11, 29)
apO=fdr.DataReader('AAPL',st, et)
goO=fdr.DataReader('GOOGL',st, et)
ap=(apO["Close"]-apO["Open"])/apO["Open"]*100
go=(goO["Close"]-goO["Open"])/goO["Open"]*100

The pd.concat() function was applied to combine the two materials to create a single object.

y=pd.concat([ap, go], axis=1)
y.columns=[i+'Change' for i in ['ap', 'go']]
y.head(3)

	apChange	goChange
Date
2020-01-02	1.390764	1.505488
2020-01-03	0.094225	1.001484
2020-01-06	2.042206	3.418171

Figure 1 shows the scatter plot for both materials.

plt.scatter(y.values[:,0], y.values[:,1])
plt.xlabel("ap(%)", size=13, weight='bold')
plt.ylabel("go(%)", size=13, weight='bold')
plt.show()

Figure 2. Distribution of the two stock data.
(Direct prorportion)

Calculate covariance from the mean of each column of the above data and the difference between each value. In the following code, the product between the columns of the object applies a function object.product(axis). This function returns the product of the corresponding values based on the specified axis.

mean=y.mean(axis=0)
mean

apChange    0.108257
    goChange    0.098101
    dtype: float64

cov=(y-mean).product(axis=1).mean()
print(f'covariance: {np.round(cov, 4)}')

covariance: 1.617

The covariance above was calculated by multiplying the value of the second column corresponding to the first column of the object. The matrix product can progress the calculation more efficiently.

$$\begin{align} &\begin{bmatrix}x_{1} & y_{1}\\x_2&y_2\\x_3&y_3 \end{bmatrix}\rightarrow\begin{bmatrix}x_{1} \cdot y_{1}\\x_2 \cdot y_2\\x_3 \cdot y_3 \end{bmatrix}\\ \\ &\begin{bmatrix}x_{1} & x_2& x_3 \\y_{1} & y_2&y_3 \end{bmatrix} \begin{bmatrix}x_{1} & y_{1}\\x_2&y_2\\x_3&y_3 \end{bmatrix} \\ &\rightarrow \begin{bmatrix} x_1x_1+x_2x_2+x_3x_3& x_1y_1+x_2y_2+x_3y_3\\ x_1y_1+x_2y_2+x_3y_3 & y_1y_1+y_2y_2+y_3y_3\end{bmatrix} \end{align}$$

The results of this operation are shown in the following expression, and this matrix is called the **Correlation Coefficient Matrix**. As shown in the following results, the diagonal elements of the matrix are the variance of each column (variables), and the non-diagonal elements represent the covariance between the two variables.

$$\begin{bmatrix} \text{Variance of row 1} & \text{Covariance of row 1 and 2} \\ \text{Covariance of row 1 and 2} & \text{Variance of row 2} \end{bmatrix}$$

The covariance matrix in this example is 2 × 2 dimension, so the above matrices must be adjusted appropriately. In other words, object y(342 × 2) dimension must adjust the dimension of the object by applying a transposed matrix as shown in Equation 3, in order for the matrix product result to be to be 2 × 2.

$$\begin{align}\tag{3} &\text{cov Matrix} = \frac{Y^T \cdot Y}{n}\\&Y^T: \text{transposed matrix of Y}\\& n: \text{sample size} \end{align}$$

y1=y-y.mean()
print(f'covariance Matrix: {np.around(np.dot(y1.T,y1)/len(y1), 3)}')

covariance Matrix: [[2.85  1.617]
     [1.617 2.013]]

y.cov(ddof=0)

	apChange	goChange
apChange	2.849691	1.616999
goChange	1.616999	2.013030

The covariance matrix can be calculated by applying the pandas object.cov(ddof)) function. The calculation of covariance matrices by matrix product is for the population. That is, for ddof=0. However, the data in this example are samples and the degree of freedom should be considered. That is, ddof=1

covMat=y.cov()
covMat

	apChange	goChange
apChange	2.855615	1.620361
goChange	1.620361	2.017215

The coefficient of correlation is the covariance divided by each standard deviation.

#standard deviation
ysd=y.std(axis=0, ddof=1)
ysd=ysd.values.reshape(2,1)
np.around(ysd, 4)

array([[1.6899],
           [1.4203]])

#Multiplication matrix of each standard deviation
ysdMat=np.dot(ysd, ysd.T)
np.around(ysdMat, 4)

array([[2.8556, 2.4001],
           [2.4001, 2.0172]])

creCoef=covMat/ysdMat
creCoef

	apChange	goChange
apChange	1.000000	0.675127
goChange	0.675127	1.000000

Apply the pandas ``object.corr(method='pearson')`` function to return the results directly from the raw data.

y.corr()

	apChange	goChange
apChange	1.000000	0.675127
goChange	0.675127	1.000000

The example above is for two materials, which show the covariance of each data as follows:

$$\begin{align} \text{Cov}(x,x)&=E[(X-E(X))(X-E(X))]\\&=\frac{\sum^n_{i=1}(x_i - \mu_x)^2}{n-1}\\&= \sigma_x^2\\ \text{Cov}(y,y)&=E[(Y-E(Y))(Y-E(Y))]\\&=\frac{\sum^n_{i=1}(y_i - \mu_y)^2}{n-1}\\&= \sigma_y^2\\ \text{Cov}(x,y)&=E[(Y-E(Y))(Y-E(Y))]\\&=\frac{\sum^n_{i=1}(x_i-\mu_x)(y_i - \mu_y)}{n-1}\\&= \sigma_{xy}\\ \end{align}$$

The above expressions can be visualized as shown in Figure 2, a scatter plots for ap, go.

plt.figure(figsize=(10, 7))
plt.subplots_adjust(wspace=0.4)
ax1=plt.subplot(2,3,1)
ax1.scatter(ap, ap)
ax1.set_xlabel('ap(%)', size=13, weight='bold')
ax1.set_ylabel('ap(%)', size=13, weight='bold')
ax1.text(-5, 5, '(a)', size=13, weight='bold')
ax2=plt.subplot(2,3,2)
ax2.scatter(go, go)
ax2.set_xlabel('go(%)', size=13, weight='bold')
ax2.set_ylabel('go(%)', size=13, weight='bold')
ax2.text(-5, 4, '(b)', size=13, weight='bold')
ax3=plt.subplot(2,3,3)
ax3.scatter(ap, go)
ax3.set_xlabel('ap(%)', size=13, weight='bold')
ax3.set_ylabel('go(%)', size=13, weight='bold')
ax3.text(-6, 4, '(c)', size=13, weight='bold')
plt.show()

Figure 3. Covariance of the same data (a) and (b) and covariance of two other data(c).

Correlation analysis

Correlation analysis is the analysis of relationships between two or more data, and the parameters of the analysis are the correlation coefficients. The null hypothesis of the analysis is ρ = 0. In other words, test that there is no correlation between the data being compared.

H0: ρ =0, H1: ρ ≠ 0

Because the distribution for the coefficient of correlation (r) is averaged 0 and the range is [-1, 1], the variance in the distribution can be expressed as 1- r². This probability variable follows a t distribution with standard error $\displaystyle \sqrt{\frac{1-r^2}{n-2}}$, degree of freedom n-2.

The test statistics standardizing the variables according to the characteristics of this distribution are shown in Equation 3.

$$\begin{align}\tag{3} t&= \frac{r-\rho_0}{\sqrt{\frac{1-r^2}{n-2}}}\\&=\frac{r}{\sqrt{\frac{1-r^2}{n-2}}} \end{align}$$

Example 2)
  Perform a correlation analysis between the above example ap (y₁) and go (y₂).

The correlation coefficient for both materials is r ≈ 0.70. It can be calculated using the np.corrcoef() function. This function returns the same result as the pd object.corr() applied above, but the arguments passed to the function must be entered separately from the data related to the calculation.

r=np.corrcoef(y.values[:,0], y.values[:,1])
r

array([[1., 0.67512745],
           [0.67512745, 1.]])

print(f'correlation coeff. Mat.:  {np.around(r,3) }')

 correlation coeff. Mat.:  [[1.    0.675]
     [0.675 1.   ]]

r12=r[0,1]
print(f'corr.coeff:{np.round(r12, 3)}')

corr.coeff:0.675

Calculates the test statistics and determines the confidence intervals from α = 0.05.

df=y.shape[0]-2
print(f'df: {df}')

df: 480

t=r12*np.sqrt(df/(1-r12**2))
print(f'statistics t: {round(t, 3)}')

statistics t: 20.051

from scipy import stats

ci=stats.t.interval(0.95, df)
print(f"Lower : {round(ci[0], 4)}, Upper : {round(ci[1], 4)}")

Lower : -1.9649, Upper : 1.9649

pVal=2*stats.t.sf(t, df)
print(f'p-value: {round(pVal, 4)}')

p-value: 0.0

The test statistic is located outside the confidence interval and is p-value 0 which is very low compared to the significance level. Therefore, the null hypothesis can be dismissed. In other words, you can conclude that the two groups are correlated. This analysis can be performed by scipy.stats.pearsonr(x, y).

corcoef, pval=stats.pearsonr(y.values[:,0], y.values[:,1])
print(f'corr.coef.: {round(corcoef, 3)},  p-value: {round(pval, 3)}')

corr.coef.: 0.675,  p-value: 0.0