Logistic Regression

class: title-slide

<br>
<br>
.right-panel[ 
<br>

# Logistic Regression

### Dr. Babak Shahbaba

]

---

### Introduction

- For linear regression models, the response variable, `$Y$`, is assumed to be a real-valued continuous random variable.

- Now consider situations where the response variable is a binary random variable (e.g., disease status).

- For such problems, it is common to use **logistic regression** instead:
`$$\begin{eqnarray*}
\log \Big(\frac{\hat{p}}{1- \hat{p}} \Big) & = & a + b_{1}x_1 + \ldots + b_{q}x_{q}
\end{eqnarray*}$$`

---

### Logistic regression

- Note that binary variable `$p = P(Y=1|X)$`; that is, `$p$` is the probability of the outcome of interest (denoted as 1) given the explanatory variables.

- The term `$\Big(\frac{\hat{p}}{1- \hat{p}} \Big)$` is called the **odds** of `$Y=1$`.

- The term `$\log \Big(\frac{\hat{p}}{1- \hat{p}} \Big)$`, i.e.,  log of odds, is called the **logit** function.

- Although `$p$` is a real number between 0 and 1, its logit transformation can be any real number between `$-\infty$` to `$+\infty$`.

---

### Logistic regression

- We can exponentiate both sides,
`$$\begin{eqnarray*}
\frac{\hat{p}}{1- \hat{p}}  & = & \exp(a + b_{1}x_1 + \ldots + b_{q}x_{q})
\end{eqnarray*}$$`

- and find `$\hat{p}$` using the **logistic function**,
`$$\begin{eqnarray*}
\hat{p} & = & \frac{\exp(a + b_{1}x_1 + \ldots + b_{q}x_{q})}{1 + \exp(a + b_{1}x_1 + \ldots + b_{q}x_{q})}
\end{eqnarray*}$$`

---

### Logistic Regression with One Binary Predictor

- As an example, we use the {birthwt} data set to model the relationship between having low birthweight babies (a binary variable), `$Y$`, and smoking during pregnancy, `$X$`.

- The binary variable {low} identifies low birthweight babies ({low} = 1 for low birthweight babies, and 0 otherwise).

- The binary variable {smoke} identifies mothers who were smoking during pregnancy ({smoke}=1 for smoking during pregnancy, and 0 otherwise).

---

### Generalized linear model (glm) in R

- We can use the `glm()` function in R to fit a regression model

```r
library(MASS)
data("birthwt")
birthwt$low <- factor(birthwt$low)
birthwt$smoke <- factor(birthwt$smoke)
fit <- glm(low ~ smoke, family = 'binomial', data = birthwt)
confint(fit)
summary(fit)
```

---

### Estimation

- For the above example, the estimated values of the intercept `$\alpha$` and the regression coefficient `$\beta$` are `$a=-1.09$` and `$b=0.70$` respectively.

- Therefore, 
`$$\begin{eqnarray*}
\frac{\hat{p}}{1- \hat{p}} & = & \exp(-1.09 + 0.70x)
\end{eqnarray*}$$`

- Here, `$\hat{p}$` is the estimated probability of having a low birthweight baby for a given `$x$`.

- The left-hand side of the above equation is the estimated odds of having a low birthweight baby.

---

### Estimation

- For non-smoking mother, `$x=0$`, the odds of having low birthweight baby is 
`$$\begin{eqnarray*}
\frac{\hat{p}_{0}}{1- \hat{p}_{0}} & = & \exp(-1.09) \\
& = & 0.34
\end{eqnarray*}$$`

- That is, the exponential of the intercept is the odds when `$x=0$`, which is sometimes referred to as the **baseline odds**.

---

### Estimation

- For mothers who smoke during pregnancy, `$x=1$`, and
`$$\begin{eqnarray*}
\frac{\hat{p}_{1}}{1- \hat{p}_{1}} & = &  \exp(-1.09 + 0.7)\\
& = &  \exp(-1.09) \exp(0.7)
\end{eqnarray*}$$`

- As we can see, corresponding to one unit increase in `$x$` from `$x=0$` (non-smoking) to `$x=1$` (smoking), the odds multiplicatively increases by the exponential of the regression coefficient.

---

### Interpretation

- Note that
`$$\begin{eqnarray*}
\frac{\frac{\hat{p}_{1}}{1- \hat{p}_{1}}}{\frac{\hat{p}_{0}}{1- \hat{p}_{0}}} & = &  \frac{\exp(-1.09) \exp(0.7)}{\exp(-1.09)} =  \exp(0.7)
\end{eqnarray*}$$`

- We can interpret the exponential of the regression coefficient as the odds ratio of having low birthweight babies for smoking mothers compared to non-smoking mothers.

- Here, the estimated odds ratio is `$\exp(0.7) = 2.01$` so the odds of having a low birthweight baby almost doubles for smoking mothers compared to non-smoking mothers.

---

### Interpretation

- In general,

- if `$b>0$`, then `$\exp(b) > 1$` so the odds increases as `$X$` increases;
  - if `$b<0$`, then `$0 < \exp(b) < 1$` so the odds decreases as `$X$` increases;
  - if `$b=0$`, the odds ratio is 1 so the odds does not change with `$X$` according to the assumed model.

---

### Inference

- Similar to linear regression, we use the functions `confint()` and `summary()` to obtian the confidence interval and p-value for each coefficient.

```r
fit <- glm(low ~ smoke, family = 'binomial', data = birthwt)
confint(fit)
summary(fit)
```

---

### Prediction

- We can use logistic regression models for predicting the unknown values of the response variable `$Y$` given the value of the predictor value `$X$`.

`$$\begin{eqnarray*}
\hat{p} & = & \frac{\exp(a + bx)}{1 + \exp(a + bx)}
\end{eqnarray*}$$`

- For the above example,
`$$\begin{eqnarray*}
\hat{p} & = & \frac{\exp(-1.09 + 0.70x)}{1 + \exp(-1.09 + 0.70x)}
\end{eqnarray*}$$`

---

### Prediction

- Therefore, the estimated probability of having a low birthweight baby for non-smoking mothers, `$x=0$`, is
`$$\begin{eqnarray*}
\hat{p} & = & \frac{\exp(-1.09)}{1 + \exp(-1.09)} = 0.25
\end{eqnarray*}$$`

- This probability increases for mothers who smoke during pregnancy, `$x=1$`,
`$$\begin{eqnarray*}
\hat{p} & = & \frac{\exp(-1.09 + 0.7)}{1 + \exp(-1.09 + 0.7)} = 0.40
\end{eqnarray*}$$`

- That is, the risk of having a low birthweight baby increases by 60% if a mother smokes during her pregnancy.

---