Cfrtjryk

Say we have a study where we want to run a logistic regression on a group of people, and we want to find out whether one attribute of a person makes them more likely to be a smoker.

So we have smoker yes/no as the dependent variable and let's say we have weight and gender as independent variables.

From this information, I'm guessing weight and gender are introduce collinearity because men generally weigh more than women, and therefore are not independent variables. Is this correct?

If so, I'm confused because I thought a regression controls for the other variables, so it seeks to look if men are more likely to smoke, regardless of their weight - and whether overweight people are more likely to smoke, regardless of gender.

Or do I have it all wrong?

Thank you.

When it comes to collinearity, we have to distinguish exact collinearity from approximate collinearity.

The former occurs when one independent variable (IV) is an exact linear combination of other independent variables. Usually this is due to some sort of elementary mistake (like entering weight in pounds and also weight in kilograms) and is easily corrected.

The latter occurs when one IV is almost a linear combination of other IVs. Assessing when "almost" is close enough to be troublesome can be done through various means, the one I think best is condition indexes and proportion of variance explained; this has been discussed elsewhere and I won't add more here.

In your case, it is very unlikely that the relationship between sex and weight is close enough to cause trouble. In a general population, it's true that men, on average, weigh more than women. But there's lots of overlap: There are plenty of light men and heavy women. If (for some reason) your sample consisted of jockeys and football players, then it would cause problems.

The problems that collinearity causes include high variances for the parameter estimates and extreme sensitivity to changes in the data. In other words, we only have a very poor idea of what the parameter estimates are and a small change in the input data can result in a huge change in the parameter estimates.

The good news is that multicollinarity has absolutely no effect on the estimated coefficients $beta$ of the model. The bad new is that they inflate its standard error.

The formula for the standard error of a coefficient $beta_i$ for the independent variable $X_i$ is
$$ s.e.(beta_i)=sqrt{VIF_{X_i}frac{sigma_{varepsilon}^2}{nsigma_{X_i}^2}}$$
with $sigma_{varepsilon}$ being the standared error of regression (the smaller this value the larger the $R^2$ (goodness of fit) of the model), $VIF_{X_i}$ is a measure of multicolinarity (1 means no multicolinarity and the larger the more multicolinearity), $n$ is just the sample size and $sigma_{X_i}^2$ is the variance of $X_i$.

The key point is that if you add a new variable to the model, it increases $VIF_{X_i}$ and decreases $sigma_{varepsilon}$. If the variable matters and is not highly correlated with $X_i$, typically the standard error $s.e.(beta_i)$ goes down rather than up. But it is true, it can also go the other way around.

This may be too layme ...

In layman's terms "too many cooks (input series) can spoil the broth"