Historical accuracy/facts needed

How did 5% of the population in the South manage to keep a system in place that was detrimental to the overwhelming majority of southerners, both black and white. What specific factor/tool/policy do you think was the key to their maintaining control and imposing their will? Why didn’t the rest of the population use their numbers to impose a system more beneficial the majority? Is there any comparison you can make to the power brokers in the north at the time?