This is one of the abundance of reasons my Wife and I chose to pursue a 'homebirth'.
The most challenging aspect of going that route was, funny enough, the pressure from family to go the 'safer' direction and go to the hospital. It still affected us even after studying the topic over the last year in preparation. The numbers on safety and complications alone are insane, specifically in the US.
We unfortunately ended up going to the hospital at the very end of the labor because of an unforseen complication. The experience between the two environments clarified for the both of us that we'll never even contemplate a hospital birth at the very least.