Leave the pauses in, if anything only edit out awkward mouth noises and replace with room tone.

Other mixing considerations:

- De-ess both guests. Nothing will make me turn off a podcast faster than harsh sybilance

- Match overall loudness of both host and guest using EQ/compression.

- Consider editing out or automating background noise with gates.