Thoughts on Joseph being a widower with children, prior to Mary?

Saint James would have been a half brother to Jesus in that case.