English readers read from left to right. It makes sense to have the question mark come first since it is a question. That information should come as soon as possible.

You went to the fair!? > Lame. I figure out that it's actually a question and not a statement after I've been told that it's a statement. It's a rug pull; a grammatical shitcoin.

You went to the fair?! > It's presented first as a question (because it is) and then the question is exclaimed. This makes more sense if the goal is to ask a question with excited emphasis.

Neither is correct in my opinion, but one clearly makes more informal sense to me.

Peace.