Not sure that formalizing the market would add much to what we already have. Prediction markets are useful for things where its not already possible to "put your money where your mouth is". But we already have that in OSS via developers investing their time, users choosing which software to run, etc.
Discussion
Can I put my money on the success of all the open-source projects I think are going to succeed? Can I bet on their failure?
That would require some sort of hard and tamper proof metric to measure success or failure.
Is it as straightforward as placing a bet? No. But there are still many more/better ways to do it than, say, going long or short the outcome of an election. Prediction markets shine in contexts like elections because there aren't other good skin-in-the-game metrics. But OSS has things like developer activity, user adoption, $ donations, etc.
Investing your own time in an open-source project is not very different from voting in an election and promoting your candidate and so on. Both kinds of actions give us no clue to whether the project or candidate is going to be successful or not.