I do agree with you that observational studies are a joke - the kicker, from what I've understood through the years, is getting the funding to do a true study with some of the metrics you pointed out...I like the ones you suggest, but wouldn't stop at a typical cholesterol panel...adding ApoB and Lp(a) as well as fasting insulin.
I know Shawn Baker has tried to get funding to run a study, but no pharma is going to fund something that shows that nutritional intervention would eliminate *most* needs for their pharma solutions.
Healthy/Unhealthy user bias is a real problem for these types of studies, too...yet the results of those have been widely published (like smokers/drinkers eating red meat and dying early, and yoga/exercising/no risky lifestyle vegans live longer.