I went no glasses ever for just over a year. When I went to the eye doc I found out that my eyes started to develop pinguecula. It's not likely to be harmful to my vision, but is unappealing and has the potential to cause problems.
I pivoted to a middle of the road approach. The polarized shades are on in specific sitarjowns, such as, on the water (beach, lake, river), skiing or biking (mainly physical protection). Otherwise I go no glasses and/or hat for day to day life.
As with most things the best solution is likely somewhere in the middle.