Hey Bill. I recently had the same dilemma. One of the reasons I got back into the hobby a few years ago was to try my hand at "prospecting". Same as you, I pay attention for dynasty fantasy leagues. I also love history and when I collected as a kid, I was always in awe at card shows when I'd see the pre-war pioneers and vintage cards of players my dad told stories about. So, I also decided to collect pre-war and vintage since I could somewhat afford those cards that I was awe-struck with growing up. Of course, that led me to this site and my interest dwindled in the modern stuff.
About six moths ago I sold most of my higher-end autos and refractors in one large lot. I do not regret it one bit! I re-invested most into my pre-war/vintage collection. Like you, I was right more often than not when prospecting. However, even when I was right, there are always cases like Matt Harvey and Jose Fernandez (and don't get me started on PED guys like your boy Braun
) that would make me sick to my stomach.
My recommendation would be to sell. I decided to keep all my modern Pirates/Steelers and Topps All-Star rookie cards (the ones with the trophy icons since I collect those from the 60's and up). Maybe you could do something similar - sell everything but your Brewers and then to keep up with the new releases, purchase just Brewer cards from time to time.
Good luck with your decision.