I like this question, as I've thought about it a couple times. Ultimately, my answer would be no because of three reasons:
1) I like the variety. I think it would be less fulfilling to just look at one card vs. a variety of different players from different eras.
2) With prices the way they are right now, I could probably only trade my run of centered, mid-grade T206 Cobbs and Goudey Ruth's for a slightly higher grade, uncentered Rose Rookie. I don't love the exchange rate right now.
3) I learned on this site that you should only pursue an investment card if you're prepared to buy 50 of them to protect the prices, so that means I'd have to trade everything for a ton of Carlton Fisk Rookie PSA 7s.