According to a recent Gallup poll, baseball has not been America's favorite sport since 1972 when football first surpassed baseball in popularity. There is a significance here that I'll get back to in a moment. I think for a long time baseball had ridden it's own coat tails in regards to popularity even after football over took it. Baseball was far and away the most popular sport from the 1920s (prior to the 1920s it could easily be argued boxing was more popular) up to the early 1970s. Baseballs popularity began to slip when television coverage of sports began to become more wide spread, the 1970s.
Baseball is a sport that is tailor made for the radio! Football is more difficult to follow I think on radio, its better suited for a television viewing audience. I do think this had a serious effect on the decline of baseball. Baseball is not as captivating on the television in comparison to what it was on the radio from the 1920's to the early 1970s.
In response to how little "action" there actually is in a baseball game,
The average baseball game contains 14 minutes of real action study finds.