My biggest problem with Oblivion was the drab, dull, generic fantasy setting. The main thing that drew me into Morrowind, and by extension the Elder Scrolls series, was the setting. It was a game that was primarily about lore and exploration, and an Elder Scrolls game with a boring setting is like a Mario game with boring platforming. If you take that away, then even if you "improve" the core game mechanics, you're still left with a bland fantasy RPG that isn't good enough at any particular thing to stand out.
I mostly agree with this Youtube guy here.