We all have a player at the table who does this. They try to use their charisma to unlock doors, their wisdom to punch things--and more. And to a certain extent, 5e allows it, with Bard X + Hexblade 3 builds opening the gate wide open for two-weapon-fighting builds that exclusively use charisma.
And the solution to this as a DM is to just let it happen, honestly. You want to swim with dex? Fine, whatever. That's a blurry line anyway. Can you add your investigate to this check to sneak up on a creature? If you say no, you'll get an explanation for why it makes perfect sense in this context. So say no first, then let them garnish the world with the beautiful nonsense that fuels their characters -- and then sigh and reluctantly allow it.
Those one-stat wonder builds are going to have to make a saving throw eventually, and that it's only broken if you let it be.