Most add-ons that can be attached to the mini-map have a setting where you can disable the mini-map button. It's tedious, but if you go into each of the add-ons and disable them you can achieve your desired result. _________________
For each addon on the minimap you'll have to do like Kitria said and go into their options and hide the button. For the default blizzard buttons I use simpleminimap to remove the ones I don't need. _________________