You might want to keep a contact record in the database, but be sure not to mail them anything else.
Open the contact and click Edit Contact.
At the bottom of the screen, place a checkmark in the "Inactive" box.
To keep track of Inactive Contacts, use the flag called "Status: Deactivated" and assign it to every person you checkmark as Inactive.
Here is a quick video on How To Exclude a Contact from Group Mailings: