The only way to avoid a member (contact) from adding or connecting to another is for the contact itself to be set to Private.

This will hide the member from the listing (Members channel) so that its not visible. You cannot hide the Add nor the Connet button for members.