Inactive users should not receive notifications. You should only be seeing active users and External Resources in the User Selector dialog. The "Remove User/External Resource" option will deactivate a user account. External Resources are essentially just users that can't log in to the system.
External Resources may be configured to receive notification, exactly like a regular user, so there are technically four categories a user account may exist in: Active user, inactive user, active External Resource and inactive External Resource. Any active account will appear in the User Selector window for notifications.