Record security is set at the individual item level, so if you want to restrict a project member from viewing a document (or task, or issue, etc.) you can do so upon creation of the item, or at any time after it has been created.

Under the Permissions and Notifications section of the create/edit item page, you can select which users you want to restrict by clicking the Browse Member Directory button to select the desired users, then deselect the "View" option next to their name before saving.  That's it!

Users who have permission to view the record will see a little restricted icon in the list and folder views (looks like a key), and restricted users won't see the record at all.

Final thing to note here, there are two types of users who cannot be restricted from viewing a record: the item creator, and any project member with the "Cannot be restricted" permission, typically just the Project Manager role, although this can be added to other project-level roles as well.

I've had the same issue and I resolved it by using Project Member Workgroups.

We had a list of users that we consistently wanted to omit from having access to certain documents, so we put their names in one workgroup (let's call it X) and the remaining project members that should have access in a different project workgroup (let's call it Y). We then instituted a process where when we create a document item, users always uncheck the view box corresponding to "Everyone Else (in this ... Project)", add workgroup Y to the list, and check the view box for Workgroup Y. We only add and check the view box for Workgroup X where necessary.

This also comes in handy when adding users in the future, since any users added to Workgroup Y later will automatically get access to documents where the workgroup has been given access previously.