It is not natively intuitive, but yes, you can. Edit the fax server and select the user(s) in the User List "Assign the users that can manage this fax extension.". Also make sure the users' group does NOT have the fax_extension_view_domain permission.
User A has access to only Fax Server A
User B to Fax Server B
User C to Fax Servers A, B, and C.