We have a few extensions that are set to not be visible in the directory, and we wanted the same sort of filter to apply to "phonebook" or "contacts" downloads (via provisioning). Unfortunately, I haven't found a built-in way to do this, so I accomplished the goal by adding the following line of code after line 615 in <fusionpbx dir>/app/provision/resources/classes/provision.php (so the original line 616 becomes 617)
$sql .= "and directory_visible = 'true' ";
$sql .= "and directory_visible = 'true' ";
