The only way I could get it to work, was to go to DialPlan Manager and select the entry for the IVR in question.
I then had to click the XML button and add the following line:
For anyone else, I just tried this. It seems if you use a : (IE Test: ) as a prefix, then the prefix is not included. If I just did Test then the caller ID shows up Test#Caller ID info.