Andrew, @ad5ou is absolutely correct, the use of keys rather than passwords does make this kind of job easier. Indeed I use ssh keys all the time for this kind of backup script, especially when it is run unattended, from a cron job, for example.
My gut feeling right now is that you are...