This is a perfectly legitimate reason to not want to be with someone. It is laughable that he thinks he has any right to get mad when you spend your money as you please. How absolutely hypocritical of him that he doesn't want you to go shopping so that you can lend him money. And for what, exactly?
You're not a bank. You do not have any obligation to lend him money. (In fact, an actual bank would not lend him money if he has a poor habit of taking too long to pay back the loan.) I'm afraid he is taking advantage of your kindness, and you have every right to put your foot down.