I shouldn't feel bad...should I?

I feel bad. I shouldn't really, but I do. Maybe I shouldn't have been so flippant and dismissed his serious intentions as a joke. Maybe I should have asked to speak with him in person instead of saying "I wasn't even considering it" meaning "I wasn't even considering YOU" on the phone. I didn't want to come off so cold, but he started off about how it took me like ten days to get back to him..well, I had other things to do (yeah, maybe that's a lie, but there's some truth to it). Oh well, I guess I'll call him and straighten things out.
I don't know, but I think its a weakness of mine because I hate people thinking bad/negatively about me. I go out of my way to hold my anger or retort and let people have their way so as to keep unstable friendships..even going as far as...Well, that's for another post.