If it makes you feel better, I have 3-4 (film) SLR bodies plus the odd compact in my closet I'll never use again, but can't seem to part with.
I don't have a vote, but if I did I'd say take both flavors of cameras. I agree, the worst photo is the one you never took, having something small with you at all times is a must (I have a pocket-sized Pentax waterproof for that role).
But having something substantial for that special shot is nice too.