In presence of foreign key constraints for example, DELETE+INSERT might break/fail when UPDATE+INSERT will not. Also, the performance characteristics might be different.
I agree. DELETING when you want to do an UPDATE or INSERT just seems wrong on all levels. DELETING is dangerous and I tend to prefer slightly slower performance of UPSERT with UPDATE/INSERT than involving a delete in the operation.