It also keeps the fingerprints so if you get a different cert on a later visit, it will give you a warning again.
To simplify adding an exception on the first visit you might want to consider this: http://kb.mozillazine.org/Browser.xul.error_pages.expert_bad_cert