Then install a foam gasket under the cover plate. Air is probably leaking under the bottom plate of the walls. You may be able to seal this by caulking under baseboards. In the worst cases ...