The outside is directly connected to the inside walls. The wall will try to reach equilibrium bringing the cold _inside_ your house and your heating _outside_ your house (+ condensation problems).
Quality homes in cold climates go to great lengths to thermally isolate inside and outside (3x pane windows, thermal breaks, special membranes, etc.).
There are many other problems too.
2/2