Fun tidbit: The opposite is true now.
In order to prevent cross-domain tracking by observing cache hits, Browsers will not use a cached Google font across multiple domains. You gain nothing from using Google Fonts.
But, because it's an extra host, the browser will need to do the whole DNS/TCP/TLS dance to fetch the file, small as it may be, so you're getting additional latency compared to just downloading the file from the server it's already connected to.