I agree with your assessment. Until and unless twitter provides a reasonable API you shouldn't try to display the cards.
What about scraping the content of the tweet and rendering it sort of like a quoted post? (Maybe that's what you already do - I don't remember and couldn't find any examples of tweet quoting posts in my current timeline).