I think basically it’s not a very effective/efficient market, you have to place your ‘bid’ (decide a fee) based on what the fees in recent blocks have been or what the fees of competing transactions in the mempool are. But the bids are taken after you’ve decided them, so there’s a degree of uncertainty and inefficiency - other higher-paying transactions can be broadcast before the next block that will delay your confirmation.
You’re effectively guessing at the right fee rate. RBF (replacing a transaction with another that pays a higher fee) helps with this a little.