I'm not really sure of the actual mechanism, but on Firefox with a fully updated block list the delay doesn't seem to happen for me. Whereas I could never quite get rid of it on Chrome. This was a while ago, though, when they first introduced it.
I use uBlock Origin with Firefox on Linux, and it seems like that delay happens maybe on 30% of the YouTube videos for me, with no rhyme or reason to which ones. And reloading the same video multiple times show consistent behavior if it loads fast/slow, not sure what's going on.
