YMMV whether this will fly in your company culture, but I titled mine "Focus time, please ask before scheduling".
And when people inevitably didn't ask, I'd just decline unless I especially wanted to attend. I find myself getting invited to meetings sometimes just because the organizer wants to be inclusive and make sure everyone is looped in who might want to be, and I figure that's what's going on if they added me without asking.
If it's really important for me to be there, they'll see my time block and ask me.