Not sure what you are using but on Windows I can set my "default" but each application can still individually select which input they want.

IE an untrusted process running should be able to pick the inbuilt mic.

Maybe linux has permissions around device inputs that can be used to stop this but if a person is worried their untrusted laptop might be listening to them then I suspect such a permission layer is already breached.