It depends on what you are doing, and how your system behaves.

If you size your RAM and swap right, you get no service degradation, but still get away with using less RAM.

But when I was at Google (about a decade ago), they followed exactly the philosophy you were outlining and disabled swap.