Pixel averages and Porter-Duff alpha blending will always give fuzzy results. You need sub-pixel geometric info to do a more accurate depth blend.
The classic reference is the A-Buffer by Loren Carpenter (SIGGRAPH 1984).
One interesting possibility is to go from a binary matte or averaged anti-aliased layer to subpixel geometry by generating a contour (e.g. Marching Squares).
Then the coarse layers (e.g. green screen) can be incorporated into a high-resolution A-Buffer rendering pipeline with 3D CGI elements.
This research was sponsored by Lucasfilm because at the time, alpha compositing of bitmap images wasn't really a concern for desktop computers. CPU, memory, and display hardware wasn't there to do anything useful with alpha (other than 1-bit masks used for game characters and GUI icons).
OTOH Lucasfilm (and other film visual effects companies) had a long history of using mattes, the physical photography equivalents of alpha compositing. When you see a group of Stormtroopers marching inside a Death Star hangar in 1977 "Star Wars", the hangar is a matte painting. They were typically created by artists directly on a large glass that was carefully placed in front of the camera to create the illusion.
Porter and Duff came up with a compositing model that's appealingly simple: it's a single formula, three variables, and you get 12 compositing operators by simply enumerating the combinations.
You can still find these Porter-Duff operators pretty much anywhere that 2D graphics are processed. They're in HTML Canvas, PDF, Apple CoreGraphics...
But, as neatly mathematical as these operators are, most of them aren't really practical for rendering actual scenes consisting of graphics layers. They're not terribly intuitive for designers, and they don't even attempt to simulate the effects of combining light.
So when Photoshop was created, its designers needed a set of blending operations (initially for drawing tools, and soon after for combining layers, which were introduced in PS version 3.0). Exposing the Porter-Duff operators to designers in the Photoshop GUI wouldn't have been very useful.
Instead they came up with a set of blending modes that were loosely derived from operations that you could do on physical photography negatives, since that was the inspiration for the application. (The PS blending modes still retain names like "dodge" and "burn" which refer to darkroom techniques that no designer aged under 50 probably remembers.)
These PS blending modes are conceptually pretty much the opposite of the Porter-Duff modes: they're not mathematically rigorous, but just a collection of algorithms that could be executed in reasonable time on a 68k processor back in the early 1990s. These original limitations are visible in the design of the algorithms: for example, they don't even try to account for gamma correction in the source images, which means that the algorithms do a rather poor job of simulating the behavior of actual light. (Photoshop has later added "linear" versions of some blending modes to address this.)
Still, they are the closest thing we have to a standard for blending modes. Many 2D graphics APIs that originally only supported Porter-Duff have added support for PS modes too, including the aforementioned PDF and CoreGraphics.
I mean, Porter-Duff only specifies coverage, and the blending modes affect the actual colors. Porter-Duff is more like the mask tool than it is the blend mode.
Thank you, I got really confused when the linked page started to discuss Photoshop blending modes and I had never considered they could be mapped directly to Porter-Duff ops.
As a graphics programmer I always want to point out that it's good to remember NONE of these are scientifically foundational. These are more or less just hacks that looks nice - pure heuristics that is. They are really pragmatic heuristics that produce nice outcomes.
I mean add and multiply pretty much work as expected (and I would say except for overlay, color, saturation and lumiosity they are the only ones you will ever need).
I would add to that list color dodge. Used in the right way, it produces results that mimic natural highlights. A highlight will impact all values except black (I.e. deep shadow) which it will ignore.
Good to hear about porter/duff. Was news to me. The furthest back I go is 'Photoshop Channel Chops' Biedny, D., Monroy, B., & Moody, N. (1996). New Riders Publishing.
If you are interested in a more rigorous collection of blending and compositing operations, the W3C "Compositing and Blending Level 1" document is a wonderful resource>
Yes, that’s true if you have samples that represent actual measured luminosity (a.k.a. linear light). But most digital images are not that. Instead they are gamma-corrected to make better use of the available encoding range.
Gamma correction is effectively a 1/2.2 power function applied to each RGB component (normalized and clamped to a 0-1 range), so more bits are used to encode darker values. If you were to store raw linear values in a typical image format that uses 8 bits per component (i.e. 24 bits for RGB), you’d get very visible banding in dark areas of the image.
So this is the problem for Photoshop’s blending modes. They want to simulate how light works, but they operate on gamma-corrected images so you can’t just do the simple linear math directly. And converting to a higher-precision linear light representation would have been expensive on 1990s CPUs. Instead the PS blending mode algorithms are more like approximated hacks to do visually interesting combinations of gamma-corrected layers “on the cheap.”
As for Porter-Duff, they don’t even pretend to simulate light in any way. The article explains this pretty well below the example images of the twelve operations:
“Porter/Duff is not a way to blend the source and destination shapes. It is way to overlay, combine and trim them as if they were pieces of cardboard. The only place where source and destination pixels are actually blended is along the antialiased edges.“
The classic reference is the A-Buffer by Loren Carpenter (SIGGRAPH 1984).
One interesting possibility is to go from a binary matte or averaged anti-aliased layer to subpixel geometry by generating a contour (e.g. Marching Squares).
Then the coarse layers (e.g. green screen) can be incorporated into a high-resolution A-Buffer rendering pipeline with 3D CGI elements.