I specialized in reverse-engineering binaries for a few years (it's shockingly common for companies to lose the source code for their own products), and I second this.
Reverse-engineering is difficult and time-consuming. Reverse-engineering something as large and complex as a browser would be a huge project all by itself.
What about getting ahold of the binary or whatever raw version of Windows you can get (C++, assembly?), and look at the diff from before this change and after. Would that drastically narrow down what you should look at? Moreover, if there is a literal CONSTANT that says "Chrome", is it possible that it's not obfuscated and sitting in the code in plain sight?
So the opening of the dialog happens when user association hash verification fails if I recall correctly.
Shell32.dll has the code for it under QueryUserAssocAndVerifyHash I believe. Perhaps they've interfered with the hash interpretation for Chrome or Chrome is hooking into it in a different way - Firefox directly computes and writes a new hash to the registry.
Raw diffs of binaries may or may not help, depending, but it's not likely to help as much as you might think. If you can get the source code (C++, assembly, whatever), then there's no need to reverse engineer anything.
I would think it would be the Windows equivalent of `grep -a Chrome -R C:/Windows`, right? Then, when it finds the file, just hexdump and/or disassemble it to learn more. Maybe they're scrambling the executable somehow?
I suspect that it’s worse than that, it’s somewhere in windows. So you’d probably have to start by hunting for a task then find the binaries associated and hope it’s not too high up in the OS mess.
Reverse-engineering is difficult and time-consuming. Reverse-engineering something as large and complex as a browser would be a huge project all by itself.