On 53 minutes from the original video, he shows how exact is the quotation of an LLM based on the text it was learning from. I wonder how did the bigtech convince the courts that this is not copyright violation (especially when ChatGPT was quoting some GPL code). I can imagine that the same thing would happen opposite, if I trained a model to draw a disney character, and my ass would be sued in a fraction of a second.
Note that he's inferring from a base model there, which are fairly capable of regurgitating their (highly-weighted) inputs since they do nothing but predict pre-training tokens. For instruct services like ChatGPT, if they regurgitate something I'd think it would more likely be their fine-tuning data, which is usually owned by the provider (and also kept secret).
what I mean is if we can describe an LLM as a lossy compression (which are words spoken by Andrej), we could define what was done during inferring as uncompressing the compressed data, and at this moment shit would hit the fan.
It's a interesting question, one I wonder now that our federal data is being exfiltrated to AI companies. If they train their models on the data; how does the law tell them to 'unlearn'?
Destroying the copies they took will be what the courts ordered, but the data will still be there.
