> So in the current state of affairs only the scraper gets paid, and the content creators are left with the bill.
Googlebot respects robots.txt. Anyone who doesn't want to be indexed, isn't. For some reason they still seem to want to be.
> I'm sure that we can agree that journalism and web scraping have nothing in common, just like xeroxing a book is not the same thing as writing a novel.
What do you mean? It's basically the same thing. When a reporter interviews some guy, they put his words in their story -- without compensation. How is that fair?
Googlebot respects robots.txt. Anyone who doesn't want to be indexed, isn't. For some reason they still seem to want to be.
> I'm sure that we can agree that journalism and web scraping have nothing in common, just like xeroxing a book is not the same thing as writing a novel.
What do you mean? It's basically the same thing. When a reporter interviews some guy, they put his words in their story -- without compensation. How is that fair?