Conversation

Is anyone else seeing a ton of GitHub spam mirrors on Google lately? Take this query: the first result is wrong (it's the user page, not the repo I'm looking for), then everything else is spam. The thing is, Google doesn't have the repo on GitHub indexed at all!

142

Hector Martin

@marcan42

Jul 29, 2021

It almost looks like Google has given up on indexing GitHub properly. The robots.txt does not block crawlers from indexing repo top-level pages, but somehow they don't get indexed. And then the spam mirror sites somehow do.

Daniel Micay

@DanielMicay

Replying to

@marcan42

GitHub has extreme throttling of crawlers as a workaround for bad performance. GitHub might not be bothering with excluding googlebot from it. Even if they did, it would still be hindering alternate search engines and backlink tracking services like ahrefs.com/webmaster-tools, etc.

4:22 AM · Jul 29, 2021Twitter Web App

Likes

Who can reply?

People @marcan42 follows or mentioned can reply