Conversation

Replying to

@David_S_Dowling

and

They're blaming search engines for taking away money from them, but the search engines are directing users to their articles. If they didn't want to have snippets included in the results, they would disable it. twitter.com/DanielMicay/st They want to have snippets included though.

Quote Tweet

Replying to @tiraniddo and @semibogan

Sites have control over how their content is used in search results: developers.google.com/search/referen If they don't want snippets of their articles used in search results, they can disable it. They can also set a maximum length on how much is used. Can set it for a specific crawler too.

2

@David_S_Dowling

Replying to

and

Thanks for sharing this. Was aware of robots.txt but was unaware about x-robots-tag and snippets.

1

Replying to

@David_S_Dowling

and

You can also mark a section of the content with data-nosnippet to exclude only a portion of the content from being included in snippets. In general, it's usually in the best interest of the site to let search engines index all of it and include all of it as snippets in results.

1

1

Replying to

@David_S_Dowling

and

It helps to drive traffic to the site. The opt-out exists so that if a site really doesn't want their content used this way, they can disable it. This is the best way to control what search engines index and include in their results though. robots.txt is to control crawling.

1

1

Replying to

@David_S_Dowling

and

If you disallow access to a section of the site in robots.txt, that stops bots respecting robots.txt from crawling it. They can and will still index it based on links to it without being able to crawl it. It's better to let them crawl and set noindex, etc. via header or meta tag.

1

1

Replying to

@David_S_Dowling

and

They can still find and use links to other content on a page that's marked noindex. It stops them indexing the page and including it in results. robots.txt is really a legacy thing as properly made modern sites shouldn't have a reason to disallow bots from crawling parts of them.

1

@David_S_Dowling

Replying to

and

Agree about it being legacy, I have seen some AU govt agencies rely too much in it and basically disallow certain files and or directories that is not smart. This was a good read, decent Dunning-Kruger reminder that my opinions do not equal actual subject knowledge

1

Replying to

@David_S_Dowling

and

It was originally based on sites being overloaded by queries from bots. No modern site should have a problem with it and they're going to have serious problems with DoS attacks. Sites using it that way are basically documenting how to overload their servers instead of fixing it.

1

Replying to

@David_S_Dowling

and

It's usually used as a misguided way of trying to exclude stuff from search engines, but it doesn't actually do that. The reason for wanting to exclude content from search engines is also often that it's private and doesn't have proper access control. robots.txt is a code smell.

1

Replying to

@David_S_Dowling

and

The robots meta tag / header is a lot more useful and rel="canonical", redirects, proper access control and sitemaps (to prioritize crawling) cover most of the other reasons people use it. I'm sure most of these news sites are all over this to try ranking better in searches.

1

1

Replying to

@David_S_Dowling

and

If you look at their pages, they generally add OpenGraph metadata, Twitter metadata, structured data (json-ld, etc.) and various other things specifically to show richer snippets, etc. on social media. If they could do it, I'm sure they'd pay Google to rank higher in searches...

5:19 AM · Jan 23, 2021Twitter Web App