Conversation

This Tweet was deleted by the Tweet author. Learn more

Replying to

@semibogan

It is like this is one big marketing push to get more Australians using DuckDuckGo.

Replying to

and

I don't think DuckDuckGo is going to start paying sites they include in their search results. I would expect them to take a stand against being forced to do that. It's hard to understand news sites complaining about links to their articles based on their titles and descriptions.

Replying to

and

You raise a good point. I think the news sites are angry at their lack of market power because Google has 90+% market share and makes over 3 billion out of AU market. But you are correct DuckDuckGo is not going to pay, I don’t have any useful fixes for this

Replying to

and

They're blaming search engines for taking away money from them, but the search engines are directing users to their articles. If they didn't want to have snippets included in the results, they would disable it. twitter.com/DanielMicay/st They want to have snippets included though.

Quote Tweet

Daniel Micay

@DanielMicay

Jan 23, 2021

Replying to @tiraniddo and @semibogan

Sites have control over how their content is used in search results: developers.google.com/search/referen If they don't want snippets of their articles used in search results, they can disable it. They can also set a maximum length on how much is used. Can set it for a specific crawler too.

Replying to

and

Thanks for sharing this. Was aware of robots.txt but was unaware about x-robots-tag and snippets.

Replying to

and

You can also mark a section of the content with data-nosnippet to exclude only a portion of the content from being included in snippets. In general, it's usually in the best interest of the site to let search engines index all of it and include all of it as snippets in results.

4:35 AM · Jan 23, 2021Twitter Web App

Replying to

and

It helps to drive traffic to the site. The opt-out exists so that if a site really doesn't want their content used this way, they can disable it. This is the best way to control what search engines index and include in their results though. robots.txt is to control crawling.

Replying to

and

If you disallow access to a section of the site in robots.txt, that stops bots respecting robots.txt from crawling it. They can and will still index it based on links to it without being able to crawl it. It's better to let them crawl and set noindex, etc. via header or meta tag.

Show replies