nice post! One small nit: I believe that Internet Archive does not "remove captures from the Wayback Machine" when a robots.txt excludes content. Instead it refuses to replay the content. The data is still sitting on disk waiting to be replayed if the robots.txt were to change.
-
-
-
Thanks! We will amend.
End of conversation
New conversation -
-
-
Great post - much appreciated!
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
This Tweet is unavailable.
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.