A move which could see Beckenham Library relocated and the site developed into housing has been met with howls of anger from residents.