Most of the problem has to do with ownership of the reference to a graph node or edge. It's also why (doubly-)linked lists are hard.
I imagine some tricks could be played with RefCell<T>, Rc<T>, and Weak<T>, or some composition thereof, to get somewhere fruitful.