we hit a bail:meme, which is urbit for "no more memory." right now, we're trying to replay the events past the bail so that we can clear some cache or otherwise free up some space.
Show this thread
~zod is running into some memory issues. trying some hotfixes to get it to turn over so we can find where we're leaking.
-
-
-
offending line, for those following along at home https://github.com/urbit/urbit/blob/a42f2cbe5ccc591c148464444264c8f6c92776e3/noun/manage.c#L39 …
Show this thread -
we got ‘em back up enough to wipe the %ford cache and clear some space; can replay events now. networking off, of course, so we don’t get “neighbored” to death.
Show this thread -
we’ll do some forensics on a copy of the pier to see what we can see and get ourselves ready for...
Show this thread -
-
challenge now is resurrecting https://github.com/urbit/arvo/blob/master/gen/hood/mass.hoon … from its watery grave so we can see which vane is to blame
Show this thread -
|mass is sort of back. now we can see who’s taking up all the space.pic.twitter.com/VlsSToR3nd
Show this thread -
interesting results! the nock->bytecode cache and the ford build history/cache were the two big culprits. ford weighing in at ~500MB. doing some cleanup and will retry on a beefier instance.
Show this thread -
well. after our emergency field surgery we got ~zod on the network...for one event.
Show this thread -
~zod had too many friends and tried to call them all back in the same event!
Show this thread -
we have a better idea now of a way to clear our caches. well give it a shot and update here!
Show this thread -
Two new PRs to address this issue: https://github.com/urbit/urbit/pull/1168 … andhttps://github.com/urbit/urbit/pull/1167 …
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.