Conversation

foone🏳️‍⚧️
@Foone
my favorite thing with doing continuous integration is that sometimes it breaks and you're like "how am I doing it wrong? what did I break?" and you spend a week investigating that and it turns out the answer is "it never worked, for anyone"
19
863
foone🏳️‍⚧️
@Foone
I just made the silly assumption that maybe one of the main repos for this organizations code would be, you know, clonable? and have files in it? that was a foolish assumption
1
188
foone🏳️‍⚧️
@Foone
we had a repo hosted on a local gitea instance and then it got changed to be an automatic mirror of a private github repo, which worked fine
1
94
foone🏳️‍⚧️
@Foone
and then someone turned on git-lfs for the github repo, which worked fine for people checking out directly from the github repo, but the gitea mirror started failing
1
90
foone🏳️‍⚧️
@Foone
because it turns out while the git repo itself was getting mirrored down to the gitea repo, the git-lfs files were not. so when you tried to check out of the local gitea mirror, it would 404 downloading all the git-lfs files
1
90
foone🏳️‍⚧️
@Foone
and this is the kind of thing that would have been spotted immediately because the automatic tests would have broken, and they did... but no one noticed because the automatic tests broke for a different but related reason: the test server didn't support git-lfs
3
109
foone🏳️‍⚧️
@Foone
so I spent a while trying to fix the "test server doesn't support git-lfs" problem, only to discover that it doesn't help at all. it just enables the test server to check out a broken repo, because git-lfs tries to download a lot of files that are missing
1
95
foone🏳️‍⚧️
@Foone
Replying to
@Foone
reminds me of that time at a previous job where someone had broken the code that compiled a support-tool for tests, but no one noticed for months because: 1. the failing to build the tool didn't kill the test 2. if it failed to build a tool, it used the last-successful-build
2
126
foone🏳️‍⚧️
@Foone
so for months there was a "trying to compile test tool.... FAILED!" in the logs, and no one noticed because it was line #2194 of 40mb of log files, and the test still passed
1
96
foone🏳️‍⚧️
@Foone
we only finally noticed because we tried to set up a new test server and it wasn't able to run any tests, because it had no previously-successful build to fall back on.
1
79
foone🏳️‍⚧️
@Foone
the two most important things I've learned in doing continuous integration 1. if it looks like it is working, it probably isn't, and you just can't see that it's failing 2. if it looks like it's failing, it probably isn't, and you just can't see that it's succeeding
1
169
foone🏳️‍⚧️
@Foone
basically, never underestimate how much stuff is broken but just seems to be working because of inertia.
3
159
foone🏳️‍⚧️
@Foone
you ever unscrew a lightswitch panel and all the screws are out but it stays attached to the wall, because the paint is effectively gluing it to the wall? The expected support is gone, but it stays in place because of the unintended support.
1
149
foone🏳️‍⚧️
@Foone
it turns out that when you're doing continuous integration, you learn that half the shit in your process is like that. The screws have been gone for years. the paint is slipping.
1
103
foone🏳️‍⚧️
@Foone
you go to put up a painting on the wall and the lightswitch panel falls down. Huh. that's weird. You spend a week trying to figure out how your painting broke the lightswitch
1
92
foone🏳️‍⚧️
@Foone
only to eventually realize that no, your painting did nothing. your lightswitch had been broken for a long time, and your painting just jostled the situation enough to make an already broken thing finish falling over.
1
115
foone🏳️‍⚧️
@Foone
if I had a nickel for every time I changed X and Y broke, and I went looking for the link only to find there was none, and Y had already been broken for a long while and we just didn't notice, I could retire from trying to fix these things
3
140
foone🏳️‍⚧️
@Foone
I remember saying that back when I was a win95 repair tech in the late 90s. we weren't engineers or scientists, we were magicians.
3
88
foone🏳️‍⚧️
@Foone
we didn't diagnose problems and repair things, we knew rituals we could do. we'd try them and maybe they'd work. sprinkle the right holy water, say the right prayers, reinstall tcp/ip, rearrange the RAM sticks, maybe the gods will be pleased today.
3
123
foone🏳️‍⚧️
@Foone
I remembered thinking that maybe this was just a problem with the current win9x state of things and one day it would be better. ahh, youthful optimism
1
93
foone🏳️‍⚧️
@Foone
computers have gotten more reliable in general, which just means that when they break, they break in far weirder ways.
2
134
foone🏳️‍⚧️
@Foone
it's like that line in Charles Stross's Accelerando, about the guy who has some weird brain cancer. In the future world where they've cured all the common stuff, that's all that's left.
1
68
foone🏳️‍⚧️
@Foone
we mostly fixed the problem that random windows system files get corrupted and can't be repaired, so now when shit breaks it breaks in ways that you can spend two weeks trying to repair and give up.
3
77
foone🏳️‍⚧️
@Foone
it's not "oh this server can't talk to that server because it's using the wrong DNS" anymore.
1
62
foone🏳️‍⚧️
@Foone
It's "this server's got the wrong certificates and we didn't notice because part of the test process refreshes the certificates but the mirror uses HTTP and the original git server uses HTTS and this system should be using the mirror but we had to move it to the original because
1
88
foone🏳️‍⚧️
@Foone
that git server is inside the DMZ which means the now isolated test beds can't talk to the mirror directly, so they have to use the original. which worked for all the existing servers, but then we tried to install new servers, which were missing certificates
1
71
foone🏳️‍⚧️
@Foone
and the test would have happily gotten the new certificates because it downloads them but it was trying to get them off the git server which it can't talk to unless IT HAS THE CERTIFICATES"
1
74
foone🏳️‍⚧️
@Foone
you're half a mile down the chicken-and-egg problem rabbit hole and this took a week to figure out, to mix your metaphors, and you're considering quitting any job that involves touching a computer and moving to montana to raise actual literal chickens and rabbits
3
113
foone🏳️‍⚧️
@Foone
I grew up on a farm. the main problems I had to debug was that chickens had gotten out of their enclosure or that a fox had gotten into it those are both a lot easier to fix and don't make you hate technology
2
110
foone🏳️‍⚧️
@Foone
and plus if you take care of a chickens they will give you eggs and you can scramble them or make a cake that's a much better output. my work with computers just gives me beeps and pages of errors. and you can't eat that
3
116
foone🏳️‍⚧️
@Foone
unless you print it out. paper is technically edible even if it's not nutritious. don't eat thermal paper, though. that's known to the state of cancer to give you california
7
134
foone🏳️‍⚧️
@Foone
also you can get printers that print on cakes. but it would be silly to have all your errors come out in the form of cakes
Image
9
95
foone🏳️‍⚧️
@Foone
that sounds hard and silly but more fun than trying to finish debugging this git-lfs-github-gitea-vm-mirroring nightmare
1
71
foone🏳️‍⚧️
@Foone
honestly some days the main thing that keeps me from going mad and running off into the wilderness to devote my life to one silly singular goal (like building a pita printer) is that I have too many silly singular goals to pick one to hyperfocus on
4
85
Show replies