Sorry in re RAM, didn't get the hires vid from twitter before now. 4 gigs in a server? Ouch
-
-
Replying to @sundhaug92
I know, right? They used to be 6+6 though I moved 2G from one to the other in order to maaybe fit in a VM or two, but never got around to it. Newer servers will have way more. The two smaller servers run an Elasticsearch cluster for debug/app logs and... yeah, that chews up RAM.
1 reply 0 retweets 1 like -
Replying to @marcan42
ES on 4 or even 8 gigs? Damn, I almost feel sorry for it
1 reply 0 retweets 0 likes -
Replying to @sundhaug92
Last night we closed all the indices except the last two months and dropped the Java heap to 800MB or so. Otherwise I'm sure we'd have had a really bad time.
1 reply 0 retweets 1 like -
Replying to @marcan42
Nice. Running ELK or something for logs and such or?
1 reply 0 retweets 0 likes -
Replying to @sundhaug92
Hector Martin Retweeted Hector Martin
EF(ilebeat)K, but yeah. We dump rich JSON logs from the app into a file and tail that. Invaluable debugging aid (and also great for dealing with annoying users). See the third image here:https://twitter.com/marcan42/status/993753850904330240 …
Hector Martin added,
1 reply 0 retweets 1 like -
-
Replying to @sundhaug92
The main dashboards are Prometheus + Grafana (though we have a couple useful event metric dashboards in Kibana too)
1 reply 0 retweets 2 likes -
Replying to @marcan42
Hosted on the same three servers? Run syslog through it too?
1 reply 0 retweets 0 likes -
Replying to @sundhaug92
Prometheus/Grafana is on the 8GB server, with exporters on every host. Syslog isn't going anywhere but local logs yet, but will once we get new hardware and revamp the entire setup. The whole thing is a little haphazard right now, but will be a lot cleaner on the new hardware.
1 reply 0 retweets 1 like
Basically what we have here is a cut-down/simplified version of the server environment we use at the actual event, which is a *lot* cleaner (and fully redundant) and runs on newer servers. We're replacing the 4G/8G ones with brand new servers soon and replicating that setup.
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.