What sort of machines are you running that on?
-
-
Replying to @sundhaug92
Ancient ones. 2 hosts on the left have 2x Xeon E5420 (Harpertown) each (8c8t total per host), right side has 1x E5520 (Nehalem-EP) (4c8c total). We rewrote the entire application and optimized the critical path to make it work out. New hardware is in the pipeline.
2 replies 0 retweets 1 like -
Replying to @marcan42
Nice. How much RAM, storage and what kind of NICs?
2 replies 0 retweets 0 likes -
Replying to @sundhaug92 @marcan42
Sorry in re RAM, didn't get the hires vid from twitter before now. 4 gigs in a server? Ouch
1 reply 0 retweets 0 likes -
Replying to @sundhaug92
I know, right? They used to be 6+6 though I moved 2G from one to the other in order to maaybe fit in a VM or two, but never got around to it. Newer servers will have way more. The two smaller servers run an Elasticsearch cluster for debug/app logs and... yeah, that chews up RAM.
1 reply 0 retweets 1 like -
Replying to @marcan42
ES on 4 or even 8 gigs? Damn, I almost feel sorry for it
1 reply 0 retweets 0 likes -
Replying to @sundhaug92
Last night we closed all the indices except the last two months and dropped the Java heap to 800MB or so. Otherwise I'm sure we'd have had a really bad time.
1 reply 0 retweets 1 like -
Replying to @marcan42
Nice. Running ELK or something for logs and such or?
1 reply 0 retweets 0 likes -
Replying to @sundhaug92
Hector Martin Retweeted Hector Martin
EF(ilebeat)K, but yeah. We dump rich JSON logs from the app into a file and tail that. Invaluable debugging aid (and also great for dealing with annoying users). See the third image here:https://twitter.com/marcan42/status/993753850904330240 …
Hector Martin added,
1 reply 0 retweets 1 like -
The main dashboards are Prometheus + Grafana (though we have a couple useful event metric dashboards in Kibana too)
-
-
Replying to @marcan42
Hosted on the same three servers? Run syslog through it too?
1 reply 0 retweets 0 likes -
Replying to @sundhaug92
Prometheus/Grafana is on the 8GB server, with exporters on every host. Syslog isn't going anywhere but local logs yet, but will once we get new hardware and revamp the entire setup. The whole thing is a little haphazard right now, but will be a lot cleaner on the new hardware.
1 reply 0 retweets 1 like - Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.