The first 5 minutes of @euskalencounter #EE26 reservations, at 5x speed. Our servers survived with zero failed requests. Another year, another huge success for #noF5day.pic.twitter.com/wLkun03f5H
-
Show this thread
-
The peak at 10:00 CEST is everyone hitting the "Reserve" button all at once. There's a slight lull as people go through the payment gateway, before moving on to seat selection. Seat selection is DB-heavy, hence the extended load on tohru (DB master). 5024 total seats available.pic.twitter.com/LQg1IzHto8
4 replies 3 retweets 24 likesShow this thread -
-
Replying to @sundhaug92
Ancient ones. 2 hosts on the left have 2x Xeon E5420 (Harpertown) each (8c8t total per host), right side has 1x E5520 (Nehalem-EP) (4c8c total). We rewrote the entire application and optimized the critical path to make it work out. New hardware is in the pipeline.
2 replies 0 retweets 1 like -
Replying to @marcan42
Nice. How much RAM, storage and what kind of NICs?
2 replies 0 retweets 0 likes -
Replying to @sundhaug92 @marcan42
Sorry in re RAM, didn't get the hires vid from twitter before now. 4 gigs in a server? Ouch
1 reply 0 retweets 0 likes -
Replying to @sundhaug92
I know, right? They used to be 6+6 though I moved 2G from one to the other in order to maaybe fit in a VM or two, but never got around to it. Newer servers will have way more. The two smaller servers run an Elasticsearch cluster for debug/app logs and... yeah, that chews up RAM.
1 reply 0 retweets 1 like -
Replying to @marcan42
ES on 4 or even 8 gigs? Damn, I almost feel sorry for it
1 reply 0 retweets 0 likes -
Replying to @sundhaug92
Last night we closed all the indices except the last two months and dropped the Java heap to 800MB or so. Otherwise I'm sure we'd have had a really bad time.
1 reply 0 retweets 1 like -
Replying to @marcan42
Nice. Running ELK or something for logs and such or?
1 reply 0 retweets 0 likes
Hector Martin Retweeted Hector Martin
EF(ilebeat)K, but yeah. We dump rich JSON logs from the app into a file and tail that. Invaluable debugging aid (and also great for dealing with annoying users). See the third image here:https://twitter.com/marcan42/status/993753850904330240 …
Hector Martin added,
-
-
-
Replying to @sundhaug92
The main dashboards are Prometheus + Grafana (though we have a couple useful event metric dashboards in Kibana too)
1 reply 0 retweets 2 likes - Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.