Gather round, kids, it's time for another Elasticsearch bedtime story.
-
-
Replying to @alicegoldfuss
Now, I've been wrassling with a mean ol' cluster for some time now. We've come to know each other pretty well.
1 reply 1 retweet 2 likes -
Replying to @alicegoldfuss
And I know that when it doesn't assign shards, it's just being coy.
1 reply 0 retweets 2 likes -
Replying to @alicegoldfuss
Sometimes it's really coy, and I end up with hundreds of unassigned shards. This is why I wrote Shardnado.
1 reply 0 retweets 2 likes -
Replying to @alicegoldfuss
I recently upgraded my curmudgeon friend, which left me with about 1400 unassigned shards. So I fired up my script.
1 reply 0 retweets 1 like -
Replying to @alicegoldfuss
Now, Shardnado isn't perfect. I wrote it, after all. But it's pretty damn good and it assigned all but some 300 shards in the first go.
1 reply 0 retweets 2 likes -
Replying to @alicegoldfuss
So why did Elasticsearch show about 900 leftover?
2 replies 0 retweets 1 like -
Replying to @alicegoldfuss
Was my script wrong? Did it somehow fail and I never caught it in tests? Had I just published a repo of lies?
1 reply 0 retweets 2 likes -
Replying to @alicegoldfuss
No.
1 reply 0 retweets 1 like -
Replying to @alicegoldfuss
Turns out (and you can see this in logs when it happens) that Elasticsearch will timeout assignment requests after 30 minutes.
2 replies 0 retweets 2 likes
eh it’s been a while they don’t really want this i guess
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.