Update on the new Reddit ingest pipeline. The new ingest code is currently being tested for any esoteric bugs but so far things are looking really good.
Here are the targets for the new pipeline:
1) Max delay for indexing of new Reddit data will be targeted at 60 seconds.
Conversation
Replying to
That means ideally that new comments and submissions will be available for search within 60 seconds of their creation on Reddit.
2) A median time of less than 15 seconds for newly indexed information. The goal is to have new objects available for search in around 15 seconds
1
2
Over 99% of the time.
3) Reduce latency of all searches to below 250ms.
I will continue testing for any edge-case bugs but my feeling is that this will be ready for production in 2-3 weeks.
Thank you!
2
1