I've gotten a lot of questions related to what stack I use on the back-end for the API and for the front-end points. Here is a list:
Conversation
Replying to
Ubuntu 16.04 LTS w/ ZFS & ECC RAM
PostgreSQL 9.6
Redis 4.0
Nginx 1.12
Python 3 w/ Falcon
Elasticsearch 5.6
1
Replying to
For cache assist, I use Cloudflare with failover capabilities. Cache for most endpoints is only 1 second but helps tremendously.
Replying to
Front-end:
CSS 3 / HTML 5
JQuery
Highcharts
D3.js
Moment.js
Underscore.js
sprintf.js
lodash.js
2
Replying to
Basic ingest concept is to rpush data into a Redis list. Focus is to make that pathway very robust, and then process the data out of Redis.
Replying to
The philosophy is to separate the ingest operations from the processing of the ingested data so the later never interferes with the former.
Replying to
Redis is backed up immediately to NVMe drive using fsync flush to NVMe after every X rpushes (usually 10)
Replying to
Redis DB size is 16GB which accommodates a day's worth of #reddit data without issue. If processing stops, that leaves ~ 24 hour buffer.