Simple nodeJS based crawler (no headless browser). Throughput ~350 req/min (support parallelism) with completely free proxies (tested on one of my website subdomains, no cloudflare to avoid being blocked).
Conversation
Replying to
Nice! Could be interesting to see how it compares to Scrapy + github.com/constverum/Pro for the free proxy part.
1
4
Replying to
Really cool link. I didn't know this libray. I've basically reimplemented a similar service that collects and checks/test free proxies continuously