The need
High-volume scraping jobs locked up the system; there was no reliable, scalable structure.
What we did
Moved scraping into a job queue and served results over a REST API in a resilient, scalable way.
Result
Collection jobs run reliably in a queue; large data is processed without slowing the system.
Technology
- REST API
- BullMQ
- Redis
- PostgreSQL
Services used
