I’m having an issue when creating a broken link crawler. I’m trying to crawl tvo.org and the site is about 16,000 pages
After a while I start getting Timeout errors with the error of Navigation Timeout Exceeded: 30000ms on some of the pages. I noticed that the Autoscaled Pool state is giving,
So I’ve tinkered with the puppeteerPoolOptions and autoscaledPoolOptions, but I’m still getting the navigation timeout errors.
Here is my source for the actor:
Anyone have any ideas to remedy this problem?