I have to crawl about 5000 couples of webpages (so 10000 total). I have written my crawler, only to figure I would explode the monthly limit of the free plan. I wish I could get a paid plan but unfortunately, currently I am really unable to afford it (I wonder what are the limits for paid plans, though ? They don’t appear on https://apify.com/pricing).
Then I guessed I may get my way with an actor, which this post seems to confirm :
With Actor you can crawl/scrape even more that 10k pages on a free plan - depends on approach (pure request / Puppeteer with headless mode / non-headless mode / AJAX calls / etc.)
But I’m not sure to understand it correctly, is there a particular efficient approach ? Or does it just mean that it depends on the memory required, in which case the webpages size is important as well ? What if I just feed my crawler to the actor ?
Also, how are actor units computed ? The pricing page indicates 1 unit = 1 hour @ 1 GB RAM, but is there a linear relationship between both ? I.e. 1 unit = 1/2 hour @ 2 GB = 4 hour @ 250 MB ?
Finally, reading this example I am also guessing that if an actor calls a crawler, the crawler is still subject to monthly crawling limit, am I right ?
Edit : I just realized apify/web-scraper tasks might be the actual successors of crawlers.