Crawl several urls in the same crawler


#1

Hi,

I am new to Apify. I just create a crawler which works great. But I am not sure to understand how to use start and pseudo urls.

I want to crawl several urls. So I add them as startsUrls. But when I run the crawler, only the first is crawled.

Do I have to loop on the url or is it done automatically ?

I didn’t find any examples in the doc.

Thanks

Bastien


#2

Hi @bastien,

It is hard to say what is wrong. Can you give me a crawler, where you found this issue?

One problem can be if you have the same URL, in that case, crawler thinks that is already crawled. You can avoid that with adding url#random_number to start URLs e.g.: http://example.com/#1, http://example.com/#2.


#3

Hi,

Thanks for your reply. I found it. It’s just that the Max pages per crawl was set to 1. I set it to 5 and it works.

Now, I try to find the best way to visualize the result through a google sheet.