Request queue in web scraper not working

#1

Hi all,

I’ve trying for a couple of hour to make request queue working but It failed every time.

I don’t know if Pseudo-URLs are essential for request queue, as Link selector alone will be enough for me, but even with Pseudo-URLs its not working.

Finally I solved my problem by adding URLs to Start URLs, but it would be better if I can get it automatically by request queue.

Do I need to add any thing in Page Function?

#2

Hi,

The Pseudo-URLs important for page classifications and also used for adding links to queue.
I build logic of my crawlers label based which helps to crawl different info type.

Recently I had an issue with it because I miss to wrap regular expressions to squire brackets [ ].
Just make sure you have them as a wrapper. After adding them everything should work.

Thanks,
Sergey

#3

I’m not sure about your question.

But you can put request to queue with function addRequest.
If you use apify/web-scraper you can use it there as enqueueRequest.