How to crawl all pages under subdirectory, and return two values

#1

I am very new to crawling, and not a developer…but looking for some help if I may.

I would like to crawl all the pages under a url;
https:\thesite.com\article\

I know there are hundreds of pages under this address.
…article\trees
…\article\air
…article\sea
…article…and so on

I need to crawl for all pages under …\article\ and obtain from them the values

I naively thought i could use the apify/web-scraper template, which recursively returns the title of its pages, and just insert my URL, at least to get the title.

It only returns the title from the page …\article\ and none of the pages under it.

I would love some help to get me started on this journey!

#2

Hi @ije,

I think you need to set up start URL and Pseudo URLs correctly.
If you want to you can check web-scraper tutorial, where you can find information, how you can set it up.