Website would require "activated JavaScript" - crawled output doesn't contain required info


#1

The information that I want to extract from a crawled website seems to require that JavaScript is activated… if I visit the crawled URL with a normal browser… I see the required information… however, the same URL crawled by Apify does not contain the required elements in the HTML code.

Is it possible that the Apify Crawler is simulating a client with active JavaScript in order to get all information that would be available with a normal browser?

Any help would be appreciated…

Best

AJ


#2

Hello @aj1,
thanks for the post in the first place.
It seems that some parts of the data on a page are loaded dynamically.
You can wait for some element even in the crawler with our example code here.
Just find a selector you need to wait for and replace it with the $('#my_element').
Or place here the URL for a site you are crawling and we can try to help you here.
Best,
Vaclav


#3

Hello Rut

I don’t think it is a timing issue…
Is there a possibility to send you the URL with a personal message… the website I want to crawl contains some personal information that I don’t want to publish here…
Thanks for your feedback
AJ


#4

Hi @aj1,

If you want to you can send me a link to jakub.drobnik@apify.com and I’ll look at it.


#5

My Google Mail server return the email jakub.drobni@apify.com, stating it is not a valid email address

Typo? :wink: