HTTP_REFERER issue with web-scraper

I am trying to convert a (working) Crawler script to an apify/web-scraper Task. I managed to convert most of the script but I am having issues with POST requests.

Advanced Settings of the Crawler allow to set Custom HTTP Headers, including the Referer header.

In the web-scraper task I cannot find such a setting. I tried adding

headers: {
    "Referer": "https://example.org",
},

to my context.enqueueRequest() calls, but I still get an “Undefined index: HTTP_REFERER” error from the website.

Update : It seems that POST settings are simply ignored.

Hi @apify1,

thanks for the issue. We will check it and let you know.

Anyway, if scraper in legacy crawler works, you can easily migrate it to legacy-phantomjs-crawler. It has the same interface as an old legacy crawler.

1 Like