Need to click a button and post a form

Hi,

I’m new to Apify, I’m working on it for a while. I can crawl some static pages easily but when I come to AJAX or Form posts I couldn’t manage to make it work. Could you please help me with below scenario ?

The page I want to parse is https://www.remax.com.tr/ofis-office-franchise-girisimci-agent-arama

First I need to click on “simpleGd” input then post the “ofisGdFormName” named form. Then get all the agent information from the search results. When every page is finished move to next page until all pages are done. All of the events seems AJAX requests so I couldn’t figure out how to do it.

But for some reason I’m getting the following error

2019-06-13T11:39:00.227Z ERROR: Request https://www.remax.com.tr/ofis-office-franchise-girisimci-agent-arama failed and will not be retried anymore. Marking as failed.
2019-06-13T11:39:00.228Z Last Error Message: Error: Protocol error (Runtime.callFunctionOn): Cannot find context with specified id

my code is below

async function pageFunction(context) {
    const { request, log } = context;
    const title = document.querySelector('title').textContent;

    log.info('Select agent filter from the form.')
    document.querySelector('#simpleGd').click();

    log.info('Submitting the search form.')
    document.ofisGdFormName.submit();

    return [];
}

Hi @cankansu,

this is an interesting problem. I just wanted to let you know that we’re looking into it and I’ll update you when we have a solution ready.

Ok @cankansu, so, this is actually a deeper rooted problem that we may not have a direct solution for. And by a direct solution I mean enabling exactly the code you use, which is perfectly valid, to work. If that interests you, the trouble lies in the fact that the page navigates (reloads) after the form submission, which breaks the pageFunction execution. We have a solution ready, but it will take a week or two to become available.

Now, that doesn’t mean that Apify can’t handle form submissions. You just need to use a different tool to do that (for now). Instead of Web Scraper, please use Puppeteer Scraper. It’s very similar but it gives you more control over the browser. We also have a tutorial for Puppeteer Scraper.

The following code does what you expect in Puppeteer Scraper:

async function pageFunction(context) {
    const { request, log, page } = context;

    log.info('Select agent filter from the form.')
    await page.click('#simpleGd');

    log.info('Submitting the search form.')
    await Promise.all([
        page.waitForNavigation(),
        page.click('input[type=submit]')
    ])
    
    return [];
}

Refer to Puppeteer documentation for more information on the page. functions, but I figure that page.click() is pretty straightforward and page.waitForNavigation just waits for the page navigation to complete after clicking the submit button. So after that, you can continue extracting data.