Link selector is not working

Hi,

For this page

https://www.remax.com.tr/ofis-office-franchise-girisimci-agent-arama

I’m trying to go over all result pages. I’m using CSS Selector as

a[onclick^=“getPage”]

but for some reason nothing happens. There is no even an error in output.

How can I be sure I’m doing it right ? I checked myself with w3schools and it seems correct.

https://www.w3schools.com/cssref/css_selectors.asp

Hi @cankansu,

you should use the selector without the double quotes:

a[onclick^=getPage]

This selector will find the elements you’re looking for.

Hi @mnmkng

I changed the CSS selector but nothing happened. How can I be sure that the selector I’m writing is correct ? Or maybe Pseudo-URLs isn’t event triggered because I cannot see any info in the console when Task is running.

Maybe it is because of the Pseudo-URL, probably I’m missing something.

Let me write down my start-url and pseudo-url;

Start-URL:
url: https://www.remax.com.tr/ofis-office-franchise-girisimci-agent-arama
method: POST
payload: sayfa=1&secim=gd&formDurum=ARA
headers:
{
“Content-Type”: “application/x-www-form-urlencoded”,
“cache-control”: “no-cache”
}

Pseudo-URL:
url: https://www.remax.com.tr/ofis-office-franchise-girisimci-agent-arama
method: POST
payload: sayfa=[\d+]&secim=gd&formDurum=ARA
headers:
{
“Content-Type”: “application/x-www-form-urlencoded”,
“cache-control”: “no-cache”
}

Link-selector:
a[onclick^=getPage]

And my Page function is:

async function pageFunction(context) {
const { request, log } = context;

var results = [];
var countOfAgents = 1;

log.info(document.querySelectorAll('.font_16_Normal')[2].innerText);

//Get the count of pages to cover
var resultPageNumbers = document.querySelector('.sayfalamaDiv').querySelectorAll('a');
var totalPageNumber = 0;

resultPageNumbers.forEach(function(page){
    if(page.firstElementChild.innerText.startsWith('..')){
        log.info('There are: ' +  page.firstElementChild.innerText.replace('..','') + ' pages to cover' );
        totalPageNumber = parseInt(page.firstElementChild.innerText.replace('..',''));
    }
});

//Get each agent url and log it.
var agents = document.querySelectorAll('.gdListTitle');

agents.forEach(function(agent){
    log.info(countOfAgents + ') ' + agent.querySelectorAll('a')[0].href);
    
    results.push({
        url:agent.querySelectorAll('a')[0].href
    });
    
    countOfAgents++;
});

return results;

}

and Navigation wait until is:

[
“load”,
“networkidle2”
]

Hi @cankansu,

I just wanted to let you know that we were looking into this and after some deliberation we’ve decided not to implement POST request navigation into Web Scraper. As I’ve outlined in another thread of yours: Need to click a button and post a form please use Puppeteer Scraper for this kind of jobs. It is much betters suited to it.

Thank you for understanding.