Scrolling page using lazy load


#1

Hi,

I am scraping this page. The page only loads the items on screen (i.e. when you scroll to the bottom, the items above are no longer loaded). I tried using ajax but that wouldn’t work, and I tried to adapt code from https://kb.apify.com/tips-and-tricks/waiting-for-dynamic-content, which wouldn’t work either.

I think if the crawler scrapes the initial products, presses the spacebar, waits 2 seconds, scrapes the newly loaded data and then repeats until there are no more products it will work. I am unsure how to implement this though?

Any chance you could point me in the right direction? Thanks!

This is my code

    function pageFunction(context) {
    
    var $ = context.jQuery;
    var result = [];
    
    var spaceBar = $.Event( "keydown", { keyCode: 32 } );
    
    var _category = $(".bc-desktopBreadcrumbs__breadcrumb:eq(1)").find('a').text();
    var _subcategory = $(".bc-desktopBreadcrumbs__breadcrumb:eq(2)").find('a').text();
    var _subsubcategory = $(".bc-desktopBreadcrumbs__breadcrumb:eq(3)").find('span').text();
            
       $(".fop-item").each( function() {
                    if (_subsubcategory === "") {
                    
                        _subcategory = $(".bc-desktopBreadcrumbs__breadcrumb:eq(2)").find('span').text();
                        
                        result.push({
                        
                        title : $(this).find('.fop-title').text().trim(),
                        price : $(this).find(".fop-price").text().trim(),
                        image : $(this).find('img').attr('src'),
                        
                        category : $.trim(_category),
                        subcategory : $.trim(_subcategory),
                 
                    });
                  }
                   else {
                        result.push({
                    
                        title : $(this).find('.fop-title').text().trim(),
                        price : $(this).find(".fop-price").text().trim(),
                        image : $(this).find('img').attr('src'),
                        
                        category : $.trim(_category),
                        subcategory : $.trim(_subcategory),
                        subsubcategory : $.trim(_subsubcategory),
                 
                    });
                  }
                  
        $(".fop-item").trigger(spaceBar), 2000;
                  
    });
    
   return result;
    
}

#2

Hi @Benjamin_Doyle,

I think you can use their XHR request, you can call XHR like:
https://groceries.morrisons.com/webshop/api/v1/products?skus=350864011,289819011,120431011,210086011,211713011,230453011,207591011,346187011,289879011,217759011,253342011,217815011,120603011,120584011,120728011,230928011,217791011,217932011,294772011,264176011,289908011,214244011,346184011,120606011,352920011,212404011,346175011,294774011,120600011,120559011

You have to just replace ids in skus, you can get these ids from page script tag:


or you can get it from window.INITIAL_STATE variable.