I have a financial advisor directory service and trying to automate some of my background checking process. I would like to pull their certification information from the SEC. These reports are publicly available but I don’t want to waste time doing it manually.
After inspecting the code, the PDF is automatically generated and has a content disposition of inline and the name is randomly generated but the content type is application\pdf.
Here is a sample url: https://www.adviserinfo.sec.gov/IAPD/Support/ReportViewer.aspx?indvl_pk=5829233
When I try the analyzer tool, it fails to pull anything. I believe this is because the report is built on the fly and takes a few seconds, sometimes up to 20 seconds, to build the report and display in the browser window.
I needs to ideas on how to code the crawler to avoid an empty pull. I have some experience coding VB and HTML, but need to some pointers to get me started.