How to import modified/rendered source code from url

[docasar]

I'm trying to insert the html code of a product page on a website for later Web Scraping Data.

I am aware of the standard practice of using "Insert from URL" to get the source code, but I would only retrieve the content of the unmodified (pre-javascript altered) page, not the page after javascript has modified the DOM.

Options I tried so far:

1) Make a webviewer from the url and use the GetLayoutObjectAttribute ( "webviewer" ; "Content" ) it shows the content of the unmodified (pre-javascript altered) page, not the page after javascript has modified the DOM on Filemaker Pro on mac.

2) As pointed from Tod on https://stackoverflow.com/questions/25561850/getlayoutobjectattributewebviewer-content-doesnt-reflect-javascript-dom/60683713#60683713  I created a new layout and run the same GetLayoutObjectAttribute ( "webviewer" ; "Content" ) on Filemaker GO on my iphone, doing that I did manage to grabe the post-javascript rendered source code.

What I would ultimately like to achieve would be to get the same data but using Filemaker Pro instead of Filemaker GO, could someone point me on the right direction to whether this is possible at all? 

- What I would like to avoid is to have the script running and having a middle step with wait for 5 seconds to be sure that the webviewer loaded correctly all the javascript content, instead would like to get the FULL code (rendered DOM) using the "import from url" function.

Thanks a lot for your help!

Luis

[docasar]

UPDATE for NOVICE developers like myself:

I finally found a solution, and apologise because I found that my first post wasn't completely right:

1) GetLayoutObjectAttribute ( "webviewer" ; "Content" ) on Filemaker Pro would actually get the modified DOM source code.

2) Since different webpages would run different scripts on Desktop version than on Mobile version (the one shown if using FILEMAKER GO), you can also get the DOM source code for the mobile version just by having an additional webviewer pointing to the mobile version of the url, normally if a web is: "htttps://www.name.com" for Desktop version, for mobile version it would likely be something like: "https://m.name.com". 

3) If you need the source code pre-javascript altered, just use the script step "Insert from url".

In my case I just solved it this way, I hope it helps anybody else that would be trying webscraping in the future.

I needed this functionality in my development because I am trying to download the top 100 sold  products for each category from a known Dropshiping site, for later review of the potential to add new products to our catalog. All this could be also done easier with the official site API, but I didnt have access to it yet. Since I couldnt found any other app or API that would retreive the DOM version of an url in bulk (only some chrome extensions that would do it but one by one by hand) I built this little tool to help me getting the HTML code with all information need for later webscrapping.

I attached to this post the sample file, that can be used for similar purposes just by creating a script that would loop through all the product urls records.

Again, thanks a lot for your time!

Luis

DOM_source.fmp12

Edited March 16, 2020 by docasar