How to clear the cache of web viewer.

[Megalo]

I have the PC, in which web-crawling script is running 24 hours straight.

As time goes by, it seems the PC is getting slower and slower.

I'm guessing that it's mainly because of the cache that web viewer makes.

If anyone knows how to manage this kind of batch transaction, what would be the best way? any solution?

As a consequence, I found out it's better to shut down FileMaker application and launch again and again repeatedly. It seems the most stable.

Thanks!!

[Agnes Riley]

Can you please describe what your script does?

I didn't know of a way to clear the cache per se, but we can help you better once we know what you're trying to achieve.

[Megalo]

It's so simple. just like this. shops/#### goes up to 10000.

But as time goes by, filemaker gets slower.

Set Variable [$shop; Value: $1]

Loop

Set Web Viewer [Object Name: "Web"; URL: ''http://test-example.com/shops/'' & $shop ]

Set Variable [$HTML; Value: GetLayoutObjectAttribute( "web" ; "content" )]

Set Variable [$shop; Value: $shop+1]

Pause/Resume Script [Duration (seconds): 3]

Set Field [shops::HTML; $HTML]

Set Web Viewer [Object Name: "Web"; Action: Reset]

Set Variable [$HTML; Value: ""]

Go to Record/Request/Page [Next; Exit after last]

End Loop

[fseipel]

In Windoze, If you go to task manager (Start|Taskmgr) you can observe the memory allocation for Filemaker go up with each successive page request. This reveals the memory leak causing the slowdown. It is expected the memory allocation will go up with successive loads, but then be released later. Some memory IS released but allocation rises continuously in WebViewer. Contrast this to what you see if you load the page in IE in Task Manager. Memory goes up, but then drops back. WebViewer is an embeded instance of Internet Explorer, at least under Windoze. For the embedded browser (WebViewer) the cache is likely not clearing until FM is closed, but that is a separate issue than the memory leak IMHO. Otherwise, the allocation would go up monotonically when you reload the page in a standalone Internet Explorer instance. Eventually I'm assuming the memory usage eventually exceeds physically installed memory, virtually memory (hard drive) is used, causing a slowdown after thousands of sequential requests in a single FM session.

Under Windoze, you can track this sorry situation by periodically executing the DOS command tasklist /fi "imagename eq FileMaker Pro Advanced.exe" (substitute your version as read from task manager) and dumping the results to a text file at intervals.

Presumably this is a screen scraping operation. In that case, turning off flash and setting browser to render text only, will presumably greatly reduce the leak. Better still, would be to never render the pages in the first place, but instead load the raw HTML using free ScriptMaster, free Baseelements, or paid Troi Url Plugin, or any other similar means which will avoid the memory leak and speed problem. Filemaker can also re-load the solution at intervals, with a startup script to resume where it left off if no one clicks a button within say 10 seconds, but the former solutions would be preferred. Adding longer delays so page loads entirely MIGHT help but you'd need to look at a log to test that theory.

[Lee Smith]

It's so simple. just like this. shops

Please update your profile to show your current version of FileMaker, Platform and OS. Use this Quick Link to MY PROFILE to speed up the process.

Lee

[Megalo]

In Windoze, If you go to task manager (Start|Taskmgr) you can observe the memory allocation for Filemaker go up with each successive page request. This reveals the memory leak causing the slowdown. It is expected the memory allocation will go up with successive loads, but then be released later. Some memory IS released but allocation rises continuously in WebViewer. Contrast this to what you see if you load the page in IE in Task Manager. Memory goes up, but then drops back. WebViewer is an embeded instance of Internet Explorer, at least under Windoze. For the embedded browser (WebViewer) the cache is likely not clearing until FM is closed, but that is a separate issue than the memory leak IMHO. Otherwise, the allocation would go up monotonically when you reload the page in a standalone Internet Explorer instance. Eventually I'm assuming the memory usage eventually exceeds physically installed memory, virtually memory (hard drive) is used, causing a slowdown after thousands of sequential requests in a single FM session.

Under Windoze, you can track this sorry situation by periodically executing the DOS command tasklist /fi "imagename eq FileMaker Pro Advanced.exe" (substitute your version as read from task manager) and dumping the results to a text file at intervals.

Presumably this is a screen scraping operation. In that case, turning off flash and setting browser to render text only, will presumably greatly reduce the leak. Better still, would be to never render the pages in the first place, but instead load the raw HTML using free ScriptMaster, free Baseelements, or paid Troi Url Plugin, or any other similar means which will avoid the memory leak and speed problem. Filemaker can also re-load the solution at intervals, with a startup script to resume where it left off if no one clicks a button within say 10 seconds, but the former solutions would be preferred. Adding longer delays so page loads entirely MIGHT help but you'd need to look at a log to test that theory.

Thanks for explanation. That helped my understanding.

It's going pretty fine so far since I switched my client PC to good one. Now it rarely slowdowns and stops.

Is there anyone who use FileMaker to do this kind of batch script?

Share the techniques for stability, please!!!

[fseipel]

I'm glad the post was useful. To clarify my previous suggestion regarding images, go to Internet Exploder and

click Tools|Internet Options|Advanced Tab|Multimedia Heading|Uncheck Show pictures

This will disable the loading of images. I suspect this will reduce the memory leak. It will certainly decrease bandwidth requirements and page load times for media heavy pages. The reason this settings change propagates to Filemaker's WebViewer, is because it is an embedded Internet Explorer instance. This recommendation applies only to Windoze platforms. I'm not even sure this problem exists for Macs?

I have done screen scraping, but I never use the WebViewer, due to speed/stability issues. The WebViewer is best used only when you need to display the data, otherwise, rendering the HTML and downloading the images, just consumes processor time.

Also if you want to track the memory leak in FM itself you can just have FM run tasklist /fi "imagename eq FileMaker Pro Advanced.exe" > c:users[Your Username Here]Documentsleaky.txt and then having an FM script import leaky.txt shortly thereafter and placing the contents and a timestamp in the same record to form a log or retrieving standard output by using ScriptMaster's run shell script command, in which case no need to redirect output as above. Doing this every 5 minutes might provide some insights.

You can always re-start Filemaker automatically once memory allocation exceeds a preset limit or after a certain number of requrests. A simple method of doing this would be to export a text file RESTARTME.TXT; if this exists when file is re-opened (an automatic startup script would check for it), then script would start automatically. Before closing FM, script would run a DOS batch file or VBSCRIPT that pauses 20 sec and then reloads the solution. That way FM would have time to close. If VBSCRIPT is used you can even have it fire the script directly. Again, though, best to avoid WebViewer altogether.

[Megalo]

Dear fseipel

Thanks for the most detailed explanation ever:) I tried exporting memory leak log, and it went well!! I'll try keep tracking this from now on.

About turning displaying images off, I was already doing it before but I was still getting slowdown:(

As far as I know, the best solution is to restart FileMaker or PC so far.

You were saying that you never used the WebViewer to do screen scraping. Then how do you do it?

If you have something that you want make PC keep doing batch transcation continuously, what would be the best way?

Cron with the batches that are written in other languages???

For instance, the users email to certain email address, then batch will edit the attached image automatically 24/7 straight.

What is the best way to keep doing this batch without stopping or slowdown???

[fseipel]

For screen scraping, I would suggest 360 Works ScriptMaster (free), Baseelements (free), the paid Troi Url Plugin, or cURL (free and powerful but klunky interface to FM), or MBS plugin (nice interface to cURL from FM, and also handles e-mail send/receive and running PHP from FM). This way you can acquire the HTML page contents without rendering it. I realize the memory is released when FM is closed; out of curiosity, now that you're logging, is memory released when you close the DB but leave FM open? Changing layouts away from WebViewer does not release the memory but I've never checked if closing the DB does, in which case DB could simply open another DB that closes the first and then reopens it and closes the second every few hours.

You can use PHP to process the e-mails. Running a PHP script on a server machine may be most reliable. Alternately, FM can process e-mails, using ScriptMaster to send/receive, or use one of the e-mail plugins from 360 Works, Troi, etc. Including a watchdog on another computer to send a test e-mail once a day & e-mail someone if it isn't processed (server died) automatically is useful.

Alternately you can simply use Outlook and VBA. Google auto-responder, e.g. http://vbaexpress.com/forum/showpost.php?p=94609&postcount=4 -- one advantage of this is plenty of code examples so it should take only a few minutes to write. If the site you are trying to scrape has webservices, e.g. Amazon, etc, much better to use them than scraping.

[Megalo]

Dear fseipel

Thanks for quick reply I'll try the things you taught. It sounds awesome, and I never tried them before.

Thanks again!!

Are the the engineer who have done any work about web service transcation??