Extrating/Parsing data via webviewer

[lsmall]

I'm sure this has been discussed, but I can't find it anywhere. Is is possible to have a webpage load in the webviewer and have filemaker extract data into database fields? Is it the same as using a parsing calculation to extract data from a text field?

Thanks

[aholtzapfel]

Yes, it is called "scraping", it's not hard to do but you should read the "terms of use" of any website that you plan to do it to and think hard about if you should. Here is an Example I have done and posted recently, there are some others on the forums as well.

[Vaughan]

Screen scraping is also very "brittle" because any change to the code of the web page, even a minor one, will probably cause the process to fail. It's a high-maintenance solution.

[aholtzapfel]

This is very, very, true. I can't even say how right Vaughan is. I am not saying "don't do it", I have and have saved the company I work for 100's if not 1000's of man hours by using it but, I am an In-House developer and I baby-sit a fairly large solution. I have access to it every day, and I know that I may have to rewrite my scripts at any time (or even need to abandon the use of it, and find another solution) . I have been "scraping" pages for a few years now (I used vbs scripts to do it before fm8.5 and the webviewer) and I have had to rewrite a couple of scripts because web pages changed. So, if you designing this for you or your company, I say go for it, the cost is right. If you are designing this for a client, think real hard if you want to do it this way. There are alternatives, you will most likely pay for it (or your client will) but are a lot less likely to break.

Edited October 6, 2007 by Guest

[lsmall]

I will be using it to get data from an internal web-based medical lab system, so no "terms of use" issues.

Is that the correct example that is posted?

Thanks for the information.

[Fenton]

Is this for an all Mac installation? If so there is an alternative using AppleScript and command line (via do shell script). The advantage is that it's usually much faster than Web Viewer. And, often more importantly, you can use command line tools to extract the data, which support regular expression pattern matching (think of it as wildcards on steroids).

FileMaker's text functions are very good, but command line is often better for quickly extracting lines and pieces of data from large loosely-structured text files (which html source is).

There is a learning curve, yes, but there is with FileMaker text extraction also, when it gets sticky. I guess it just depends on how sticky -]

[aholtzapfel]

My experience is that using the webviewer is often quicker from an end users perspective but only because you can put the webviewer on a layout and have it load the page before the user requests the data (makes the process seem almost instantaneous). The down side is that capturing errors is a lot more difficult using the webviewer (and impossible to tell what kind of error occurred). Assuming you don't and won't have connection issues and the source code for requested pages will not change; using the webviewer may be the best option. I find it is easier to do and performs better than using any external solution.

In the example I linked to look at the script called "WebViewUSPSLookup". It is the script that "scrapes" the page.