Jump to content

Extrating/Parsing data via webviewer


lsmall

This topic is 6105 days old. Please don't post here. Open a new topic instead.

Recommended Posts

I'm sure this has been discussed, but I can't find it anywhere. Is is possible to have a webpage load in the webviewer and have filemaker extract data into database fields? Is it the same as using a parsing calculation to extract data from a text field?

Thanks

Link to comment
Share on other sites

Yes, it is called "scraping", it's not hard to do but you should read the "terms of use" of any website that you plan to do it to and think hard about if you should. Here is an Example I have done and posted recently, there are some others on the forums as well.

Link to comment
Share on other sites

Screen scraping is also very "brittle" because any change to the code of the web page, even a minor one, will probably cause the process to fail. It's a high-maintenance solution.

Link to comment
Share on other sites

This is very, very, true. I can't even say how right Vaughan is. I am not saying "don't do it", I have and have saved the company I work for 100's if not 1000's of man hours by using it but, I am an In-House developer and I baby-sit a fairly large solution. I have access to it every day, and I know that I may have to rewrite my scripts at any time (or even need to abandon the use of it, and find another solution) . I have been "scraping" pages for a few years now (I used vbs scripts to do it before fm8.5 and the webviewer) and I have had to rewrite a couple of scripts because web pages changed. So, if you designing this for you or your company, I say go for it, the cost is right. If you are designing this for a client, think real hard if you want to do it this way. There are alternatives, you will most likely pay for it (or your client will) but are a lot less likely to break.

Edited by Guest
Link to comment
Share on other sites

Is this for an all Mac installation? If so there is an alternative using AppleScript and command line (via do shell script). The advantage is that it's usually much faster than Web Viewer. And, often more importantly, you can use command line tools to extract the data, which support regular expression pattern matching (think of it as wildcards on steroids).

FileMaker's text functions are very good, but command line is often better for quickly extracting lines and pieces of data from large loosely-structured text files (which html source is).

There is a learning curve, yes, but there is with FileMaker text extraction also, when it gets sticky. I guess it just depends on how sticky :B-]

Link to comment
Share on other sites

My experience is that using the webviewer is often quicker from an end users perspective but only because you can put the webviewer on a layout and have it load the page before the user requests the data (makes the process seem almost instantaneous). The down side is that capturing errors is a lot more difficult using the webviewer (and impossible to tell what kind of error occurred). Assuming you don't and won't have connection issues and the source code for requested pages will not change; using the webviewer may be the best option. I find it is easier to do and performs better than using any external solution.

In the example I linked to look at the script called "WebViewUSPSLookup". It is the script that "scrapes" the page.

Link to comment
Share on other sites

This topic is 6105 days old. Please don't post here. Open a new topic instead.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.