getting data from the web

[databaser]

Is there any way to use a filemaker script to act as if you went to a webpage, hit ctrl + A (select all), ctrl + C (copy) and then paste into a text field in the database.

it's always seemed that i'd have to do the 'select all' and 'copy' manually.

seriously, i would love to know how to do this, even if it requires using something in addition to filemaker to make it work.

alternatives that might accomplish the same thing would also be appreciated

[comment]

See here:

http://fmforums.com/forum/showtopic.php?tid/199043/post/307564/#307564

[aholtzapfel]

Or this.

(Good example of "Scraping" a web page.)

[databaser]

I looked at those topics and tried everything there, and then tried to think of more ideas on my own, but nothing worked.

I would be willing to have filemaker execute some non-filemaker script, in between script steps, if that would make this possible.

the site i'm trying to "scrape" is like this

http://www.joincalifornia.com/candidate/13284

it would be so easy, because i could just have the script continuously add 1 to that number at the end of the URL, and grab every page, then i could have text calculations sort it all out.

I would be happy with any type of solution. it doesn't have to involve copy and paste.

i wish there were a way i could download all those pages into a folder, automatically. then i have a program that would convert them all from html files to text files, then i could import them. (i tried ftp, but couldn't get access.

is there a way to have filemaker save the page that's open in webviewer?)

but any solution at all would do.

[databaser]

does anyone know if i could use this thing -

new File( pathToFile ).getText();

with 360 works' scriptmaster, to do this

i confess, i've never used anything by 360 works before

[databaser]

I figured out how to accomplish what i wanted with this - GetLayoutObjectAttribute( "webviewer" ; "content" )

(I took a two-year hiatus from seriously using filemaker, so i guess i didn't know how to do simple 'new' things, like giving an object name to your webviewer)

So, I guess nevermind, unless anyone has already started authoring good insight on this stuff.

I appreciate the links to the other topics. thanks.

Edited November 3, 2008 by Guest

[comment]

I'm afraid I cannot help you with Windows automation. On a Mac, this would be relatively easy using Applescript, esp. since Applescript can return a result directly into Filemaker - something I believe VBS cannot do.

Perhaps you should consider using a plugin such as Troi URL or 360Works Textractor (I haven't tested any of these).

Another thought:

Parsing the raw HTML shouldn't be much more difficult (in this case) than parsing the rendered text. It seems that the pages folow a rigid common structure, so if you just save each page's source as a record, you can then figure out where each field's contents is and parse it out for all records at once.

Of course, the easiest way would be to contact the database owners and ask them to provide electronic access to the data. You should probably talk to them anyway, since what you describe sounds dangerously close to violating their copyright.

[Fitch]

Here's something to get you started (attached).

webparse.zip