Grab Webpage html and paste into text field

[BobWeaver]

This sample is the result of a question posed in another thread. The requirement was to execute a Google search from a Filemaker database and insert the results (html source) into a text field.

This example uses a small java application which is executed via Perform Applescript (Mac) or Send Message (Windows) script step. The web page is downloaded into a text file which is then imported into the database. No web browser is required.

I tested it with FM6 running on OSX 10.3. Since I don't have a Windows computer, I was not able to configure the Send Message script step. However, I have provided instructions for doing this.

GrabWebPage.zip

[bruceR]

Doesn't work. The very first step fails, there is no such directory as /js

[BobWeaver]

You will have to create a JS directory, or just drag and drop the attached JS folder onto your hard drive. If that doesn't help, then maybe FM7 behaves differently from FM6.

Actually, whichever directory you use is not critical, but if you locate things in different places, you will have to modify the command line text to suit. In any event, the text file will be created in the same folder in which Grab.class resides.

[Lee Smith]

Got a message that it doesn't work for OS 9 and below.

[BobWeaver]

Sorry Lee. I just didn't have time to come up with the necessary applescript to run it on OS9. But, there is a spot in the script there to add the OS9 version of the Perform Applescript command if you or anyone else wants to figure it out.

The trouble is that the URL is passed to the java program as one of its command line parameters. That's very easy to do on OSX and Windows, but I don't know how it would be accomplished on OS9. Although I'm a long time Mac person, I learned all my Java programming on a PC. I played with Java only very briefly on OS9.

I suspect that this could all be done on OS9 without the Java program at all, using one of the scripting additions to directly download a web page to a textfile. I know such scripting additions exist on OSX. So, there may be an OS9 version. I'll have a look around to see if I can find anything.

[DTPerry]

you can use the SHELL extention (freeware) to execute shell commands on mac and windows.

then use CURL to put the source code into a field. then just calculate....

(Curl is a standard osX tool, but there is a windows version, somewhere)

to grab a webpage just put the url after the curl command, something like:

curl http://www.fmforums.com

with the -o option you can save an image on your harddisk and import this into a container.

info:

SHELL plugin

mac-> http://www.abstrakt.com/files/shell.sea.hqx

win-> http://www.abstrakt.com/files/shell.zip

CURL example

http://jesse.cnsplug-ins.com/

http://curl.haxx.se/

[BobWeaver]

Thanks for the info. I'm new to unix commands and wasn't familiar with "curl." That certainly simplifies things. In fact, it means that the java program and the plug-in that you referred to are both unnecessary in OSX. You can just use the Perform Applescript step to execute a "do shell script" command.

Unfortunately, there's still no solution for doing this on OS9.

[Fenton]

For OS 9 (also on X) wouldn't the URL Access Scripting application (in Scripting Additions) work?

[OttawaBusinessContacts.com]

I looked at this thing and it looks worth consideral skinned knees to get it working!

I down loaded and installed the shell plugin. I believe any Filemaker programer that has installed a plug in will succeed.

I downloaded curl. I wasn't sure where to install it but I think I did it right.

I found that I was missing libeay32.dll There were several windows versions.

Apparently the date of the .dll is critical I'll take another run at it tonight

The .dll I found is earlier than supposed to be but it may work. Apparently the one you want is dated 2003-04-02

There is a note that google doesn't like this thing because they want you to use the google API That means that you have to deal with cookies so they cannot tell the difference.

Ie. They want to charge for searches. I suggest that their effort may be as popular as a polecat at a perfume convention!

[BobWeaver]

Barry,

I think Google was mainly concerned with people doing automated searches to mine massive amounts of data from them for evil purposes. Apparently one internet based scam made use of an automated Google search in order to locate prospective victims.

I have tried my files and they will work with Google as-is. However, it is necessary for it to impersonate a web browser. So, to use curl you will probably have to use the -A option to set the user-agent parameter to something like mozilla or explorer. In any event, you should check that what you are doing falls within Google's terms and conditions.

One difference between using curl and Grab.java is that Grab inserts a reference number in the downloaded file which makes handshaking a bit simpler.

Fenton,

The URL Access Scripting application was probably what I was thinking about in my earlier reply to Lee, but I don't have access to anything running OS9 at the moment, and wasn't sure whether that application was standalone, or whether it worked in conjunction with a browser. The way I implemented this with the java program is a bit clunky, and was geared towards use with operating systems that can be controlled by a command line (Windows and OSX). It would likely be easier to come up with a completely different solution for use with OS9.

[OttawaBusinessContacts.com]

Perhaps this should be a seperate thread but the free shell plug in intrigues me. What it is is a direct pipeline into the OS.

You can use it to play solitaire sol.exe

or bring up the calcultor calc.exe as I recall

We should be able to bring up Word with any file unlike commnets or DDE Execute.

Can we run the sound or other software without a plug

We should be able to control Excel for batch faxing and batch email without

a plug in? Are we missing something here?

[Zero Tolerence]

I'm fairly new to Java, but I love this example. It works great on fp7 also. My only question is, How can I change it so its not google i'm searching? I assume your using the field as part of a URL to grab, lets say I want to do that with like amazon or something. Can I change it without having to rebuild your java?

[BobWeaver]

You don't need to modify the java program. The filemaker script sends the complete URL to the java program in the command line.

If you look at the globals layout in the filemaker file, there are several fields that contain the URL and other bits and pieces of OS command line which are concatenated into the overall shell script command or windows command line. You can modify these as necessary to give you what you need.

[Zero Tolerence]

Bob, I'm having problems getting this to work with FM7 (or 8) on the windows side. I assume its becasue there is no (Send Message) script step anymore and the Send Event doesn't work like the Send Message does, any ideas?

[BobWeaver]

I suggest using the following text in the template fields (without the quotes):

gWinTemplate1:

"cmd.exe /c c:java jsGrab "

Note the space at the end.

gWinTemplate2:

" http://www.google.ca/search?q="

Note the space at the beginning.

gWinTemplate3:

leave blank

Then the Send Event script step should be:

Send Event ("";"aevt";"odoc";cFullMessageWin)

I can't test this, since I don't have a windows machine, but I think this syntax should be close.

[Zero Tolerence]

I got this working bob, the problem was, that the output file has to be in the root of where you command prompt opens up to.

So If i did

C:JavajsGrab

I got an error, however, if I manually went to

C:Javajs

and then just did a java Grab command it worked great.

So, I copied my output file to just the C:

and sent the java C:JavajaGrab command and it worked.

This has been a life saver for me, thanks a lot.

[BobWeaver]

After some private messages back and forth with Zero Tolerance, I have updated the java program and the Filemaker example file so that you can download non-text files in addition to html and text. The example file now includes a script which downloads a weather map jpg file and inserts it into a container field.

I've also included the java source code file (grab.java) so that you can modify it to suit your needs.

GrabWebPage2.zip

[ino]

Bob,

I tray to use your solution (GrabWebPage2) in fp 7(converted) but it doesn't work. My interest is to grab domestic meteo info picture from domestic web page (http://www.dhmz.htnet.hr/vrijeme/web_analiza_h.jpg). Formally when I tested your Java script in Xcode (Mac OS x 10.4) script work OK.

When I do it thru your db GrabWebPage - db don’t do that. Only what I can do is to input picture which is in your folder. Db doesn't download the exact picture neither from wheter.com neither from my www info?

MapURL:

"do shell script ("cd /js; java Grab http://www.dhmz.htnet.hr/vrijeme/web_analiza_h.jpg"

&

" myMap2.jpg ")"

Where is my mistake? What I do wrong?

Ino

[Zero Tolerence]

first of all, is your Grab.class file at "Macintosh HD:js:Grab.class"?

if it is just do

"do shell script(" cd /js; java Grab "http://www.dhmz.htnet.hr/vrijeme/web_analiza_h.jpg" "myMap2.jpg" ")"

[ino]

:thankyou: