How to get the "Submit" URL format?

[SoCalMacDude]

I need to query a web site by entering an account number, then hitting Submit. I assume that I need to intercept the HTML query as it submits in order to build my URL for Web Viewer.

How is this normally done? Is there a utility for MacOSX that will help, or can it be gleaned from View Source on the HTML page (form data is there)?

When the page returns, how do I scrape the data to populate my database with the results? PatternCount on the object?

[SoCalMacDude]

WOW! After 15 days and 63 views, no one can offer a hint as to how to submit or read data to/from a web page using viewer?

OK, here's the site:

https://efile.boe.ca.gov/boewebservices/verification.jsp?action=SALES

Try a number like this:

17678299

Want me to upload my database?

[Genx]

FMWebSchool will be running an article on the submission side of this in this month's newsletter.

[Lee Smith]

Genx mention one source.

In order to do this, you will have to do some research, and some work. It isn't an automatic thing.

Start by joining ISO FileMaker Magazine, at

http://www.filemakermagazine.com/

Matt Petrowsky has written a couple of articles on this subject, and linked at least two plucins maker sites.

I used some of the CFs at Brian Dunnings site to parse out data, there is also a CF or two posted in our CF Topic that will help.

Good Luck

Lee

[kiwisilva]

Hi SoCalMacDude,

You should be able to accomplish this without a web viewer with TCPDirect plugin (www.fusionplugins.com/tcpdirect). It will allow you to send http request headers built up from filemaker data to a web server, and receive the response into filemaker where you will then be able to parse out the data you want using the various text functions.

In order to do this you will also require another application which will allow you to view outgoing / incoming tcp packets on your network interface. If you are using a mac then check out the command line tool tcpflow (http://www.circlemud.org/~jelson/software/tcpflow/) which will display header information and received header/data on an interface you specify.

So in other words what you would do is run tcpflow or an equivalent program, go into your web browser as usual and go to the URL, enter your code, hit submit etc, but check out everything thats happening underneath the hood, what the headers being sent/received are, and contain etc.

Then using TCPdirect, rebuild the same thing by sending the same header, but only this time the information you receive will come into FileMaker, from there you can send more tcp requests, or parse out the data you require.

The end result is quite elegant for a Filemaker database, no need for a web viewer, just a simple filemaker field where you enter your code, and a filemaker button attached to a script which handles the plugin function calls, and the data parsing.

[Genx]

You know... that really doesn't sound all that straight forward to me... And I don't see the advantage of using the plugin over the web viewer, from what I can see they achieve exactly the same thing.

Edit: Additionally, no one really said the web viewer had to be visible, and i can't be sure but you may or may not have been implying that you can't extract the content's of the web viewer.

Edited October 20, 2007 by Guest

[SoCalMacDude]

Thanks, guys. I really appreciate the input.

I was at an FMPug meeting when 8.5 came out, and several Filemaker Inc. reps were there, and they had said this was possible with web viewer. So I didn't realize it was such an involved process; I thought it was a feature!!

(Sorry for the slow response, I thought I was subscribed to this topic via instant email - now I am!)

[Genx]

Just go subscribe to the fmwebschool newsletter. You may be pleasantly surprised with its content.

[SoCalMacDude]

Actually, I did subscribe on the 22nd.

I chose "FMWebschool Newsletter "Confessions of a Webaholic" on the check list, but haven't received the newsletter yet. What do I need to do to read it?

Thanks

[SoCalMacDude]

I just got a reply that the newsletter will not be out until October 31st.

Can you provide any hints to help me make this project happen now?

Thanks

[_henry_]

Hello,

The URL for the Query of the website (for the number - 17678299) will be:

https://efile.boe.ca.gov/boewebservices/verification_results.jsp?account=17678299

You can change the number to the field name.

Moreover, for scraping the data, I found similar help in this topic:

http://fmforums.com/forum/showtopic.php?tid/190411/

[Genx]

Henry, that would only work if the result page expected a form submitted with the GET method - in this case if you look at the HTML you will notice it expects POST.

[SoCalMacDude]

Thanks for the attempts, guys!

Luckily I'm on a Mac, so I solved it by using Quick Keys to drop the cursor into the field (using pixel coordinates from top left of page), pasting the number in, then clicking the coordinates of the submit button. The quickeys script is triggered via Applescript. Looping it through, works like a charm! Gotta love Macs!!

Still, I would like to see a solution that doesn't involve purchasing third party software!

Thanks,

SoCalMacDude

[Zero Tolerence]

Is there a reason a simple html form pointing to their website won't work? Just use javascript to submit the form after it loads, call the (local) web page in your web viewer.

I'm not savvy with https, but I don't see why that wouldn't work. You'd have to make sure you send all the right POST params, (like their little radio button thingy) but it should work.

[Genx]

Is there a reason a simple html form pointing to their website won't work? Just use javascript to submit the form after it loads, call the (local) web page in your web viewer.

Yes that would work fine - though the point of the article in question was to provide 1 file that would be suitable for all forms, and additionally not force the user to manually enter data into a form when it is already available in FM.

This links submits post data to https://efile.boe.ca.gov/boewebservices/servlet/BOEVerification

Gives type a value of SALES

Gives account a value of 17678300

http://www.realevo.com/downloads/examples/get2post/get2post.html?actionURL=https://efile.boe.ca.gov/boewebservices/servlet/BOEVerification&type=SALES&account=17678300

I'm not savvy with https, but I don't see why that wouldn't work. You'd have to make sure you send all the right POST params, (like their little radio button thingy) but it should work.

Actually you wouldn't need to worry about the actual format of the inputs - these are just for the end user and don't exist to any meaningfull extent in the Post arrays on the other side:

would be the same as:

Test

The sample file and article are here

The converter file is attached as a txt file - rename to have an html extension.

get2post.txt

[_henry_]

Thank you Alex for sharing the Webviewer. It will definitely help us.

Well, I got a little error page shown below. I tried to change the URL on the Webviewer from "http://www.realevo.com/downloads/examples/get2post/get2Post.html" to "http://www.realevo.com/downloads/examples/get2post/get2post.html" .

If you can notice the small difference on the get2Post (1st one) and get2post (2nd one).

It's only small glitch.

After you change to the small "p", then the WebViewer on FM would work perfectly great!

[Genx]

I've added another copy of the file as get2Post.html - recently moved servers and seem to have lost some of my files. Anyway, it should work with either get2post.html or get2Post.html.

The file doesn't have to be used remotely though - its just an html file so it can be used locally - i.e. you could place it in the same folder as your FM file, or somewhere on your network and determine the appropriate file path using Get(FilePath) etc.

[_henry_]

The file doesn't have to be used remotely though - its just an html file so it can be used locally - i.e. you could place it in the same folder as your FM file, or somewhere on your network and determine the appropriate file path using Get(FilePath) etc.

Can you please show on the usage for locally used?

Thanks

[Genx]

Attached should work, though URL's are always a little iffy with different browsers so i'd appreciate if someone could test this on Mac.

Essentially it just calculates a local file path based on the assumption that you will dump the get2Post.html file in the same directory as your FM File.

FormSubmission_Local.zip

[SoCalMacDude]

Wow, Genx, now that's cool!!

I'll have to play with it a bit on my server.

Thanks!

SoCalMacDude

[Fenton]

It doesn't quite work on a Mac, because of the "Macintosh HD" in the path. I tweaked your calculation a little to remove it on Macs. I think it'll still work on Windows.

Let( [

y = Right( Get(FilePath) ; Length(Get(FilePath)) - 6 );

drive = Case ( Abs ( Get (SystemPlatform)) = 1; Get ( SystemDrive ))

];

Substitute (

"file:///" & Left( y ; Position ( y ; "/" ; 1 ; PatternCount(y ; "/" ) ) ) & "get2Post.html";

drive; "/")

)

[P.S. I notice that the POST method works even if there are spaces in the 1st file path. At least it seems to. Or does the JavaScript escape them?]

Edited December 6, 2007 by Guest

[Genx]

Thanks for the path fix.

Not 100% sure I get the question - are you referring to the path to the get2Post.html file?

In any case, all browsers (at least the 6 i have on my PC: IE7, IE6, Netscape 9, Opera, Safari and FF2) encode white space as %20 automatically - the entirety of the url which is:

file:///path Here/get2Post.html?blah=blah test&name=frank williams

would become: 

file:///path%20Here/get2Post.html?blah=blah%20test&name=frank%20williams

Given that the web viewer uses the OS browser - with Windows IE and with Mac i think it uses Safari - the same should hold true for any url you submit to it.

Javascript in turn interprets %20 as white space.

But again, i'm not sure if that's what you were asking.

Edited December 7, 2007 by Guest

[_henry_]

Hi Alex,

I want to know if this method would work on IWP inside the webviewer? If yes, is there a format to use on IWP inside the webviewer?

Thank you.

[Genx]

It should work fine, you'd just have to change the location of the file to somewhere accessible by the end user... other than that the url format should be exactly the same.

[_henry_]

Thanks for your prompt response... :