web security db and -find

[Garry Claridge]

Here what I have found using:

http://ip:port/FMPro?-db=mydb.fp5&-format=-raw&-findall

and "Web Security" as:

'All Users' : 'Browse'

A number of fields 'Don't Show'

The result is that all the field names are shown, however the data for those fields is not displayed.

Hope this is what others are experiencing!!!

Garry (nervous)

[Keith M. Davie]

Ok, I set the fields in events_.fp3 (this is the same db and format files I used in the Turansky thread, but in FMPro 4.0v3) to Browse and Don't Show in Web Security.fp3. After that I shut my Mac down for the night. Now I've restarted and rechecked my work over a fake LAN (flan), using address 198.0.0.1.

As I cruise through my browser I get a url to appear:

http://198.0.0.1/fmpro?-db=events_.fp3&-format=list.htm&expire=<=&expire=2/13/2002&-find

I edit that url to:

http://198.0.0.1/fmpro?-db=events_.fp3&-format=-raw&-findall

When I hit the return key (Mac), all the data in all the fields from all the records are displayed in the browser.

I will now go do the same in FMPro 5 and see if I get different results. If so, then the problem will exist only with 4.0. I'll report back soon.

[Keith M. Davie]

Ok. I tried this in Pro 5.0v3. Used the same permissions/restrictions as above (and same as Garry's) and again was able to display all the data in all the fields of all the records.

I don't know what I am missing.

[Keith M. Davie]

Regarding Anatoli's post of 2/12. He gives two links to FileMaker. The first (not a public site) does display the field names only. The second, "The link http://tidb.filemaker.com/ti/FMPro?-db=ti.fp5&-findall=&-format=-raw is also working.", reveals the field names and what appears to be all the data from all the records. Yet it also appears much as a Read Me for Pro 5.5.

"0 23 artnmbr title article datemade datemod product platform articleshort artsearch faxanscat relevancecalc relevancestatic wordsearchtext wordsearchcalc wordsearchextract articlebold articleboldOLD ftp_marker

ftp_mac ftp_pc article_count datemod_month_calc datemod_sort_cal ffffft15t6ffffffffffffffff ttttttttttttttttttttttt nccddcccccccccccccccnnn eeeeeeeeeeeeeeeeeeeeeee 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 107823 FileMaker Pro 5.5 Unlimited Readme Thank you for licensing a FileMaker product. March 05, 2001 This file contains additional information and known issues regarding FileMaker Pro 5.5: ..."

What am I missing?

[Keith M. Davie]

Regarding Anatoli's other link which was posted 2/12,

"The FileMaker site interrogated with http://prdb.filemaker.com/FMPro?-dbnames URL is sending to the browser:

0 ets fsa_stories poweredby etspr_login_temp etspr_login ets_Questionnaire_temp ets_Questionnaire contacts fm_newsletter web_survey xmlstory madewithfm resellers PlugInReg tirelevence ti contactsales customerassistance PRYourStory Evaluation Stories intl PReval Press_Edit PressExt PRCustStories PR feedback jobsext jobemail Web Fields_ Web Security ",

the thought occurs to me that it is possible that only field names and no data are showing because there is only one record and it is void of data.

[Keith M. Davie]

As I mentioned above, I've been trying things on a fake lan (flan).

Well I decided to try things with my website. Here is some of what I have found. The online experience is a bit different than the flan.

When I set the field username to "Don't Show" in one db, the username did not show in the force action. Unfortunately it did not show on the web page either.

Similarly, when I set some other fields to "Don't Show" certain finds did not work.

Conclusion: While it is possible to "hide" certain fields and their data, other fields cannot be so restricted. Other steps can be taken.

Fortunately I am using scripts. Because of that I can remove some data completely to db files which are not web enabled. Of the four db's which are web enabled, one contains no records. Two maintain one blank record, though more records can accumulate should the user quit the site, and those db's require regular housekeeping chores. The fourth db is basically a list of usernames and a bunch of numbers. Even though this data can be viewed by the force action, it is rather harmless.

If any of the experts out there are able to determine the number of db's which my site uses which are not web connected and what their file names are, I think we would all be interested.

Meanwhile, anyone interested can try these links:

http://www.simplifyfm.com:591/simply/fmpro?-db=signon_.fp3&-format=-raw&-findall

http://www.simplifyfm.com:591/simply/fmpro?-db=long_.fp3&-format=-raw&-findall

http://www.simplifyfm.com:591/simply/fmpro?-db=simply_.fp3&-format=-raw&-findall

http://www.simplifyfm.com:591/simply/fmpro?-db=show_.fp3&-format=-raw&-findall

Also, there has been mention of being able to run scripts once their names are known. Well I guess that if you use D-Base you will be able to get the script names in these four db files. If anyone can get a script to run from that information I think we would all like to know how you accomplished that.

[Keith M. Davie]

Thank you.

[Garry Claridge]

Keith,

I'm not sure if that was me or not. However, I used Charlies program to get the script names. Then I included '-script=copya' in the url.

I couldn't tell what it was doing, though.

Garry

[Anatoli]

Fortunately I am using scripts. Because of that I can remove some data completely to db files which are not web enabled. Of the four db's which are web enabled, one contains no records. Two maintain one blank record, though more records can accumulate should the user quit the site, and those db's require regular housekeeping chores. The fourth db is basically a list of usernames and a bunch of numbers. Even though this data can be viewed by the force action, it is rather harmless.

Keith, I know you are hard working man. But your scripts will not work on heavy load in single Unlimited -- the most used configuration.

It will not pass through my testing procedure regardless of any workarounds good or bad.

The testing is in IE:

Standard page is using -max 10-15.

Test page is using -max 50 and will display not only found records, but also chapter with 1-3k text and no caching is allowed at all.

On the bottom of result page there is JavaScript immediately firing another second test page with different search logic.

Second page is triggering third page, which is submitting page full of data to generate mew record in FileMaker database in size 30k.

After submit is done another, fourth page is called to display yet another search result.

You get quite load with this. And again loop to first request.

The same test is started in Navigator browser.

Everything is running on 100Mb Ethernet.

And then is starting another computer the same firing squad test with multiple browser requests.

The result is full load on FileMaker. Fast HD is singing under this load. Pages are blinking fast.

I doubt, that shifting data in and out with scripts will work for hours in this test, consistently and in average speed 6-9 request per second.

That is load.

[Keith M. Davie]

Probably was Garry.

Anatoli you wrote, "... your scripts will not work on heavy load in single Unlimited -- the most used configuration. It will not pass through my testing procedure regardless of any workarounds good or bad."

When you say "your scripts", are you running your test program on my site which contains my scripts? Or have you constructed your own scripts based on the example of the workaround which I sent you last April, and those are what you are testing? I am not quite following what you are doing.

[Keith M. Davie]

To elucidate a bit further on Garry's activity, you may receive a FileMaker message, or if you use a findall, you may get a browser force to appear. However, as the client you have know way of knowing if you have affected a db or run the script.

[Anatoli]

Under such load it is impossible to shift something from database to another database via script and back. This round trip will take longer time, than 3-5 replies from WC under that stress. So when the data will be there, already noting is valid any more in the database.

Sorry, scripts are not for web with heavy load.

Just from curiosity, how many of such "round trips" can computer manage per second?

And how many with heavy load from WC?

[Keith M. Davie]

Anatoli, I understand you are talking theory. Therefore I must assume that you have not tried to run a script which would handle the data load about which you posted.

In theory, what would you say is the maximium time length for a script and its subscripts to run under the load about which you are Speaking?

Less than 250 milliseconds

Less than 100 millisedonds

Less than 50 milliseconds

If less than 50 milliseconds, how many milliseconds?

Please notice that the scripts which I constructed for my demonstration were purposely designed in a cumbersome way in order to increase the time of their running such that the opportunity to create an error page is greatly enhanced for the curious Developer who is following the instructions which are contained in the website.

Also, it is obvious that the security issue of this thread needs further investigation. As a Developer, you can be assured that I will be striving to find an answer.

[Keith M. Davie]

"If anyone can get a script to run from that information I think we would all like to know how you accomplished that."

Ok. How did you accomplish that?

[Anatoli]

Even 50 milliseconds is long time. When you finish the transfer, and you release the thread, there is another 1-5 CDML requests waiting in cue and one will be picked as the next. Thus all results shown in FM are lost.

In FileMaker code is not single mechanism which will halt ALL request from WC plugin until another job is finished. Did you tried to work in FM Unlimited while connected to web with some load? That is simply impossible.

Yes, if there is some kind of flag, which I can set, and everything will stay in cue until I released this flag, then yes, we can do lot. In multi-user scenario it can be done with scripts because everything can be processed with scripts.

But not in WebCompanion unless EVERYTHING is running through scripts and not from CDML. In that case I will need 10 RAIC system to replace single Unlimited.

[Keith M. Davie]

Anatoli,

In the test you speak of "Test page is using -max 50 and will display not only found records, but also chapter with 1-3k text...". That is what is being displayed., and in fact your test is running at this heavy load to display search results.. The test you are running is not creating new records which are putting in large amounts of text and then displaying the results.

You also said, "I doubt, that shifting data in and out with scripts will work for hours in this test, consistently and in average speed 6-9 request per second."

About the concept of shifting data for hours, in the test you are performing don't you think a better test of capability would be to create a new record which includes 3K paragraph of text which record is then displayed in the results screen which as you are already doing? How many hours would that run at 6 - 9 requests per second, either before there was an overload or until FMP ran out of room? Search is one thing. New record creation and search is more demanding at heavy load.

Theoretically, at an average speed of 10 requests per second (faster still than your test) the request rate could be described to be one request every 100 milliseconds.

Now you suggest that 50 milliseconds is a long time for a script. But speaking in this theoretical way, that would suggest that 20 scripts of 50 milliseconds each could be handled in that second.. Far faster than the test you are running.

When done properly the data handling is being controlled by the script such that there is no data loss within the databases and even to the display page. And should a "collision" occur at ScriptMaker, the workaround ensures that the client(s) whose script does not run is appropriately informed and given a chance to resubmit.

I might point out once again that I have been to more than one website where I have encountered the need to resubmit a request because it was stated the load was heavy. And these sites are running main-frames.

I might point out that when one receives a message such as I have encountered that message has been hard-coded into the solution. The message may or may not be true. That is because the client does not need to know the actual source of the problem.

At my site, if you enter a name to register (hey try it with some made up names), the next page asks for you to enter your favorite password (so make up a few). But before you enter a password click the button. This will give you a page where you can (re-)submit the password. This is the same page you would get if there were a conflict at ScriptMaker. My comments on that page reflect this.

Within my solution, should one cause a conflict at ScriptMaker, there are a couple of different responses. It depends on where one is when the conflict occurs. In one scenario of a conflict the password is hard coded into the page and displayed with a button which allows the resubmit. All the client needs to do is click. Of course, one would be unlikely to handle a clients password in such a manner, but the example can be extended to handling data in other fields.

Anatoli (and others), please feel free to visit the site and try to cause a ScriptMaker conflict by making multiple submissions.

I might also point out that I earlier posted four threads on the cdml forum about Entertaining a script. I provided all the information necessary to create a simple db or two, and all the code for the format files. This is so that the Developer who does not understand the problem can create the files necessary to see the problem first hand. That demonstration also shows the design problem of running a ScriptMaker script which loops on the www.

Sure, setting up the db's etc. will take some time, but learning takes time. And if you are at all interested in the power of scripts you will need to know all this and more.

The workaround at my site looks quite simple when you click the button and get the results. It is extremely complex in its construction. The workaround affects the way things are done in very unusual ways.

[ February 16, 2002, 08:07 AM: Message edited by: Keith M. Davie ]

[Anatoli]

Read again:

Second page is triggering third page, which is submitting page full of data to generate mew record in FileMaker database in size 30k.

RE: But speaking in this theoretical way, that would suggest that 20 scripts of 50 milliseconds each could be handled in that second.. Far faster than the test you are running.

You didn't count the latency of FM. There are slight delays in between operations.

Furthermore, still not a single word from you about logic and there is the main problem:

When you finish the transfer, and you release the thread, there is another 1-5 CDML requests waiting in cue and one will be picked as the next. Thus all results shown in FM are lost.

In FileMaker code is not single mechanism which will halt ALL request from WC plugin until another job is finished. Did you tried to work in FM Unlimited while connected to web with some load? That is simply impossible.

Yes, if there is some kind of flag, which I can set, and everything will stay in cue until I released this flag, then yes, we can do lot. In multi-user scenario it can be done with scripts because everything can be processed with scripts.

But not in WebCompanion unless EVERYTHING is running through scripts and not from CDML. In that case I will need 10 RAIC system to replace single Unlimited.

[Keith M. Davie]

Gee Anatoli, I'm not using Unlimited. I'm using FMPro 4.0v3. You mean there is a problem with 5.0 that does not exist in 4.0 Some improvement, huh? Good ol' FMI.

Yes my scripts are causing the proper displays.

Please feel free to visit and test the site as described therein. You have not done that so far.

[Anatoli]

It is the same as in v. 4.

Ask someone to work through web and you try to work behind that same (web serving) FM directly.

If you run script to do something, then come web request producing another result and then your another script will continue with e.g. foreign found set, you'll end up nowhere.

No one can force web users to work in linear fashion. Someone will display his found set and immediately will progress. Another will display his result and will wait 10 minutes before progressing. You will need some kind of history tracing.

And still everything must run through single script from top to get data consistency. And if someone will make request and the script will cancel that request or the web request will cancel the script, everything will be in mess.

Good thing with WC is that in web work it is keeping the history and always will start and end single request without interference with another request.

Scripts no. They start and end, wait and always assume, that only single user is working with the solution. Found set will stay days until another user request.

I see 3 or more find requests with different found set per second. You cannot rely on that in scripts.

Again, to run script in split second, the only script in solution the same for all users works. I am using that because of bad date format in WC. But it is setting only the flag for today's date.

But to run any logic through scripts without history, no way.

PMFv0.11.zip

[Hammerton]

RETRACTION

My apologies. After further testing with D-Base and the url hacks posted on this list, it appears that IPNetSentry is protecting the FMserver machine as I had said, but the hacks can access the data on that server via the server/web-server-connector machine, just as long as they are not password protected via web security database. Password protected dbs return a signle code number "971" and no data. However, the presence of the password protected dbs are revealed by D-base. I will try to use the security features of webstar to block the url hacks.

[Keith M. Davie]

Anatoli, you are still talking theory.

Well, the fact is that you or anyone else can test your theory at my website. You or anyone else can make (or attempt to make) near-simultaneous calls on one or both scripts. One can also call other actions at the same time. All one needs to do is run two or more browsers on ones' 'puter at once, or coordinate with a few other people to make (near) simultaneous calls on a script. I've already tested the site quite well and am satisfied with the way the scripts run successfully.

The only issue is that of security, as has been discussed in this thread and elsewhere. And that security issue applies to all of us who are using FMPro through WC.

Still, you have not been through the site even once to use it, much less 6 to 9 requests in one second (your test). At some point your theory must meet the road. You say, in theory it can't be done. I say at my website it is being done.

If you can set up 9 or 10 browsers so that you can click on a Run Script button, then in theory, you can cause my scripts and workaround to fail. The scripts from those buttons are running at ~185 millisecond and ~245 millisecond when served at 500MHz. In your theory, the latter can be crashed with as few as 4 requests per second. I previously tested this over the web when served at 266 MHZ and the scripts were running in excess of 1000 milliseconds. In that testing requests were being made from 10 browsers.

Don't you think it is time you tested your theory now that you have the opportunity? http://simplifyfm.com:591/

[Anatoli]

I am not talking theory, no. I am programming solution every week or two. Without scripts. With scripts, which I've tested 3 years ago I cannot do that, it didn’t worked, it was slow, crashing, freezing, everything bad.

Then I did thorough research with my good friend and the conclusion was -- not, that is no way we can move ahead.

So we did think otherwise and -- voila -- everything works just great.

If you want use your way -- go ahead.

[ February 17, 2002, 04:17 PM: Message edited by: Anatoli ]

[Keith M. Davie]

"With scripts, which I've tested 3 years ago I cannot do that, it didn

[Anatoli]

quote:

---

Originally posted by Keith M. Davie:

"With scripts, which I've tested 3 years ago I cannot do that, it didn’t worked, it was slow, crashing, freezing, everything bad."

Okay, so three years ago you could not do that. BFD - and I don't mean Berkeley Fire Department.

One year ago I could not do that either.

So what? Today I can.

What is your point?

---

Nothing at all, but you are for big surprise in future...

[Keith M. Davie]

Big surprise? Like what? That FMI is going to go multi-threaded with ScriptMaker?

Well, that would make my workaround useless to that generation of users. But surprise, no.

Anatoli, do you like to go fishing? I do a lot of fly fishing. Just curious.

[Anatoli]

RE: Big surprise? Like what? That FMI is going to go multi-threaded with ScriptMaker?

Wouldn't help much. Did you tried to work on FM while interrogated from web?

[Keith M. Davie]

You inquire, "Did you tried to work on FM while interrogated from web?"

I have presented a workaround to the problem of near-simultaneous calls to run ScriptMaker scripts over the web. That is all. Nothing fancy.

Integrating that solution with an in-house solution is another matter which I do not address. I leave that solution to Developers such as yourself. There are too many different scenarios. It may be the best solution for the situation you now posit would be to serve the in-house solution via browsers as well.

Anatoli do you run scripts in your in-house solutions which are also served on the web? How do those in-house scripts affect the web for the clients interrogating from your site while those scripts are in use? How big is the in-house operation? What happens when two or more in-house employees call a script simultaneously, and how does that affect client using the web solution? Do you run loops to print invoices or anything?

Sure, scripts are not the answer for all solutions. But their power certainly has advantages when used intelligently.

One can always create scenarios where some course of action will be inappropriate, like a ScriptMaker loop. However, when there is a time that a course of action is appropriate through the web what will you do?

You've already concluded, "With scripts, which I've tested 3 years ago I cannot do that...".

Okay, maybe you can't figure out how to run scripts. For me it was not okay. So I figured out how to do it. After all, I'm a Developer. I can do that.

Last April you said it would not work. Well I finally have been able to put my web solution on-line and it is available for everyone to test. You think it does not work. You can test it yourself. Follow the instructions included in the site and make as many near-simultaneous calls on scripts from as many browsers as you can.. You can click on non-scripts as well if you want. My web solution will not give you the wrong results.

You can talk the talk.

I do walk the walk.

[Keith M. Davie]

For more about the security issues discussed in this thread including FMI's position as I understand it, see the posting on Web Companion / Security Loophole for today.

[ February 20, 2002, 09:01 AM: Message edited by: Keith M. Davie ]

[Anatoli]

quote:

---

Originally posted by Dr.J.

Security Problem Solved - (for MACs)

The following set-up thwarts both the impressive D-base program and the use of -raw, or any other URL packet data that can be used to execute hacks.

IPNetSentry on box A running a server and the web server connector (the folks at sustainable softworks will tell you how to configure it - ask for Brad)http://www.sustworks.com

IPNetSentry on a box B running FMPro. Set IPNetSentry to block all requests other than those coming from box A

---

Probably not.

WSC on A will send those hacking requests to B, so you are not blocking that.

BTW, we are now running our FM Security Filter 4th day and it looks like final solution. It is on PC NT and our programmer is saying, that MacX version is feasible, but not Mac OS.

If someone is interested in testing, contact me.

[Hammerton]

Security Problem Solved - (for MACs)

The following set-up thwarts both the impressive D-base program and the use of -raw, or any other URL packet data that can be used to execute hacks.

IPNetSentry on box A running a server and the web server connector (the folks at sustainable softworks will tell you how to configure it - ask for Brad)http://www.sustworks.com

IPNetSentry on a box B running FMPro. Set IPNetSentry to block all requests other than those coming from box A

[ February 21, 2002, 12:29 PM: Message edited by: Dr.J. ]

[Hammerton]

Anatoli - IPNetSentry on Box A (the server) runs more or less right at the ethernet card. It runs on the system, not the server. It does not let the request ever hit the server, and thus it cannot hit the web server connector. I have tested this pretty throughly, and as long as you know what code a hack might use to access the FM files, you can block the requests. IPNetSentry can also block all requests from that client for a specifiable amount of time.

I urge Mac users to look into this software.

http://www.sustworks.com

[Anatoli]

That will work, if it will not block any useful requests. I was wrongly focusing on IPNetSentry on a box B running FMPro. Set IPNetSentry to block all requests other than those coming from box A in your post.