how successful is separation model?

[mscholtz]

I've got a question for you pros out there.

If you've been using the 'separation model' for your apps, how successful would you say it's been?

Here's the context of asking this question:

I've used the SM in various apps, and most of the time I've ended up questioning whether it was worth it. It takes longer to develop, runs a little slower, and personally I've found it to be buggy in certain ways (see http://fmforums.com/forum/showtopic.php?tid/183087/).

Moreover, the professed benefit doesn't seem to actually come up very often. The whole reason to do SM is so that you can implement updates to the front-end without having to mess with the back-end. But since so many field calcs need to be in the back-end, and for me inevitably most any update ends up needing to change at least one field calc, I end up usually needing to update both halves. So there goes the benefit, right?

I'd be curious to hear whether others think it's been worth it for them.

Thanks!

[Ted S]

Put me in the "Good in theory, mediocre in practice" camp.

[Genx]

I develop apps in FM that get used over LAN and WAN... They also more often than not have a lot of eye candy to try and make them look like real apps.

I use a simple front end / backend split whereby the front end resides on the client and the backend on the server -- This way the only thing ever exchanged is data, no images etc. clog network traffic or delay layouts.

If you're developing small projects it probably wont be worth it, but in larger projects, it definitley is -- Plus i just prefer to split the data and the interface, seems to make more sense.

And of course...

http://fmforums.com/forum/showtopic.php?tid/170155/

[mscholtz]

Genx, I could imagine doing things that way if my apps were being served (as native FMP) over WAN. Given that's not the case, I personally would never do things the way you describe. Trading a little bit of speed, and the separation, for the need to distribute new clients out to every workstation when an update occurs? Ugh!

I'm not trying to talk you out of it, just not what I would do.

But I'm still hoping to hear from someone who has found consistent success in the separation model for its more traditionally touted benefit; i.e. has been consistently able to make needed changes to the front-end without having to add/change calcs in the back-end.

Thanks for the link to that other post, some helpful stuff in there.

Thanks, all!

[Brian C]

For smaller solutions, even those that I have developed myself I don't really think that the Separation model is well suited at all. It is faster to develop, and easier to just replace the entire file in most cases in just a few minutes.

For large solutions I highly recommend using the separation model.

It takes a LOT more time to develop but it is worth the piece of mind IMO.

Consider the following:

Initial Development Time

Possible Server Crash/Hardware Failure

Potential Corruption of Either GUI or Data

Data Recovery Time Due To Corruption

Developemt Time To Update

Time Required to push an update out to the client

IMO, It is far easier to push out a replacement GUI update and make some "minor" modifications to existing data file table calculation fields as needed.

The alternative without the separation model would be to push out an entire replacement file and have to re-import several gigabytes of data for countless hours over hundreds of tables.

Not my idea of a good time I have to tell you.

[Genx]

Given that's not the case, I personally would never do things the way you describe. Trading a little bit of speed, and the separation, for the need to distribute new clients out to every workstation when an update occurs? Ugh!

Besides the WAN benefit, the server also has to do a lot less processing in a lan environment. My FE happens to contain over 200 layouts, around 160 or so of which are purely GUI. This makes the file around 20mb in size.

I use an install package which is opened automatically for the user from a specific directory on their server held in the backend whenever their version is less than the current version.

I never do manual installs of anything on my client's pc's -- I'm paid to develop not waste my time with installs that they can run themselves : .

Don't get me wrong, if i'm building a single user app (which i rarley do except maybe for my own use) I'd probably never use the seperation model.

[Genx]

Also, chance of corruption is reduced -- If you have client copies installed, and one of your clients crash while performing scripts or on particular layouts or any thing like that, you can easily just have them "re-install" the program via the install package without the corruption disturbing anyone else.

If this happens to be a hosted version, you have to go there, close it down, find a recent backup, close down the front end and distrupt the entire office, restore the backup (and the data if you aren't using the separation model to any extent) to the backup, then open the file again.

Plus with the backend files, about the only corruption issue you can ever have is corrupt data (or in the worst case scenario the few relationships you might have for calc fields in the backend), layouts don't matter in most cases, nor do the very few scripts that might sit there and are executed vary rarely.

Imagine trying to locate a corruption issue in a single file you knew was corrupt? You have to look at every single relationship including ones used for scripts and the major ones in most cases -- the ones that relate to the way info is shown on layouts. Then there's the possibility of corrupt elements in the layouts themselves and the probable several hundred scripts you have laying around.

Plus backup sizes can be greatly reduced.

I only have to backup the backend of my clients files -- roughly 15-25mB for several hundred thousand records. Add the GUI in a single file scenario and that turns into around 50mB... And if you happen to backup to remote location, that really makes a difference too.

[IdealData]

Doesn't separation model make IWP virtually impossible? IWP relies on the interface and data being in the same file.

[Brian C]

I use IWP. It doesn't seem to care that my data and my gui are in completely separate files.

[Stuart Taylor]

I would love to work with a seperation model but the following things have always stopped me:

Label Printing

Complex (custom) Searching and Sorting

Managing User Accounts

Complex report printing

The resetting of Found Set Portals when switching layouts

Am i missing something. That would help make some of these issues easier?

[Genx]

Stuart, To tell you the truth i don't understand where your issue really lies... The main issue I could see is the Managing of User Accounts, but theres a few nifty tricks you can deal with here. If both files are on the server, you just generate an account with the same name password in the front end and backend -- When the user logs into the front end, access is automatically granted in the backend. If you use my scenario, where the client is on the clients computer, there are other work arounds for that.

Like i said, re the rest, they function the same as they would in a standard single file environment.

Mark -- Why Would it care?

Doesn't separation model make IWP virtually impossible? IWP relies on the interface and data being in the same file.

You just have to make a file reference in the front end to the external ip address of your server (remember it's evaluated on the client side).

P.s. I'm not trying to convert anyone, just trying to dispel the idea's that splitting your files is inherently bad.

[Søren Dyhr]

I use IWP. It doesn't seem to care that my data and my gui are in completely separate files.

Well I was puzzeled by his statement as well, and made a quick investigation of it. I downloaded:

http://www.newcenturydata.com/downloads/separation_demo.zip

...and found something I couldn't explain. So the question is how it the separation you use, differs from from the mentioned template that obviously can't work under IWP??

--sd

[Brian C]

IdealData: You may need to check your network access settings on your external data files and your iwp settings on those files as well. This is usually something that gets overlooked. If these are not set correctly you will see layouts but the links will be broken because the databases will not be open due to permission issues.

[Brian C]

StuartT:

I am not sure what issues you can have with printing or searching/sorting... perhaps you can expand on that a bit.

As for portals being reset - If the user actually clicked on a portal row - you can store that row number in a $$ variable and use it to return to that row after the layout switch occurs.

As for account management in the separation model:

The only situation that requires you to maintain unique accounts at the file level is if access to the layouts is required.

Since all of your layouts are usually in the GUI file, this is where you keep your unique accounts. For the remainder of your data files, you only need to create a default account to be used on opening each of the files that allows record creation/editing/etc since that is all that is needed by the non-developer users.

The only file beyond the gui I allow users to visit is my print file - and that file only allows searching and printing so no accounts are needed there because no modification of the data occurs there.

Developers that see a need to use multiple GUI files so that they can make their solution more modular generally use role based accounts instead of unique user accounts and use a re-login command to drop the user into the proper role based account - which the rest of your files can then already have in common.

Any additional info on the specific user can be stored in global variables and placed in global fields if there is a need to print this information on a layout.

For myself, I keep individual user accounts at the gui level and use scripts to handle account creation, login, etc.

For additional gui files it is not such an insurmountable task to script creation of accounts and password changes when you only need to deal with the gui files only.

[Søren Dyhr]

The resetting of Found Set Portals when switching layouts

Please explain further, neither do I get it?:

--sd

[Søren Dyhr]

As for portals being reset - If the user actually clicked on a portal row - you can store that row number in a $$ variable and use it to return to that row after the layout switch occurs.

Scott Love et gang, says it's bad practice - should instead be done via:

http://www.filemaker.com/help/Script-Steps4.html

... I know it then takes two scripts a smallish one which calls the other via a script parameter, and then a line to return in. This statement isn't particular substantiated in the the book, but it follows common rejection of global variables - found elsewhere:

http://en.wikipedia.org/wiki/Global_variable

--sd

[Genx]

Think about it, File References are evaluated on the client side...

If your file reference says file:myBackend.fp7 -- of course fm won't find it. Just like you have to type 192.168.111.10:591 into the browser to get to your front end, you have to provide the full network reference to be evaluated from the client side i.e. fmnet:/192.168.111.10/myBackend.fp7

[Brian C]

After downloading and reviewing the sample files for the separation model provided in Soren's post - they will not work under IWP unless you fix the IWP Network settings on each of the files.

[IdealData]

Regarding IWP...

I thought one of the key features of the separation model was to deploy the UI at the client - thereby reducing the network traffic from the server.

If the UI is located on the client then IWP would not have the UI to deliver to web clients - hence my previous comment.

[Genx]

No, your previous point said that it was impossible...

The separation model only specifies that certain portions of the database are separated in some fashion -- In most cases Data and Interface. There are plenty of other benefits by just having that split besides the speed extras if the client is on the users PC.

[mscholtz]

Thanks, everyone. This has been very eye-opening.

Most of the considerations brought up here (IWP, ease of backup, user accounts, worries about locating corruption, etc.) don't really apply to me or concern me. But I'm glad they're on the table for the community at large.

Brian C, I think you have made the most compelling case for success with the SM. And I definitely hear you that if you have a large amount of data, you never want to have to do a full-fledged import migration. Been there, never want to do that again.

My projects (including my current one) tend do be "large" in terms of table and UI complexity, but aren't destined to contain GB's worth of data. They're fairly highly-engineered, but not high-volume. So on one level the SM isn't as essential a time-saver for me as it sounds like it is for you. Server crashes don't happen often (knock wood) and it wouldn't take me too long to recover from one.

But I am still curious, for you and Genx and anyone else out there who have done a lot of SM apps: have you ever experienced any problems that seem to relate to SM? Have you seen any bugs such as the one in the link in the original post above, or this one: http://fmforums.com/forum/showtopic.php?tid/169734/post/175338/hl// which I suspect has to do with SM as well?

Do you see any drawbacks to it, other than the extra development time?

[Genx]

Bugs -- Specific to the seperation model? Not that i've ever seen, except maybe if your in define relationships and want to add a TO from a referenced file, the Tables are listed in creation order not the order you've specified -- Thats about it.

"large" in terms of table and UI complexity

How quickly can you find something in your relationships graph though? If i know i have to redefine a calc i go to my backend, open up define database and instantly have access to every relationship who's only real purpose is the definition of calculations.

With the FE / BE split the definition is very clear. The backend is for data, the front end is for interface, reports etc. Your scripts, relationships, layouts are much more easily found and modified.

Edited February 10, 2007 by Guest

[BruceJ]

My problem with the separation model..

Yes it can be done.

However,FMPro was designed for ease of use - it's one of the main selling features.

You start acting as if it's an Oracle database, then maybe you should start using an Oracle database.

My point being, if you are in a situation of creating an enterprise wide, mission-critical system, then you ought not be using a workgroup database application.

FileMaker is a great tool. You just have to be sure that the square peg goes in the square hole and the round one in the round hole.

If FileMaker comes out with a developer application that works in the three file system (Interface, Business Rules and Data) then I'm sold - otherwise - I'm sticking to the "stuck together" file system.

The original Claris FileMaker nerds would be pulling off their pocket protectors if they heard you guys trying to make their database more complicated!

[Genx]

If they didn't want their database becoming complicated, file references wouldn't exist.

File references allow a group of file to act as one... so there's not really any issue in over complicating anything there.

[bme]

Thank you for an interesting thread.

I´m new to post-FM6 development however, and quite pressed to get a solution running without to much special considerations. If I´m to deploy the SM I would like to get it right from the start.

What is your opininon on developing a single file solution to start with, and let that file be the data layer, for when I eventually get the time and knowledge to level up to the SM?

Is this viable? What are the considerations regarding table and layouts?

I figure I could make use of a pure calculation table for example. Any ideas greatly appreciated!

[Genx]

Well... it's viable, but not necessary...

There's nothing wrong with developing in a Single File but if you start in the single file, a lot of your presentation layer Table Occurances, relationships and scripts are likely to be in your data file -- It'll be too hard to separate and reconstruct so it probably wont be worth it.

There seriously isn't anything wrong with single file development so don't stress!

[Fenton]

Even with the Separation Model, one still has layouts in the Data file, usually what I think of as "developer" layouts. And it has relationships, if only to provide lookups, auto-enter and relational calculations (yes, I put calculations in the Data file, in a simple SM). So one can begin with the Data file, for tables, fields and basic data relationships.

I think one should begin this way. I've seen too many FileMaker files which looked like they had a finished interface, but whose underlying structure was not quite what it should be. It's a mistake to try and complete the interface before you've got your basic relational structure.

But once you got that, and you begin on the interface file, by creating the table occurrences of the Data file on its Relationship Graph, and tying them together as needed for navigation, etc., you're pretty much in the interface file almost all of the time. I find that the SM method (especially a 2-file system) is so transparent that you have to remember to switch files to go into Define Fields.

Which model is "best"? That's impossible to say without taking into consideration the entire development environment, current and future. But neither is difficult.

[igotit]

Hi Everyone,

I've read this thread with great interest and hope you can help me with a question. What process would you follow to take an existing solution built with FMP 8.5 Advanced and convert it to the separation model?

As I understand it, the separation model is more difficult to build, but it seems to me is well worth the effort from what I'm reading here.

Can you tell me the ABC's of how to make the conversion, or maybe tell me where I can find the information?

Thank you all for your help, this thread has been very enlightening.

Milo

Edited May 11, 2007 by Guest

[Fenton]

First create a new Data file, before you start on the Interface file. Copy/paste tables from the old file(s). Rebuild a very basic Relationship Graph,* only what you need, including those for calculation fields using relationships, and lookups. Go through the fields and fix those; the calcs should just need the /*comments*/ removed; the lookups retargeted.

If you have a lot of the above, you probably should create the tables manually. Just add the basic ID fields to start with. Then you can build the basic relationship graph BEFORE copy/pasting the rest of the fields for each table. If you're careful all the calculations and lookups should come in OK. Exact naming of table occurrences is critical.

You don't really need any tables in the "interface" file, but you might want one or more for temporary globals, interface elements. The main thing you need to do is to rebuild your relationship graph, in the interface file, to be more or less the same as the one you've got now in the original file. Make sure you name the table occurrences exactly the same.

If you have mulitple files in your existing solution, then you really need to think hard about whether to only have 1 interface file or not. Same with the data file. Or whether use a (very) few "module" files. That gets more complex, though it's essentially the same; navigation scripts are affected more than anything else. I won't go into detail now; we'll assume for now 1 interface file, 1 data file.

Then you should be able to create layouts in the new file, then copy/paste layouts from the original file; the fields should line up; name the layouts exactly the same also. Next you've got scripts, which can also copy paste, or even Import. But this would be a great time to rewrite awkward scripts. Any "cross-file" scripts, such as those left over from a 6 conversion should be rewritten.

You would also need to rebuild your Accounts & Privileges, in the new Interface file. Though the privileges would be quite different, mostly to do with access to layouts and scripts rather than data. If you have many accounts you should consider building a table and mechanism to create and delete accounts, and assign privileges. It would be run from the data file (to store the account names), but do the above in both files.

At some point above you'd want to import some data from the old file. If there's a lot I'd just bring in a little, for testing. Hundreds of thousands of records are not needed to see if it basically works; it just slows you down.

*Use the anchor-buoy table occurrence "group" method, right from the start. Use a good naming convention. There's a comprehensive document at FileMaker on naming conventions, and several sites with more material on the anchor-buoy structure.

Lastly, ask yourself why you want to do all this work :-? What problems will it solve? You may still need to do imports from the data file into a clone on rare occasions, Server machine crashes, etc.. If you also include a comprehensive script to handle that you should be in pretty good shape.

[Ender]

What process would you follow to take an existing solution built with FMP 8.5 Advanced and convert it to the separation model?

It's not too hard to split an existing file into Interface and Data. See here:

http://www.fmforums.com/forum/showpost.php?post/123239/

[Fenton]

Yeah, clone the file. Duh :-]. Or take 10 times as long and do it my way. I guess when I did this I was not just "separating" but was also condensing many FileMaker 6 created files into one. So I could only get some of it from any one file. I think I did clone the file.

[igotit]

A friend of mine works for a large ad agency in Chicago and they recently replaced an FMP 8 solution that was built on the separation model. He said they replaced it for a lot of reasons but one is that it was so slow over the server.

Harry Glosenger, who Ender knows very well, built the new solution. As a matter of fact it was the solution Harry featured on the FileMaker Cafe called “Under Glass”.

Anyway, He also built it on the separation model and speed is no longer an issue. I guess my question is why?

I know it can be for a number of reasons and I remember a conversation on the Cafe where Harry was talking about the separation model and warned that it is not for the faint hearted. He said unlike other non-separation solutions where you can make a few mistakes and survive the life. With the separation model, you either do it right or it’s wrong!

My concern as an advanced novice, if there is such a thing, am I getting into something that will bite me?

Thanks everyone the is a very interesting conversation.

Milo

[Fenton]

I don't see that the Separation model is so much more "unforgiving" than a regular solution. If you do things wrong in regular FileMaker file it ain't gonna work so well either.

I think it becomes a little more difficult if you have "modules", i.e., several interface files. Mostly this shows up in navigation using Go To Related Record. Because the relationships (TOs) are actually to instances of the Data file, you cannot target another "interface" file with Go To Related Record. It works transparently within 1 Interface file, not between 2.

The work-around, which is not very difficult, is to set a global field in the other interface file to the key, go the other interface file, do the GTRR using the global's relationship, then complete the navigation there. And I say "navigation," because the above only matters if you GTRR and expect to end up on another file's layout. If you're just using GTRR to process data, then you just put a TO on the current interface file's graph, create a hidden "developer" layout, do what you need to do, and return to a user layout.

But you should rarely have to do this. The whole idea of only a few modules is to put all tables that interact together in that way in one interface file. And, as I said, you won't see this problem at all if there's only 1 interface file.

Another small glitch, with Separation model, is when you absolutely have to run a script with Full Access privileges that modifies data. That script setting is file-specific, so it will not work on data when run in the Interface file. The solution is to create a small subscript to do only that part in the Data file. Or find another way to do it that does not require Full Access.

About the only time I've had to do this is with the Copy All Records step, which cannot be run by people who have no Export privileges. An alternative is to use a Loop; but that's slower. A newer alternative, in 8.5, would be to use the List() function, which I think is pretty fast.

Sometimes you may run into small irritations, which require you to build an "extra" relationship for a calculation or lookup to evaluate in the Data file. So you just do it.

In my opinion, whether or not the Separation model is a worth it or not has more to do with what your access to the solution is, how much development you're going to do on it, etc.. If you are an in-house developer, with easy access at all times to the solution, and you are only doing small tweaks, and you always do everything right the first time :-], then it is not so much needed. If however you need to do serious re-development for a complex live solution, which you do not have such easy access to, which has a lot of data, then the Separation model really begins to pay off. You have more flexibility.

[Stuart Taylor]

Hello,

Had forgotton about this Topic.

I still have not been able to make and test one. This is an expansion on some questions raised quite some time ago.

• In my current solution i have an address table. I use this for bulk label printing. I assume in the seperation model my tables are all in the backend. So i want to print my 2000 labels, do i have my print layouts setup in the backend file, if not how do i efficiently print them?

• In my current solution i have an invoice table and line items table. When i print the invoice i do it by bringing up the print layout in the line items table ... whats the best way to approach this.

• My users are used to complex searches, in the seperation model am i right in thinking that the data is all displayed via portals and relationships. searching is kind of redundant?

best

Stuart

[Fenton]

No. You place table occurrences of the data tables in the Interface file. You create layouts based on them. For almost all things, everything appears and works as if it was the actual data file. None of things you mention would be a problem at all.

Here is the world's simplest Separation Model solution :-] You can hardly tell them apart, except one has a table and data and the other doesn't.

Separation_fej.zip