Import Mapping bug

[LaRetta]

This is an obvious bug. A very simple import between two tables. Then if you add a field into the target table, FileMaker MAPS from a random source field to it! REPEAT ... FileMaker decides to ADD a field to your map. Everyone? If you EVER script an import and expect your map to hold, you should take a moment to review this file.

We have been told that, once we create an import and finish designing a solution, we should NOT be adding any more fields or changing anything!! I am sorry, FileMaker, but that is unrealistic; we need to be able to add fields into a table without FileMaker deciding that all imports into that target table should suddenly map to it ... FileMaker solutions are always an ongoing process!

There are many other inconsistent breaks with import mapping; this is only the most obvious, serious and ridiculous. FileMaker, please pay more attention to your import mapping problems. You stress how we can interact with outside sources but we don't dare ... it is too dangerous because our field data can be overwritten at random.

I am posting this on FileMaker's bug website but I wanted to provide a file here for them to review as well.

ImportMappingBug.zip

[bcooney]

Here, here, LaRetta! (From someone who has also spent hours of her life dealing with the very fragile Import Script step).

[mr_vodka]

I hate FM's import maps...

[LaRetta]

Yeah, fragile is one thing ... having FM decide to add to the map simply because you add a field into target is another. What it means is that we can NEVER do anything in any table without opening EVERY SINGLE import from ANY other table which imports into it ... and recheck the maps.

I believe that incorrect data is writing from many source tables to targets and CHANGING important data and people don't know it because this issue isn't visible and it will only happen 'at random' when an import is ran.

What does it take to get attention from FileMaker? There is no excuse for this specific map break; some of the others, one could debate but not this one!! BUG BUG BUG

[Raybaudi]

Hi LaRetta

"Examine the import map again - the new field has been added to the import map!"

... but why not ?

There are new field/s in the Target Table now and for me it is right that they show up.

But your last import order is saved.

So, what's the problem ?

[LaRetta]

But why not?

Just because I create a field in Target doesn't mean I want to import a random field from source into it! If I wanted it to map, I should say, "Well, I've added a field in Target and now that source import needs to also pull in that value" and *I* should map it myself!!

There are new field/s in the Target Table now and for me it is right that they show up.

Show up? Of course they should show up in the map list of fields for the target side but that does NOT mean that a random date field should have mapped to the text field.

So, what's the problem ?

... because I don't want to import a random date from source into a field I create in Target, maybe?? Maybe that field in target was a global I created to accept User input after the import. And now, I modify the script to start with Custom Dialog to accept the User input then the import runs and ... guess what ... my user global data is overwritten by a date from a Source table!!

I don't know how better to explain it than that. IT IS WRONG. PERIOD.

[Raybaudi]

Just because I create a field in Target doesn't mean I want to import a random field from source into it!

Maybeit is me, but I do not see any random import !

FileMaker will import based on the last order. ( so, if you choosed to import ONLY Text and Number, it will import ONLY them )

[LaRetta]

Did you do as the instructions said to do? Did you create a new field (such as text) in target and then look at the map? If your original source had a date in that date field, after you add the text field, that date will IMPORT into the new text field.

[mr_vodka]

:iagree:

I can not agree more...

It absolutely makes no sense for it to automatically assign a field to map from the source to a newly added target field...

It bad enough that import maps break when moving fields around on the source side... But it absolutely stinks that it decides to add its own mapping when adding a field to the target.

:goodpost:

[Raybaudi]

Did you do as the instructions said to do?

Yes, I did.

But now, after new trys, I can confirm that it is a bug ( even if a random bug ).

Random bug because, if you add a new field after correcting the first wrong import, the new import is indeed correct.

[LaRetta]

If you have more fields on the Source side and you add another field to Target, it will break again by mapping another randon source field to the new target field. It is NOT random at all, unfortunately.

[comment]

I agree.

I can sympathize - to some extent - with the issues caused by changing the structure on the source side. After all, the source could be an external file, even a text file - and it may not be easy to assign some sort of ID to its fields.

However, this does in no way apply to the target side, and I can see no excuse for Filemaker spontaneously enabling a target field for import.

[viscous]

I have to chime in and say that this has been driving me nuts lately too. I have a nasty table with a bunch of ever-increasing fields and about 5 or 6 different import scripts that get screwy each time a change occurs. FM definitely needs to get a handle on it and map the fields in each database by the internal field ID's somehow and keep it that way.

[fabriceN]

Hi LaRetta, I agree this is one of the many annoying bugs with custom import orders.

I published an alternative for FileMaker 10 here (field map).

At first I thought it was a bit overdone, but after all I find myself using it quite a lot.

http://www.bh-a.com/index.php?option=com_content&view=article&id=88〈=en

Hope this helps

Edited June 16, 2009 by Guest

[LaRetta]

I totally agree with you on this one LaRetta. My point was just to try to help you here and now, having reported this issue several times and never seing a fix... :

Funny : this post seems to be yours now. No problem, I'm happy to give it to you ;)

Edited June 16, 2009 by Guest

[RodSierra]

However, this does in no way apply to the target side, and I can see no excuse for Filemaker spontaneously enabling a target field for import.

This has caught me more than once also, and I could not agree more this should be changed to maintain target field integrity based on the developers script contents.

[IdealData]

Many thanks LaRetta.

I'm right in the middle of a project where the next phase will lead to a scripted 'build' of data from other tables. I'm in two minds to script using looping and $vars or used the import script step - so thanks for the 'heads up' - just in the nick of time.

[LaRetta]

Fabrice ... :girlgiggle:

I had responded to you here and it existed here for an hour before it changed to YOUR post. It seems that you EDITED my post, deleted my words and wrote your own instead. No problem.

I'm glad you have offered alternative but I do not want people to accept alternatives on an obvious SERIOUS break. There are many thousands of developers using imports that will NOT read this thread nor will they see your alternative!! This is not simple break when one changes a name of a field and the import is based upon name ... this is ADDING new fields created in a file to an import map in ScriptMaker which probably has nothing to do with the field added.

It needs to be fixed in the very next updater. Period.

[LaRetta]

Here is another update to the 'spontaneous' unasked-for mapping from a source to target when new fields are added in target ...

Do not count on disabled scripts (with imports) from being safe from the random mapping either ... if you print a disabled script with an import, it will have mapped all newly created target fields back through time even if the script has been disabled for 6 months (and never activated). Nothing is safe from this issue!!

Why would I have a disabled import script? I have to because FM will not allow mapping SQL query from an FM calculation to target - I must create a manual import, map it, then change the import step to using the calculation (which I consider another major bug). So to save having to re-map any time, I save one disabled import script (which should contain my perfect map) and then when I switch to the calculation, it's already mapped. But no ... it now contains garbage (it should only have Source fields 1-4. 26, 27, 31 and 32 in this one disabled script were added while the script is disabled.

UPDATE: These two import scripts you are viewing were IDENTICAL a year ago (I duplicated the manual to change to calculation) and they haven't been touched at all since.

Edited July 8, 2009 by Guest
Added update

[LaRetta]

This serious spontaneous mapping issue is still broken in vs. 10 and was not fixed in vs. 11.

I have asked FMI several times to at least confirm whether they will fix it although they acknowledge it is broken and all we get is very loud silence. This is the most dangerous, unexpected behavior in FileMaker. It makes importing almost worthless and worse still ... it silently, behind the scenes, can overwrite existing data.

Someone else said that they were told that FMI could not fix it without major change to file structure but FMI remains publicly silent about it. That is not right; FMI needs to make people aware of this issue and since they won't, I will continue to bring it forward for everyone's attention.

Do not trust imports if 1) you are not mapping by field name and 2) add ANY fields into your target tables. If you don't thoroughly understand how this bug can bite you, please ask.

[Scottish Joe]

I just discovered this bug last week and it's thrown me into a tail-spin. We are in the high-tech industry and use FileMaker for absolutely EVERYTHING, customer orders, customer profiles, manufacturing input, statistical process control, invoicing, corrective actions, audits, etc, etc and have made copious use of the import script step (it's probably used at least 100 times per day across 81 separate databases).

I've just reported it as a bug to FileMaker, been issued a case number and await input tomorrow.

The only difference I see to what's been said here is it may be what the script is reporting as the field mapping that's actually wrong, not what the field mapping actually is. For instance, I ran an import that appeared to change my data, closed the file and then re-opened it (hosted on our server) and the data was unchanged, i.e. as if the import never took place!

If I edit the field mapping and move the last/newest field in my target file/table to somewhere else in the order, save the script then edit it again, all the newer/remapped fields are no longer mapped and the field mapping is back to how I originally set it up with the exception of the newest field I changed in the last step. So, my fix has been to go into every script that uses import and do the same routine over and over. Has anyone else experienced it like this?

[LaRetta]

If you use the test file I provided in the opening post, you will see the break in action and it does overwrite data.

And FMI knows about the bug - there have been long discussions, several posts and complaints on the FileMaker forum here for instance . I hope you weren't charged for that call and why will you have to wait for a response when their own people should be able to pull that up easily, since they've known about it for several years?

Currently, we (like you) must track every script which imports into every table. Like you, we have many ad hoc files and external sources which import. So every time we add a field to a table, we must check our maps. If we forget, it is disaster.

FMI should, since they can't/won't fix it, provide us with the ability to quickly find/open all imports related to a specified table so we can correct the maps. Mostly, it should be public knowledge, easy to find and even included in FM Help under importing as an important notice. Instead, they stick their heads in the sand and hope people don't notice it.

[Scottish Joe]

Thanks LaRetta but I can see now it does overwrite the data. I was looking at global fields - duh!

It certainly isn't consistent and that's quite a worry also. I have two identical PC's using identical (yet local) databases and a new field was mapped on one system, from server hosted files, but not the other.... There's more to this than meets the eye.

I'm pretty annoyed about this issue. Why the users aren't informed of this massive bug is beyond me. We have a VLA, maintenance contract and priority support so should be a reasonably important customer. I guess FM just doesn't have such a system or culture. Needless to say I've asked my account manager to come by........

[LaRetta]

I also highly suggest you voice your dissatisfaction here in the Report a Bug forum. It is only if enough people speak out publicly that something will be done.

This is my complaint ... we are not warned and it is severely hurting developers and customers every day and has for years. I can't believe that they spend time designing us an Quick Screen and My Favorites (new almost-worthless features) but yet leave severe bugs. Those that make the decisions about what should be added or fixed (whether Product Managers, Marketing Managers) should be fired.

[RodSierra]

Quite frankly I think this software company is only now focusing on the new user market, and has pretty much abandoned the developers. Why spend resource time fixing past issues when you can get new customers by offering fluff, and foof versus the issues that the developers want fixed and added. This will catch up with them eventually, as I for one have had it. We are now doing no new module development in FM and switching to another platform. We have frozen development at FM10, and although we may use future releases of FM to maintain our offerings we will not add any new features beyond what FM10 supports. I've personally had it with these folks and no matter how painfull we are moving on. The amount of basic bugs found in FM11, along with another long standing issue, that of lack of support for Thai language input via standard unicode (since unicode was introduced, I think FM7). So based on my griping about the Thai issue I think there is little chance the import bug will ever be fixed, done complaining now moving on.

[Søren Dyhr]

Quite frankly I think this software company is only now focusing on the new user market, and has pretty much abandoned the developers.

This is the mourning this company always have been subjected to. It's however pretty much more profitable to pursue that direction, than to pamper the developers as such - it's a strategy due to it's ownership and in fact what makes Apple different from fellow gadgeteers. Never delivering engineering to the nerds, but instead focus on enduser experience.

--sd

[RodSierra]

Soren to a certain extent I agree that there must be a draw for new users, and I think the FM is one program that can draw the spreadsheet crowd over to using databases, I have no problem with that, but then again this company has gone beyond that and added features and multiuser environments that are well beyond the basic intro users capability and provided a product that developers have supported. So what is their direction today...it seems to have switched to foof and fluff, are they really starting to abandon the development market...they would say no, but it sure seems that way to me and my team, we had about 4 painful meetings before we decided to switch platforms, and it will be a long road out from under these folks, but that's our direction now.

And another thought, I use a 3d cad system that is very complex in its interface, and takes a long time to come up the curve to be productive, they have not tried to simplify this product to make gains in market share, but have instead introduced a new product that allows entry users to use their product more easily then transition those files to their high end solution later. I suspect this is the dilemma that FM is facing, but they are handling it very badly IMHO.

Edited April 8, 2010 by Guest

[Søren Dyhr]

it seems to have switched to foof and fluff

As I'm told are three different versions usually in the mold, arriving to marked in a "tiled manner" ... and what could be interesting to establish is if one of those teams are worse managed or performing than the others.

We the developers are with some of the previous versions releases better served than others, but some upgrades seems ridiculously slim seen from the developers point of view.

I have not found the inclination to upgrade to 11 yet, my customers could easily continue running their tasks on fm8.5, with the addition of a few free plugins. However from a devlopers point of view makes fm10 sense to me with the debugger but have bought clipmanager from MyFmButler to streamline my productivity.

but have instead introduced a new product that allows entry users to use their product more easily then transition those files to their high end solution later.

Isn't it here fair and squarely Bento lands?

--sd

[RodSierra]

Soren, well as far as Bento is concerned, I must admit after FM Mobile, I never even bothered to download a trial. I found FMTouch to be more interesting, and especially now with iPad I think there is good potential in that market, this is what is so frustrating, I see the potential, but FM has their head up their__ well, anyway...we're on to new frustrations, at least away from the same old ones!

[mr_vodka]

Holy moley!!! Could it possibly be?

From the FM12 bahavior changes document....

Importing data from

FileMaker Pro no longer imports data from fields that were added or changed after field mapping was defined in the Field Mapping dialog box. Data is imported only from fields specified during the original import mapping.

The previous import mapping ( at runtime and design time ) automatically added fields to import into if they have been added after the import was defined, and the source has enough fields to pull data from. Also, there had been some reports that adding/removing fields from the SOURCE file (fmp format) "messed up" a pre-defined import ( at runtime and design time ).

In FileMaker Pro 12, if a field is added to the destination, it is not set to import into, just because there is a source field available. This avoids developers from updating all the imports in all their scripts in all their solution files, just because they added a field to the destination FileMaker file. Also, when importing from a FileMaker format file, the keys are stored along with the order. When resolving the mapping, if the key of the nth destination field doesn't match what's in the file, that entry is treated as no-import.

A side effect of this new behavior is the first time an import is performed for that file, the user has to explicitly turn on each field to import into, as opposed to the legacy behavior of the user having to clear out fields they don't.

Conversion: FileMaker Pro 12 is unable to detect what the keys were for existing imports from a FileMaker file since that information is only available when the order was last made/modified, but the first time the order is modified it is updated with the current field keys in the file.

When switching to a different source table when altering the import order in the UI, the stored import order is thrown away for the table. So re-selecting the original source table acts as normal/legacy. Also, when manually importing from FileMaker files in the same session, you may see "" entries if subsequent source files have less fields.

[LaRetta]

ROCKIN' COOL!!! FINALLY!!! Thanks, John! This is going to save a lot of headaches for all of us! We've been waiting for this ever since 7 came out! :yep:

[bcooney]

I thought of you, LaRetta, when I read this. Congrats! :cloud: