Can somone help me with a script that will delete duplicate records as they are imported into a database?

I have a DataSource with more than 4 million records with many of the Order Numbers Being duplicate. I would like to filter out all but the first occurence of these Order numbers as the data is being imported.

Is this possible?

Thanks,

Steve

Take a good look at the import dialog box. One of the options avialable is to Update existing records, and to add new records.

Lee

Hi Lee,

Thanks for the quick response but I don;t see anything in the import dialog that would allow me to filter out duplicate records as they are coming in.

I'm not sure if I stated my question correctly originally...

I have a data source (ODBC) with over 4 million records. Many of these individual records share a matching "order Number" I would like to import only the first occurence of the Order Number.

Or I would like to Import all of the records and then run a sub script that deletes all of the duplicates after import, basically accomplishing the same thing if the original question is not doable.

thanks,

Steve

Perhaps you can run a loop or use the self join method that the FM help suggests.

Duplicate Loop

- Set your OrderNumber field to be unique and always validate.

- When you define your import select

--- "Update Matching Records in found set" radio button

--- "Add remaining data as new records" check box

--- change the field mapping for your ordernum fields to an equal sign.

Files attached for your reference, not sure if it will save the import info.

MyOrderNum.zip

When you select your ODBC data source and click continue, it will open a SQL Query Builder window.

Your SQL Query should look similar to this:

Select Distinct TableName.OrderID

From TableName

By adding the word "Distinct", you will only import the first occurence of each OrderID.