Delete Duplicates on Import

[Baylah]

Can somone help me with a script that will delete duplicate records as they are imported into a database?

I have a DataSource with more than 4 million records with many of the Order Numbers Being duplicate. I would like to filter out all but the first occurence of these Order numbers as the data is being imported.

Is this possible?

Thanks,

Steve

[Lee Smith]

Take a good look at the import dialog box. One of the options avialable is to Update existing records, and to add new records.

Lee

[Baylah]

Hi Lee,

Thanks for the quick response but I don;t see anything in the import dialog that would allow me to filter out duplicate records as they are coming in.

I'm not sure if I stated my question correctly originally...

I have a data source (ODBC) with over 4 million records. Many of these individual records share a matching "order Number" I would like to import only the first occurence of the Order Number.

Or I would like to Import all of the records and then run a sub script that deletes all of the duplicates after import, basically accomplishing the same thing if the original question is not doable.

thanks,

Steve

[mr_vodka]

Perhaps you can run a loop or use the self join method that the FM help suggests.

Duplicate Loop

[sbg2]

- Set your OrderNumber field to be unique and always validate.

- When you define your import select

--- "Update Matching Records in found set" radio button

--- "Add remaining data as new records" check box

--- change the field mapping for your ordernum fields to an equal sign.

Files attached for your reference, not sure if it will save the import info.

MyOrderNum.zip

[JoniS]

When you select your ODBC data source and click continue, it will open a SQL Query Builder window.

Your SQL Query should look similar to this:

Select Distinct TableName.OrderID

From TableName

By adding the word "Distinct", you will only import the first occurence of each OrderID.