Jump to content
Sign in to follow this  
Baylah

Delete Duplicates on Import

Recommended Posts

Can somone help me with a script that will delete duplicate records as they are imported into a database?

I have a DataSource with more than 4 million records with many of the Order Numbers Being duplicate. I would like to filter out all but the first occurence of these Order numbers as the data is being imported.

Is this possible?

Thanks,

Steve

Share this post


Link to post
Share on other sites

Take a good look at the import dialog box. One of the options avialable is to Update existing records, and to add new records.

Lee

Share this post


Link to post
Share on other sites

Hi Lee,

Thanks for the quick response but I don;t see anything in the import dialog that would allow me to filter out duplicate records as they are coming in.

I'm not sure if I stated my question correctly originally...

I have a data source (ODBC) with over 4 million records. Many of these individual records share a matching "order Number" I would like to import only the first occurence of the Order Number.

Or I would like to Import all of the records and then run a sub script that deletes all of the duplicates after import, basically accomplishing the same thing if the original question is not doable.

thanks,

Steve

Share this post


Link to post
Share on other sites

- Set your OrderNumber field to be unique and always validate.

- When you define your import select

--- "Update Matching Records in found set" radio button

--- "Add remaining data as new records" check box

--- change the field mapping for your ordernum fields to an equal sign.

Files attached for your reference, not sure if it will save the import info.

MyOrderNum.zip

Share this post


Link to post
Share on other sites

When you select your ODBC data source and click continue, it will open a SQL Query Builder window.

Your SQL Query should look similar to this:

Select Distinct TableName.OrderID

From TableName

By adding the word "Distinct", you will only import the first occurence of each OrderID.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information

By using this site, you agree to our Terms of Use.