Jump to content
Sign in to follow this  
Baylah

Delete Duplicates on Import

Recommended Posts

Can somone help me with a script that will delete duplicate records as they are imported into a database?

I have a DataSource with more than 4 million records with many of the Order Numbers Being duplicate. I would like to filter out all but the first occurence of these Order numbers as the data is being imported.

Is this possible?

Thanks,

Steve

Share this post


Link to post
Share on other sites

Take a good look at the import dialog box. One of the options avialable is to Update existing records, and to add new records.

Lee

Share this post


Link to post
Share on other sites

Hi Lee,

Thanks for the quick response but I don;t see anything in the import dialog that would allow me to filter out duplicate records as they are coming in.

I'm not sure if I stated my question correctly originally...

I have a data source (ODBC) with over 4 million records. Many of these individual records share a matching "order Number" I would like to import only the first occurence of the Order Number.

Or I would like to Import all of the records and then run a sub script that deletes all of the duplicates after import, basically accomplishing the same thing if the original question is not doable.

thanks,

Steve

Share this post


Link to post
Share on other sites

Perhaps you can run a loop or use the self join method that the FM help suggests.

Duplicate Loop

Share this post


Link to post
Share on other sites

- Set your OrderNumber field to be unique and always validate.

- When you define your import select

--- "Update Matching Records in found set" radio button

--- "Add remaining data as new records" check box

--- change the field mapping for your ordernum fields to an equal sign.

Files attached for your reference, not sure if it will save the import info.

MyOrderNum.zip

Share this post


Link to post
Share on other sites

When you select your ODBC data source and click continue, it will open a SQL Query Builder window.

Your SQL Query should look similar to this:

Select Distinct TableName.OrderID

From TableName

By adding the word "Distinct", you will only import the first occurence of each OrderID.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.