Dealing with Large Datasets

Exporting expressions and information including exporting data in presort order or import order, and importing information, along with error messages that may apply to this category.

Moderators: Chrissy, Steve, Terry, Donna, Dave, Eric, MartyH, Kristen, Geoff, Vince

Post Reply
Kristen
Moderator
Posts: 726
Joined: Wed Sep 20, 2006 3:46 pm

Dealing with Large Datasets

Post by Kristen » Tue Jan 13, 2009 2:08 pm

AccuZIP supports a maximum file size of 2GB. When dealing with large datasets, here are some steps to create a template that will allow you to optimize/maximize the number of records you can import and process in AccuZIP:

Here are the steps for creating a template:

1. Create a 500 or so record file from the same large database. (Make sure the structure is the same, example FIRST,LAST,ADDRESS...)
2. Bring the smaller database into AccuZIP6
3. Validate the records with NCOALink, DPV and any other options in the validation screen selected that you need.
4. After the validation is done then delete all records from the database (Edit>Delete) and pack to remove the records (Edit>Pack)
5. You have now created a template.
6. >From the Edit>Fields menu, reduce all fields down to 1 character wide that you're not going to use for example the Errno_ field can be reduced. Also the Barcode field if you don't need reduce it as well. You don't want to remove any fields just reduce the size. If AccuZIP6 sees the field it won't try to create it.
7. Note: You can determine how many records will fit in the table by following the formula below:
>From the Edit>Fields options click the Table Tab
You will see Records: Fields: and Length:
Divide 2000 by the Length and you will get the amount of records you can bring into this table. (Length =680 then 2000 ÷ 680 = 2.94 million records. )
8. Now that we have a template created with the exact structure as the file you are bringing in, there are two ways to import the file.
*NOTE* Make sure the AccuZIP6 working disk is set to a folder other than the Windows\Temp folder. You can change this in the File>Properties window. Click the folders tab.

1. Open the template and use the File - Import feature

2. Use our command window

From the command window (Edit>Command) type the following into the command if the file is a CSV format:

Append from ? type csv

If the file is a .dbf already the command would be:
Append from ?

Note: You can always replace the "?" with the actual path to the file, the "?" will simply let you choose the file to be imported.
(This will prompt you for the file you want to bring in. Replace CSV with TXT for tab separated or SDF for fixed length files)

9. Now you can do the normal validation.
10. There is no need to OPTIMIZE the database. If the database is already in zip order then it's fine.

Post Reply