Page 1 of 1

Customer validates at 5 - 8.5 million records per hour

Posted: Fri Nov 12, 2010 8:48 am
by Eric
From a customer's email:
I moved Accuzip into a virtual machine so that I could move the VM (and thus Accuzip) from server to server as required. That works very well, in fact I built a new server with tons of cores and RAM and moved the VM to that and it (Accuzip) ran significantly slower, due to lower core clock AFAICT. So I just created a little server specifically for the VM and Accuzip and it is zipping along now.

I am consistently getting well over 5 million records / hour and sometimes getting around 8.5 million records / hour by the end of the validating records process and I thought I'd share what I found when working to get my speeds up.

You guys told me long ago to sort by Zip5/Zip4 (Optimize Database - Order Database for Optimum Performance) which got my speeds up from around 2 million to something like 3.5 million records / hour.

The next thing I did was to give the VM a 30G Solid State Disk as a work disk. I placed all of the USPS database files out there. I also made the SSD the disk that held all of the input / output directories (where you build your db processing files). Just doing that allowed me to get about 5.3 million records / hour.

Finally I pre-filtered all of the records sent to Accuzip to get rid of the "Move - no forwarding address", ANK and otherwise invalid addresses. Essentially after the first Accuzip pass (I Accuzip monthly) I only send Accuzip addresses that you told me were valid addresses on the last pass.
Doing that bumped my speeds up, sometimes though not always dramatically.