Difference between revisions of "MatchUp Object:Best Practices"

From Melissa Data Wiki
Jump to navigation Jump to search
Line 8: Line 8:
*Network data traffic
*Network data traffic


We recommend that the source data to be processed by local with repect to the installed Melissa Data
:We recommend that the source data to be processed be local with respect to the installed Melissa Data
program. Network permissions, throughput, and in some cases, MatchUps need to access record 'x' to complete
program. Network permissions, throughput, and in some cases, MatchUp's need to access record 'x' to complete
consolidation with record 'y', are all potential sources o f a slower process.
consolidation with record 'y', are all potential sources of a slower process.


*Source datatype
*Source datatype


Some database or file types can be read by the calling language or IDE more efficiently than others. Matching your
:Some database or file types can be read by the calling language or IDE more efficiently than others. Matching your
environment to the most efficient file type requires trial and error testing by the developer.
environment to the most efficient file type requires trial and error testing by the developer.


*Hardware :
*Hardware :


It goes without saying that the more hardware you dedicate to a process, the faster it will run. However,
:It goes without saying that the more hardware you dedicate to a process, the faster it will run. However,
many processes can not take advantage of additional hardware, or show diminishing returns. For example,
many processes can not take advantage of additional hardware, or show diminishing returns. For example,
varied zip code demographics may be able to use multi-processors to process individual clusters of records,
varied zip code demographics may be able to use multi-processors to process individual clusters of records,
but a database of the same zip code may not. Additionally, for the above factors, hardware may not be the
but a database of the same zip code may not. Additionally, for the above factors, hardware may not be the
overriding factor governing a fast process, ie. a good matchcode may be the most important factor.
overriding factor governing a fast process, ie. a good matchcode may be the most important factor.


==Optimizing Speed: Matchcodes==
==Optimizing Speed: Matchcodes==

Revision as of 16:37, 26 February 2014

Back to MatchUp Object Main Page

Matchup Object Best Practices contains support recommendations when performance is not optimal


Optimizing Speed: General

BP_MUXX_001

  • Network data traffic
We recommend that the source data to be processed be local with respect to the installed Melissa Data

program. Network permissions, throughput, and in some cases, MatchUp's need to access record 'x' to complete consolidation with record 'y', are all potential sources of a slower process.

  • Source datatype
Some database or file types can be read by the calling language or IDE more efficiently than others. Matching your

environment to the most efficient file type requires trial and error testing by the developer.

  • Hardware :
It goes without saying that the more hardware you dedicate to a process, the faster it will run. However,

many processes can not take advantage of additional hardware, or show diminishing returns. For example, varied zip code demographics may be able to use multi-processors to process individual clusters of records, but a database of the same zip code may not. Additionally, for the above factors, hardware may not be the overriding factor governing a fast process, ie. a good matchcode may be the most important factor.

Optimizing Speed: Matchcodes

BP_MUXX_002
1. Matchcodes: Components: Before MatchUp dedupes, it clusters records into groups of possible matches. If your matchcode does not have any components in every used combination, it can not place records into those sub group clusters. In general, the greater number of components used in every combination, the faster the process will be.

2. Matchcode: Fuzzy: MatchUp has an extensive list of Fuzzy Options. Some are performed during the key building process (ie. Soundex) and do not slow the process down. Others are performed on the constructed matchkeys (ie. Near, Jaro, etc.) and therefore slow down the process. If the latter types are required by your process, place them in the component order below an exact component which is also used in every combintation if possible.


Order of Components in Matchcode

BP_MUXX_003
Although the Matchcode Editor interface lets you place the components in any order, the Object does have a few restrictions when calling the AddMapping methods. Namely, Address Line AddMappings must be called last, even if you have added another component after the Address matchcode components. Calling AddMappings in the wrong order will throw an error, therefore when using the matchcode editor, place your address components last. The exception would be rare cases where address components are used in every specified column, but a different component is not used in all combinations (specified columns).


Using Efficient SetUserInfo

BP_MUOB_001
By default, SetUserInfo, the unique identifier attached to built match key is 1024 bytes, allowing the developer to pass an advanced custom identifier, or even source data to the key file. While this can have data handling advantages, this will cause the key file and temporary sort files to grow much larger than needed for most jobs, and will slow down the process. A new reserve funcion has been added, allowing the user to override the default UserInfo size. Example...

ReadWrite->SetReserved("UserInfoSize","12");

Our tests have shown this to reduce the key and temporary disk storage usage to decrease by a factor of 10 and the processing time to decrease by as much as 60%


Keep Work File Location Local

BP_MUXX_004
MatchUp uses this location to store the process key file as well as temporary sorting files.
By default, Windows will store these files in the temp directory of the logged in User. For *nix platforms, the directory where the executable is being ran.
Although users can override this location, we do not recommned it, unless you are pointing this location to a fast local drive with plenty of writable disk space and full read write permissions.


Back to MatchUp Object Main Page