MatchUp Object:Best Practices - Revision history

Admin at 20:17, 25 May 2018

2018-05-25T20:17:40Z

← Older revision		Revision as of 20:17, 25 May 2018
Line 4:		Line 4:

	Matchup Object Best Practices contains support recommendations when performance is not optimal.		Matchup Object Best Practices contains support recommendations when performance is not optimal.

			==Intersecting Deduper==
			When a matchcode that circumvents the first component restrictions (not used in all combinations, or fuzzy algorithm applied) is used, throughput should be expected to be significantly slower. This can also cause stability issues. When processing large amounts of records, we do not recommend using this type of matchcode. Test thoroughly with small amounts of records before scaling up to larger data sets and a production environment. Using the Hybrid Deduper or small amounts of data will not show the problem. For more info: [[MatchUp Object:Matchcode Combinations\|Matchcode Combinations]].


			==Matchcodes with Fuzzy Algorithms==
			Since fuzzy algorithms can exponentially slow down a process or raise stability issues for enterprise-level processes, we recommend that you establish acceptable throughput benchmarks with a standard exact matchcode. Then make small incremental changes that progress to the desired matching strategies. Given the quality of data and amount of records, use of certain matchcode properties may be impractical to achieve desired speeds.


	==Optimizing Speed: General==		==Optimizing Speed: General==

Admin at 16:48, 30 April 2015

2015-04-30T16:48:45Z

← Older revision		Revision as of 16:48, 30 April 2015
Line 6:		Line 6:

	==Optimizing Speed: General==		==Optimizing Speed: General==
	~~;BP_MUXX_001~~
	#Network data traffic: We recommend that the source data to be processed be local with respect to the installed Melissa Data program. Network permissions, throughput, and in some cases, MatchUp's need to access record 'x' to complete consolidation with record 'y', are all potential sources of a slower process.		#Network data traffic: We recommend that the source data to be processed be local with respect to the installed Melissa Data program. Network permissions, throughput, and in some cases, MatchUp's need to access record 'x' to complete consolidation with record 'y', are all potential sources of a slower process.
	#Source datatype :Some database or file types can be read by the calling language or IDE more efficiently than others. Matching your environment to the most efficient file type requires trial and error testing by the developer.		#Source datatype :Some database or file types can be read by the calling language or IDE more efficiently than others. Matching your environment to the most efficient file type requires trial and error testing by the developer.
Line 13:		Line 12:

	==Optimizing Speed: Matchcodes==		==Optimizing Speed: Matchcodes==
	~~;BP_MUXX_002~~
	#Matchcodes: Components: Before MatchUp dedupes, it clusters records into groups of possible matches. If your matchcode does not have any components in every used combination, it can not place records into those sub group clusters. In general, the greater number of components used in every combination, the faster the process will be.		#Matchcodes: Components: Before MatchUp dedupes, it clusters records into groups of possible matches. If your matchcode does not have any components in every used combination, it can not place records into those sub group clusters. In general, the greater number of components used in every combination, the faster the process will be.
	#Matchcode: Fuzzy: MatchUp has an extensive list of Fuzzy Options. Some are performed during the key building process (ie. Soundex) and do not slow the process down. Others are performed on the constructed matchkeys (ie. Near, Jaro, etc.) and therefore slow down the process. If the latter types are required by your process, place them in the component order below an exact component which is also used in every combination if possible.		#Matchcode: Fuzzy: MatchUp has an extensive list of Fuzzy Options. Some are performed during the key building process (ie. Soundex) and do not slow the process down. Others are performed on the constructed matchkeys (ie. Near, Jaro, etc.) and therefore slow down the process. If the latter types are required by your process, place them in the component order below an exact component which is also used in every combination if possible.
Line 19:		Line 17:

	==Order of Components in Matchcode==		==Order of Components in Matchcode==
	~~;BP_MUXX_003~~
	Although the Matchcode Editor interface lets you place the components in any order, the Object does have a few restrictions when calling the AddMapping methods. Namely, Address Line AddMappings must be called last, even if you have added another component after the Address matchcode components. Calling AddMappings in the wrong order will throw an error, therefore when using the matchcode editor, place your address components last. The exception would be rare cases where address components are used in every specified column, but a different component is not used in all combinations (specified columns).		Although the Matchcode Editor interface lets you place the components in any order, the Object does have a few restrictions when calling the AddMapping methods. Namely, Address Line AddMappings must be called last, even if you have added another component after the Address matchcode components. Calling AddMappings in the wrong order will throw an error, therefore when using the matchcode editor, place your address components last. The exception would be rare cases where address components are used in every specified column, but a different component is not used in all combinations (specified columns).


	==Back up your Matchcode database==		==Back up your Matchcode database==
	~~;BP_MUXX_005~~
	If you create your own matching strategies, you should occasionally back up this file - in the event that someone changes a matchcode or it becomes corrupted.		If you create your own matching strategies, you should occasionally back up this file - in the event that someone changes a matchcode or it becomes corrupted.

Line 35:		Line 31:

	==Using Efficient SetUserInfo==		==Using Efficient SetUserInfo==
	~~;BP_MUOB_001~~
	By default, SetUserInfo, the unique identifier attached to built match key is 1024 bytes, allowing the		By default, SetUserInfo, the unique identifier attached to built match key is 1024 bytes, allowing the
	developer to pass an advanced custom identifier, or even source data to the key file. While this can have		developer to pass an advanced custom identifier, or even source data to the key file. While this can have
Line 57:		Line 52:

	==Keep Work File Location Local==		==Keep Work File Location Local==
	~~;BP_MUXX_004~~
	MatchUp uses this location to store the process key file as well as temporary sorting files.		MatchUp uses this location to store the process key file as well as temporary sorting files.

Admin at 18:40, 25 November 2014

2014-11-25T18:40:28Z

← Older revision		Revision as of 18:40, 25 November 2014
Line 1:		Line 1:
	[[MatchUp Object\|~~Back to~~ MatchUp Object ~~Main Page~~]]		[[MatchUp Object\|← MatchUp Object]]

	~~'''Matchup Object Best Practices contains support recommendations when performance is not optimal'''~~		{{CustomTOC}}

			Matchup Object Best Practices contains support recommendations when performance is not optimal.

	==Optimizing Speed: General==		==Optimizing Speed: General==
	BP_MUXX_001 ~~<br>~~		;BP_MUXX_001
	#Network data traffic: We recommend that the source data to be processed be local with respect to the installed Melissa Data program. Network permissions, throughput, and in some cases, MatchUp's need to access record 'x' to complete consolidation with record 'y', are all potential sources of a slower process.		#Network data traffic: We recommend that the source data to be processed be local with respect to the installed Melissa Data program. Network permissions, throughput, and in some cases, MatchUp's need to access record 'x' to complete consolidation with record 'y', are all potential sources of a slower process.
	#Source datatype :Some database or file types can be read by the calling language or IDE more efficiently than others. Matching your environment to the most efficient file type requires trial and error testing by the developer.		#Source datatype :Some database or file types can be read by the calling language or IDE more efficiently than others. Matching your environment to the most efficient file type requires trial and error testing by the developer.
Line 12:		Line 13:

	==Optimizing Speed: Matchcodes==		==Optimizing Speed: Matchcodes==
	BP_MUXX_002 ~~<br>~~		;BP_MUXX_002
	#Matchcodes: Components: Before MatchUp dedupes, it clusters records into groups of possible matches. If your matchcode does not have any components in every used combination, it can not place records into those sub group clusters. In general, the greater number of components used in every combination, the faster the process will be.		#Matchcodes: Components: Before MatchUp dedupes, it clusters records into groups of possible matches. If your matchcode does not have any components in every used combination, it can not place records into those sub group clusters. In general, the greater number of components used in every combination, the faster the process will be.
	#Matchcode: Fuzzy: MatchUp has an extensive list of Fuzzy Options. Some are performed during the key building process (ie. Soundex) and do not slow the process down. Others are performed on the constructed matchkeys (ie. Near, Jaro, etc.) and therefore slow down the process. If the latter types are required by your process, place them in the component order below an exact component which is also used in every combination if possible.		#Matchcode: Fuzzy: MatchUp has an extensive list of Fuzzy Options. Some are performed during the key building process (ie. Soundex) and do not slow the process down. Others are performed on the constructed matchkeys (ie. Near, Jaro, etc.) and therefore slow down the process. If the latter types are required by your process, place them in the component order below an exact component which is also used in every combination if possible.
Line 18:		Line 19:

	==Order of Components in Matchcode==		==Order of Components in Matchcode==
	BP_MUXX_003 ~~<br>~~		;BP_MUXX_003
	Although the Matchcode Editor interface lets you place the components in any order, the Object does have a few restrictions when calling the AddMapping methods. Namely, Address Line AddMappings must be called last, even if you have added another component after the Address matchcode components. Calling AddMappings in the wrong order will throw an error, therefore when using the matchcode editor, place your address components last. The exception would be rare cases where address components are used in every specified column, but a different component is not used in all combinations (specified columns).		Although the Matchcode Editor interface lets you place the components in any order, the Object does have a few restrictions when calling the AddMapping methods. Namely, Address Line AddMappings must be called last, even if you have added another component after the Address matchcode components. Calling AddMappings in the wrong order will throw an error, therefore when using the matchcode editor, place your address components last. The exception would be rare cases where address components are used in every specified column, but a different component is not used in all combinations (specified columns).


	==Back up your Matchcode database==		==Back up your Matchcode database==
	BP_MUXX_005 ~~<br>~~		;BP_MUXX_005
	If you create your own matching strategies, you should occasionally back up this file - in the event that someone changes a matchcode or it becomes corrupted. ~~<br>~~		If you create your own matching strategies, you should occasionally back up this file - in the event that someone changes a matchcode or it becomes corrupted.
	For the Object and SSIS, and Contact Zone, this file is named mdMatchUp.mc ~~<br>~~
	For the MatchUp Software version, the file is named DTake.mc ~~<br>~~		For the Object and SSIS, and Contact Zone, this file is named mdMatchUp.mc

			For the MatchUp Software version, the file is named DTake.mc

	An example of good backup practice would be mdMatchUp_20140123.mc, allowing you to see the original matchcode used in processes before Jan 23, 2014		An example of good backup practice would be mdMatchUp_20140123.mc, allowing you to see the original matchcode used in processes before Jan 23, 2014


	==Using Efficient SetUserInfo==		==Using Efficient SetUserInfo==
	BP_MUOB_001 ~~<br>~~		;BP_MUOB_001
	By default, SetUserInfo, the unique identifier attached to built match key is 1024 bytes, allowing the		By default, SetUserInfo, the unique identifier attached to built match key is 1024 bytes, allowing the
	developer to pass an advanced custom identifier, or even source data to the key file. While this can have		developer to pass an advanced custom identifier, or even source data to the key file. While this can have
	data handling advantages, this will cause the key file and temporary sort files to grow much larger than		data handling advantages, this will cause the key file and temporary sort files to grow much larger than
	needed for most jobs, and will slow down the process. A new reserve funcion has been added, allowing the		needed for most jobs, and will slow down the process. A new reserve funcion has been added, allowing the
	user to override the default UserInfo size. Example~~...~~		user to override the default UserInfo size. For Example:

			<pre>
	ReadWrite->SetReserved("UserInfoSize","12");		ReadWrite->SetReserved("UserInfoSize","12");
			</pre>

	Our tests have shown this to reduce the key and temporary disk storage usage to decrease by a factor of		Our tests have shown this to reduce the key and temporary disk storage usage to decrease by a factor of
	10 and the processing time to decrease by as much as 60%		10 and the processing time to decrease by as much as 60%

	To determine if you have the necessary Update Build 2072 or newer, Programmatically: <br>		To determine if you have the necessary Update Build 2072 or newer, Programmatically:

			<pre>
	printf(" BUILD NUMBER: %s\n",mdMUReadWriteGetBuildNumber(ReadWrite));		printf(" BUILD NUMBER: %s\n",mdMUReadWriteGetBuildNumber(ReadWrite));
			</pre>


	==Keep Work File Location Local==		==Keep Work File Location Local==
	BP_MUXX_004 ~~<br>~~		;BP_MUXX_004
	MatchUp uses this location to store the process key file as well as temporary sorting files. ~~<br>~~		MatchUp uses this location to store the process key file as well as temporary sorting files.

	By default, Windows will store these files in the temp directory of the logged in User. For *nix		By default, Windows will store these files in the temp directory of the logged in User. For *nix
	platforms, the directory where the executable is being ran. ~~<br>~~		platforms, the directory where the executable is being ran.

	Although users can override this location, we do not recommned it, unless you are pointing this location to a fast local drive with plenty of writable disk space and full read write permissions.		Although users can override this location, we do not recommned it, unless you are pointing this location to a fast local drive with plenty of writable disk space and full read write permissions.


	[[MatchUp Object~~\|Back to MatchUp Object Main Page~~]]		[[Category:MatchUp Object]]

Admin: Admin moved page Best Practices:MatchUp API to MatchUp Object:Best Practices without leaving a redirect

2014-05-23T16:45:42Z

Admin moved page Best Practices:MatchUp API to MatchUp Object:Best Practices without leaving a redirect

← Older revision	Revision as of 16:45, 23 May 2014
(No difference)

Tim: /* Back up your Matchcode database */

2014-02-27T19:10:50Z

Back up your Matchcode database

← Older revision		Revision as of 19:10, 27 February 2014
Line 23:		Line 23:

	==Back up your Matchcode database==		==Back up your Matchcode database==
	~~BP_MUXX_004~~ <br>		BP_MUXX_005 <br>
	If you create your own matching strategies, you should occasionally back up this file - in the event that someone changes a matchcode or it becomes corrupted. <br>		If you create your own matching strategies, you should occasionally back up this file - in the event that someone changes a matchcode or it becomes corrupted. <br>
	For the Object and SSIS, and Contact Zone, this file is named mdMatchUp.mc <br>		For the Object and SSIS, and Contact Zone, this file is named mdMatchUp.mc <br>

Tim: /* Order of Components in Matchcode */

2014-02-27T19:05:21Z

Order of Components in Matchcode

← Older revision		Revision as of 19:05, 27 February 2014
Line 21:		Line 21:
	Although the Matchcode Editor interface lets you place the components in any order, the Object does have a few restrictions when calling the AddMapping methods. Namely, Address Line AddMappings must be called last, even if you have added another component after the Address matchcode components. Calling AddMappings in the wrong order will throw an error, therefore when using the matchcode editor, place your address components last. The exception would be rare cases where address components are used in every specified column, but a different component is not used in all combinations (specified columns).		Although the Matchcode Editor interface lets you place the components in any order, the Object does have a few restrictions when calling the AddMapping methods. Namely, Address Line AddMappings must be called last, even if you have added another component after the Address matchcode components. Calling AddMappings in the wrong order will throw an error, therefore when using the matchcode editor, place your address components last. The exception would be rare cases where address components are used in every specified column, but a different component is not used in all combinations (specified columns).


			==Back up your Matchcode database==
			BP_MUXX_004 <br>
			If you create your own matching strategies, you should occasionally back up this file - in the event that someone changes a matchcode or it becomes corrupted. <br>
			For the Object and SSIS, and Contact Zone, this file is named mdMatchUp.mc <br>
			For the MatchUp Software version, the file is named DTake.mc <br>
			An example of good backup practice would be mdMatchUp_20140123.mc, allowing you to see the original matchcode used in processes before Jan 23, 2014

	==Using Efficient SetUserInfo==		==Using Efficient SetUserInfo==

Tim at 17:15, 26 February 2014

2014-02-26T17:15:37Z

← Older revision		Revision as of 17:15, 26 February 2014
Line 9:		Line 9:
	#Source datatype :Some database or file types can be read by the calling language or IDE more efficiently than others. Matching your environment to the most efficient file type requires trial and error testing by the developer.		#Source datatype :Some database or file types can be read by the calling language or IDE more efficiently than others. Matching your environment to the most efficient file type requires trial and error testing by the developer.
	#Hardware :It goes without saying that the more hardware you dedicate to a process, the faster it will run. However, many processes can not take advantage of additional hardware, or show diminishing returns. For example, varied zip code demographics may be able to use multi-processors to process individual clusters of records, but a database of the same zip code may not. Additionally, for the above factors, hardware may not be the overriding factor governing a fast process, ie. a good matchcode may be the most important factor.		#Hardware :It goes without saying that the more hardware you dedicate to a process, the faster it will run. However, many processes can not take advantage of additional hardware, or show diminishing returns. For example, varied zip code demographics may be able to use multi-processors to process individual clusters of records, but a database of the same zip code may not. Additionally, for the above factors, hardware may not be the overriding factor governing a fast process, ie. a good matchcode may be the most important factor.


	==Optimizing Speed: Matchcodes==		==Optimizing Speed: Matchcodes==
Line 14:		Line 15:
	#Matchcodes: Components: Before MatchUp dedupes, it clusters records into groups of possible matches. If your matchcode does not have any components in every used combination, it can not place records into those sub group clusters. In general, the greater number of components used in every combination, the faster the process will be.		#Matchcodes: Components: Before MatchUp dedupes, it clusters records into groups of possible matches. If your matchcode does not have any components in every used combination, it can not place records into those sub group clusters. In general, the greater number of components used in every combination, the faster the process will be.
	#Matchcode: Fuzzy: MatchUp has an extensive list of Fuzzy Options. Some are performed during the key building process (ie. Soundex) and do not slow the process down. Others are performed on the constructed matchkeys (ie. Near, Jaro, etc.) and therefore slow down the process. If the latter types are required by your process, place them in the component order below an exact component which is also used in every combination if possible.		#Matchcode: Fuzzy: MatchUp has an extensive list of Fuzzy Options. Some are performed during the key building process (ie. Soundex) and do not slow the process down. Others are performed on the constructed matchkeys (ie. Near, Jaro, etc.) and therefore slow down the process. If the latter types are required by your process, place them in the component order below an exact component which is also used in every combination if possible.


	==Order of Components in Matchcode==		==Order of Components in Matchcode==

Tim at 17:15, 26 February 2014

2014-02-26T17:15:18Z

← Older revision		Revision as of 17:15, 26 February 2014
Line 35:		Line 35:
	To determine if you have the necessary Update Build 2072 or newer, Programmatically: <br>		To determine if you have the necessary Update Build 2072 or newer, Programmatically: <br>
	printf(" BUILD NUMBER: %s\n",mdMUReadWriteGetBuildNumber(ReadWrite));		printf(" BUILD NUMBER: %s\n",mdMUReadWriteGetBuildNumber(ReadWrite));


	==Keep Work File Location Local==		==Keep Work File Location Local==

Tim: /* Using Efficient SetUserInfo */

2014-02-26T17:14:58Z

Using Efficient SetUserInfo

← Older revision		Revision as of 17:14, 26 February 2014
Line 33:		Line 33:
	10 and the processing time to decrease by as much as 60%		10 and the processing time to decrease by as much as 60%

			To determine if you have the necessary Update Build 2072 or newer, Programmatically: <br>
			printf(" BUILD NUMBER: %s\n",mdMUReadWriteGetBuildNumber(ReadWrite));

	==Keep Work File Location Local==		==Keep Work File Location Local==

Tim: /* Optimizing Speed: Matchcodes */

2014-02-26T16:43:25Z

Optimizing Speed: Matchcodes

← Older revision		Revision as of 16:43, 26 February 2014
Line 13:		Line 13:
	BP_MUXX_002 <br>		BP_MUXX_002 <br>
	#Matchcodes: Components: Before MatchUp dedupes, it clusters records into groups of possible matches. If your matchcode does not have any components in every used combination, it can not place records into those sub group clusters. In general, the greater number of components used in every combination, the faster the process will be.		#Matchcodes: Components: Before MatchUp dedupes, it clusters records into groups of possible matches. If your matchcode does not have any components in every used combination, it can not place records into those sub group clusters. In general, the greater number of components used in every combination, the faster the process will be.

	#Matchcode: Fuzzy: MatchUp has an extensive list of Fuzzy Options. Some are performed during the key building process (ie. Soundex) and do not slow the process down. Others are performed on the constructed matchkeys (ie. Near, Jaro, etc.) and therefore slow down the process. If the latter types are required by your process, place them in the component order below an exact component which is also used in every combination if possible.		#Matchcode: Fuzzy: MatchUp has an extensive list of Fuzzy Options. Some are performed during the key building process (ie. Soundex) and do not slow the process down. Others are performed on the constructed matchkeys (ie. Near, Jaro, etc.) and therefore slow down the process. If the latter types are required by your process, place them in the component order below an exact component which is also used in every combination if possible.