RightFielder Object:Config Example - Revision history

Admin at 15:59, 13 May 2016

2016-05-13T15:59:57Z

← Older revision		Revision as of 15:59, 13 May 2016
Line 1:		Line 1:
			[[RightFielder Object\|← RightFielder Object]]

	'''mdRightFielder.cfg Case Studies'''		'''mdRightFielder.cfg Case Studies'''

Tim at 21:53, 25 February 2014

2014-02-25T21:53:33Z

← Older revision		Revision as of 21:53, 25 February 2014
Line 19:		Line 19:
	Example: You're processing a database of car dealerships and the company recognition isn’t fielding car companies correctly. Processing might be a lot more accurate if you would modify the LeftToken table in this way:		Example: You're processing a database of car dealerships and the company recognition isn’t fielding car companies correctly. Processing might be a lot more accurate if you would modify the LeftToken table in this way:

			<b><span style='font-size:12.0pt;font-family:"Courier New"'>
	[LeftToken] <br>		[LeftToken] <br>
	FORD,C <br>		FORD,C <br>
Line 29:		Line 30:
	ALFA ROMEO,C <br>		ALFA ROMEO,C <br>
	MOTORS,C <br>		MOTORS,C <br>
			</span></b> <br>

	Notes:		Notes:
Line 82:		Line 84:
	Now that we designed our regular expression, we need to add it to the PostalCodeRegEx table:		Now that we designed our regular expression, we need to add it to the PostalCodeRegEx table:

			<b><span style='font-size:12.0pt;font-family:"Courier New"'>
	[PostalCodeRegEx] <br>		[PostalCodeRegEx] <br>
	5,(?<=^\| )[0-9Ol]{5}[- ]?([0-9Ol]{4})?(?= \|$)		5,(?<=^\| )[0-9Ol]{5}[- ]?([0-9Ol]{4})?(?= \|$)
			</span></b> <br>

	When a regular expression finds a match, the match is removed from the input data, so a later expression (which may be more fitting) will not find the match. Thus, processing order is important. However, we can’t simply sort regular expressions like we do with lookup tables. Instead, you must provide a number (the 5 in our example) which will indicate it’s place in the regular expression processing order. The lower the number, the sooner it is processed.		When a regular expression finds a match, the match is removed from the input data, so a later expression (which may be more fitting) will not find the match. Thus, processing order is important. However, we can’t simply sort regular expressions like we do with lookup tables. Instead, you must provide a number (the 5 in our example) which will indicate it’s place in the regular expression processing order. The lower the number, the sooner it is processed.
Line 96:		Line 100:
	Say your data sometimes contains a dash between the state and zip code (for example, “Braintree, MA-02184”). This data anomaly will befuddle Right Fielder’s state and zip code recognition abilities. However, we can easily fix this with an entry in the PreProcessRegEx table:		Say your data sometimes contains a dash between the state and zip code (for example, “Braintree, MA-02184”). This data anomaly will befuddle Right Fielder’s state and zip code recognition abilities. However, we can easily fix this with an entry in the PreProcessRegEx table:

			<b><span style='font-size:12.0pt;font-family:"Courier New"'>
	[PreProcessRegEx] <br>		[PreProcessRegEx] <br>
	5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2		5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2
			</span></b> <br>

	The regular expression can be broken up into these parts:		The regular expression can be broken up into these parts:

Tim: /* NOTES on cfg overrides */

2014-02-25T21:51:51Z

NOTES on cfg overrides

← Older revision		Revision as of 21:51, 25 February 2014
Line 167:		Line 167:
	Example of an invalid entry (expression will be ignored….<br>		Example of an invalid entry (expression will be ignored….<br>

	<span style='font-size:12.0pt;font-family:"Courier New"'>		<b><span style='font-size:12.0pt;font-family:"Courier New"'>
	[PreProcessRegEx]<br>		[PreProcessRegEx]<br>
	5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2		5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2
	</span> <br>		</span></b> <br>

	The <id> in the above example must be unique. If you create new expressions with the same <id> as another cfg edit or an existing mdRightFielder.dat entry, it will be ignored.<br>		The <id> in the above example must be unique. If you create new expressions with the same <id> as another cfg edit or an existing mdRightFielder.dat entry, it will be ignored.<br>
	Existing pattern <id>s in mdRightFielder.dat file start with id=10 and imcrement by 10, so to override and existing pattern, use a single digit <id>, to add a lower priority use a higher multiple		Existing pattern <id>s in mdRightFielder.dat file start with id=10 and imcrement by 10, so to override and existing pattern, use a single digit <id>, to add a lower priority use a higher multiple

Tim: /* NOTES on cfg overrides */

2014-02-25T21:47:33Z

NOTES on cfg overrides

← Older revision		Revision as of 21:47, 25 February 2014
Line 160:		Line 160:
	Example of a valid entry ….<br>		Example of a valid entry ….<br>

	<span style='font-size:12.0pt;font-family:"Courier New"'>		<b><span style='font-size:12.0pt;font-family:"Courier New"'>
	[PreProcessRegEx] <br>		[PreProcessRegEx] <br>
	5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2		5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2
	</span><br>		</span></b><br>

	Example of an invalid entry (expression will be ignored….<br>		Example of an invalid entry (expression will be ignored….<br>

			<span style='font-size:12.0pt;font-family:"Courier New"'>
	[PreProcessRegEx]<br>		[PreProcessRegEx]<br>
	~~<span style='font-size:12.0pt;font-family:"Courier New"'>~~5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2 </span> <br>		5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2
			</span> <br>

	The <id> in the above example must be unique. If you create new expressions with the same <id> as another cfg edit or an existing mdRightFielder.dat entry, it will be ignored.<br>		The <id> in the above example must be unique. If you create new expressions with the same <id> as another cfg edit or an existing mdRightFielder.dat entry, it will be ignored.<br>
	Existing pattern <id>s in mdRightFielder.dat file start with id=10 and imcrement by 10, so to override and existing pattern, use a single digit <id>, to add a lower priority use a higher multiple		Existing pattern <id>s in mdRightFielder.dat file start with id=10 and imcrement by 10, so to override and existing pattern, use a single digit <id>, to add a lower priority use a higher multiple

Tim: /* NOTES on cfg overrides */

2014-02-25T21:44:43Z

NOTES on cfg overrides

← Older revision		Revision as of 21:44, 25 February 2014
Line 166:		Line 166:

	Example of an invalid entry (expression will be ignored….<br>		Example of an invalid entry (expression will be ignored….<br>
	[PreProcessRegEx]<br>		[PreProcessRegEx]<br>
	<span style='font-size:12.0pt;font-family:"Courier New"'>5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2 </span> <br>		<span style='font-size:12.0pt;font-family:"Courier New"'>5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2 </span> <br>

	The <id> in the above example must be unique. If you create new expressions with the same <id> as another cfg edit or an existing mdRightFielder.dat entry, it will be ignored.<br>		The <id> in the above example must be unique. If you create new expressions with the same <id> as another cfg edit or an existing mdRightFielder.dat entry, it will be ignored.<br>
	Existing pattern <id>s in mdRightFielder.dat file start with id=10 and imcrement by 10, so to override and existing pattern, use a single digit <id>, to add a lower priority use a higher multiple		Existing pattern <id>s in mdRightFielder.dat file start with id=10 and imcrement by 10, so to override and existing pattern, use a single digit <id>, to add a lower priority use a higher multiple

Tim: /* NOTES on cfg overrides */

2014-02-25T21:44:06Z

NOTES on cfg overrides

← Older revision		Revision as of 21:44, 25 February 2014
Line 166:		Line 166:

	Example of an invalid entry (expression will be ignored….<br>		Example of an invalid entry (expression will be ignored….<br>
	[PreProcessRegEx]<br>		[PreProcessRegEx]<br>
	<span style='font-size:12.0pt;font-family:"Courier New"'>5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2 </span> <br>		<span style='font-size:12.0pt;font-family:"Courier New"'>5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2 </span> <br>

	The <id> in the above example must be unique. If you create new expressions with the same <id> as another cfg edit or an existing mdRightFielder.dat entry, it will be ignored.<br>		The <id> in the above example must be unique. If you create new expressions with the same <id> as another cfg edit or an existing mdRightFielder.dat entry, it will be ignored.<br>
	Existing pattern <id>s in mdRightFielder.dat file start with id=10 and imcrement by 10, so to override and existing pattern, use a single digit <id>, to add a lower priority use a higher multiple		Existing pattern <id>s in mdRightFielder.dat file start with id=10 and imcrement by 10, so to override and existing pattern, use a single digit <id>, to add a lower priority use a higher multiple

Tim at 21:43, 25 February 2014

2014-02-25T21:43:20Z

← Older revision		Revision as of 21:43, 25 February 2014
Line 160:		Line 160:
	Example of a valid entry ….<br>		Example of a valid entry ….<br>

			<span style='font-size:12.0pt;font-family:"Courier New"'>
	[PreProcessRegEx] <br>		[PreProcessRegEx] <br>
	5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2 <br>		5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2
			</span><br>

	Example of an invalid entry (expression will be ignored….<br>		Example of an invalid entry (expression will be ignored….<br>
	[PreProcessRegEx]<br>		[PreProcessRegEx]<br>
	5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2 <br>		<span style='font-size:12.0pt;font-family:"Courier New"'>5,(?<=^\| \|,\|[\|]\|\t\|\r\|\n)([a-z]{2,3})(?:[-]?)(\d{5}(?:-\d{4})?)(?=$\| \|,\|[\|]\|\t\|\r\|\n),$1 $2 </span> <br>

	The <id> in the above example must be unique. If you create new expressions with the same <id> as another cfg edit or an existing mdRightFielder.dat entry, it will be ignored.<br>		The <id> in the above example must be unique. If you create new expressions with the same <id> as another cfg edit or an existing mdRightFielder.dat entry, it will be ignored.<br>
	Existing pattern <id>s in mdRightFielder.dat file start with id=10 and imcrement by 10, so to override and existing pattern, use a single digit <id>, to add a lower priority use a higher multiple		Existing pattern <id>s in mdRightFielder.dat file start with id=10 and imcrement by 10, so to override and existing pattern, use a single digit <id>, to add a lower priority use a higher multiple

Tim: /* Regular Expression Overrides for Defined DataTypes */

2014-02-25T21:39:38Z

Regular Expression Overrides for Defined DataTypes

← Older revision		Revision as of 21:39, 25 February 2014
Line 43:		Line 43:

	<b><span style='font-size:11.0pt;font-family:"Courier New"'>		<b><span style='font-size:11.0pt;font-family:"Courier New"'>
	<span style="color:#~~000099~~">(?<=^\| )</span>		<span style="color:#0000FF">(?<=^\| )</span>
	[a-z][0-9][a-z]		[a-z][0-9][a-z]
	<span style="color:#009900">[- ]?</span>		<span style="color:#009900">[- ]?</span>
	[0-9][a-z][0-9]		[0-9][a-z][0-9]
	(?= \|$)		(?= \|$)
	<span style="color:#~~000099~~">(?= \|$)</span>		<span style="color:#0000FF">(?= \|$)</span>
	</span> </b><br>		</span> </b><br>

Line 58:		Line 58:
	Say, for example, you have data that was run through OCR software. Unfortunately, in many cases 0’s and 1’s were accidentally recognized as O’s and l’s. We can enhance Right Fielder’s recognition of abominations such as “Ol234” with this expression:		Say, for example, you have data that was run through OCR software. Unfortunately, in many cases 0’s and 1’s were accidentally recognized as O’s and l’s. We can enhance Right Fielder’s recognition of abominations such as “Ol234” with this expression:

	<b><span style='font-size:11.0pt;font-family:"Courier New"'><span style="color:#~~000099~~">(?<=^\| )</span>		<b><span style='font-size:11.0pt;font-family:"Courier New"'><span style="color:#0000FF">(?<=^\| )</span>
	[0-9Ol]{5}		[0-9Ol]{5}
	<span style="color:#009900">[- ]?</span>		<span style="color:#009900">[- ]?</span>
	([0-9Ol]{4})?		([0-9Ol]{4})?
	<span style="color:#~~000099~~">(?= \|$)</span></span> </b><br>		<span style="color:#0000FF">(?= \|$)</span></span> </b><br>

	This regular expression can be broken up into 5 parts:		This regular expression can be broken up into 5 parts:

	{\| class="wikitable"		{\| class="wikitable"
	\| style="color:#~~000099~~" \|(?<=^\| ) \|\| Like the preamble, this indicates that a break or delimiter of some sort must follow the zip code.		\| style="color:#0000FF" \|(?<=^\| ) \|\| Like the preamble, this indicates that a break or delimiter of some sort must follow the zip code.
	\|-		\|-
	\| [0-9Ol]{5} \|\| This indicates that we need to see any number, an uppercase O or a lowercase l. And we need to see 5 of them in a row.		\| [0-9Ol]{5} \|\| This indicates that we need to see any number, an uppercase O or a lowercase l. And we need to see 5 of them in a row.
Line 75:		Line 75:
	\| ([0-9Ol]{4})? \|\| This indicates that we need to see any number, an uppercase O or a lowercase l. We need to see 4 of them in a row. However, there’s catch here, because sometimes people omit the Plus 4, so the sub-expression is surrounded by parentheses and followed by the ? quantifier.		\| ([0-9Ol]{4})? \|\| This indicates that we need to see any number, an uppercase O or a lowercase l. We need to see 4 of them in a row. However, there’s catch here, because sometimes people omit the Plus 4, so the sub-expression is surrounded by parentheses and followed by the ? quantifier.
	\|-		\|-
	\| style="color:#~~000099~~" \|(?= \|$) \|\| Like the preamble, this indicates that a break or delimiter of some sort must follow the zip code.		\| style="color:#0000FF" \|(?= \|$) \|\| Like the preamble, this indicates that a break or delimiter of some sort must follow the zip code.
	\|-		\|-
	\|}		\|}
Line 90:		Line 90:

	Experienced reg-exers may be concerned about the preamble and post amble, as it would appear that we’ve forgotten to include many delimiters (ie, tab, pipe, carriage returns, etc). For the purposes of regular expression processing, all delimiters are temporarily transformed into spaces, so the only things that your regular expression really need to be looking for are spaces and the ‘start of string’ and ‘end of string’ indicators. There is one exception to this rule, and that is for the PreProcessRegEx table. Regular expressions in this table are processed on the raw data, before any transforms are done.		Experienced reg-exers may be concerned about the preamble and post amble, as it would appear that we’ve forgotten to include many delimiters (ie, tab, pipe, carriage returns, etc). For the purposes of regular expression processing, all delimiters are temporarily transformed into spaces, so the only things that your regular expression really need to be looking for are spaces and the ‘start of string’ and ‘end of string’ indicators. There is one exception to this rule, and that is for the PreProcessRegEx table. Regular expressions in this table are processed on the raw data, before any transforms are done.


	==Regular Expression Overrides for general Data Patterns==		==Regular Expression Overrides for general Data Patterns==

Tim at 21:35, 25 February 2014

2014-02-25T21:35:22Z

← Older revision		Revision as of 21:35, 25 February 2014
Line 34:		Line 34:
	*These entries will override any existing dictionary entries (for example, ‘LINCOLN’ is by default a First Name indicator). The second field, containing the ‘C’ indicates what kind of word is being described (it’s ‘token’).		*These entries will override any existing dictionary entries (for example, ‘LINCOLN’ is by default a First Name indicator). The second field, containing the ‘C’ indicates what kind of word is being described (it’s ‘token’).
	*Only one token can be used per entry. Each table has different tokens that can be used in it, see the mdRightFielder.cfg for details.		*Only one token can be used per entry. Each table has different tokens that can be used in it, see the mdRightFielder.cfg for details.


	==Regular Expression Overrides for Defined DataTypes==		==Regular Expression Overrides for Defined DataTypes==
Line 89:		Line 90:

	Experienced reg-exers may be concerned about the preamble and post amble, as it would appear that we’ve forgotten to include many delimiters (ie, tab, pipe, carriage returns, etc). For the purposes of regular expression processing, all delimiters are temporarily transformed into spaces, so the only things that your regular expression really need to be looking for are spaces and the ‘start of string’ and ‘end of string’ indicators. There is one exception to this rule, and that is for the PreProcessRegEx table. Regular expressions in this table are processed on the raw data, before any transforms are done.		Experienced reg-exers may be concerned about the preamble and post amble, as it would appear that we’ve forgotten to include many delimiters (ie, tab, pipe, carriage returns, etc). For the purposes of regular expression processing, all delimiters are temporarily transformed into spaces, so the only things that your regular expression really need to be looking for are spaces and the ‘start of string’ and ‘end of string’ indicators. There is one exception to this rule, and that is for the PreProcessRegEx table. Regular expressions in this table are processed on the raw data, before any transforms are done.


	==Regular Expression Overrides for general Data Patterns==		==Regular Expression Overrides for general Data Patterns==
Line 104:		Line 106:
	(\d{5}(?:-\d{4})?) The Zip Code (and optional Plus 4).		(\d{5}(?:-\d{4})?) The Zip Code (and optional Plus 4).
	(?=$\| \|,\|[\|]\|\t\|\r\|\n) The post amble.		(?=$\| \|,\|[\|]\|\t\|\r\|\n) The post amble.


	==Pattern Table Overrides==		==Pattern Table Overrides==
Line 109:		Line 112:

	This type of override is more complex and should only be attempted with the help of a Melissa Data support engineer. While the other .cfg file overrides can affect how the token is identified, editing this type can affect how other adjoining tokens are identified, making it possible for RightFielder to inconsistently identify tokens you did not intend to edit.		This type of override is more complex and should only be attempted with the help of a Melissa Data support engineer. While the other .cfg file overrides can affect how the token is identified, editing this type can affect how other adjoining tokens are identified, making it possible for RightFielder to inconsistently identify tokens you did not intend to edit.


	==Example mdRightFielder.cfg Overrides==		==Example mdRightFielder.cfg Overrides==
Line 150:		Line 154:
	<nowiki/>PAGER <br>		<nowiki/>PAGER <br>
	<nowiki/>BLACKBERRY <br>		<nowiki/>BLACKBERRY <br>


	==NOTES on cfg overrides==		==NOTES on cfg overrides==

Tim: /* Pattern Table Overrides */

2014-02-25T21:34:18Z

Pattern Table Overrides

← Older revision		Revision as of 21:34, 25 February 2014
Line 108:		Line 108:
	The addition or removal of patterns of words and phrases. Words and phrases are first identified via Lookup Tables and assigned tokens (specified in the Lookup Table itself). Sequences of tokens (patterns) are matched to entries in this table and transformed into output data.		The addition or removal of patterns of words and phrases. Words and phrases are first identified via Lookup Tables and assigned tokens (specified in the Lookup Table itself). Sequences of tokens (patterns) are matched to entries in this table and transformed into output data.

			This type of override is more complex and should only be attempted with the help of a Melissa Data support engineer. While the other .cfg file overrides can affect how the token is identified, editing this type can affect how other adjoining tokens are identified, making it possible for RightFielder to inconsistently identify tokens you did not intend to edit.

	==Example mdRightFielder.cfg Overrides==		==Example mdRightFielder.cfg Overrides==