Tokenization - Revision history

imported>Martins at 11:28, 30 January 2015

2015-01-30T11:28:12Z

← Older revision		Revision as of 11:28, 30 January 2015
Line 202:		Line 202:
	:tokenization output: [ab][c][d][e] (the first combination of shortest temporary nodes from left to right after [a][b][c][d][e], which was blocked by the disambiguation rule. Note that [ab] is not in the dictionary and, therefore, will be treated as TEMP)		:tokenization output: [ab][c][d][e] (the first combination of shortest temporary nodes from left to right after [a][b][c][d][e], which was blocked by the disambiguation rule. Note that [ab] is not in the dictionary and, therefore, will be treated as TEMP)

	=== <nowiki>#FINAL</nowiki> ===		=== <nowiki>#FINAL AND #PREFERRED</nowiki> ===
	The attribute #FINAL is used to ~~create context-sensitive rules,~~ as described in [[~~D-rules##FINAL\|~~D-rules]].		The attribute #FINAL and #PREFERRED are used to alter the default order of replacements during tokenization as described in [[D-rules]].

	As the tokenization algorithm goes from left to right, the system will try to replace first the rightmost entries in case of blocking patterns. In order to prevent this, the attribute #FINAL should be introduced to the entries that are not expected to be replaced.		As the tokenization algorithm goes from left to right, the system will try to replace first the rightmost entries in case of blocking patterns. In order to prevent this, the attribute #FINAL should be introduced to the entries that are not expected to be replaced, whereas #PREFERRED is introduced to the entries that are expected to be replaced only if there is no other alternative.

	;Case 14		;Case 14
Line 211:		Line 211:
	:disambiguation rule: ("/.{2,5}/")=0; (A1)(B1)=0;		:disambiguation rule: ("/.{2,5}/")=0; (A1)(B1)=0;
	:tokenization output: [a,A1][b,B2][c,C1][d,D1][e,E1] (Since the sequence (A1)(B1) is prohibited by the D-rule, the system tries to replace first B1, which is the rightmost entry)		:tokenization output: [a,A1][b,B2][c,C1][d,D1][e,E1] (Since the sequence (A1)(B1) is prohibited by the D-rule, the system tries to replace first B1, which is the rightmost entry)
	:disambiguation rule: ("/.{2,5}/")=0; (A1)(B1,#~~FINAL~~)=0;		:disambiguation rule: ("/.{2,5}/")=0; (A1)(B1,#PREFERRED)=0;
	:tokenization output: [a,A2][b,B1][c,C1][d,D1][e,E1] (Since the sequence (A1)(B1) is prohibited by the D-rule, the system tries to replace first A1, which is the rightmost entry without the attribute #~~FINAL~~)		:tokenization output: [a,A2][b,B1][c,C1][d,D1][e,E1] (Since the sequence (A1)(B1) is prohibited by the D-rule, the system tries to replace first A1, which is the rightmost entry without the attribute #PREFERRED)

	=== Retokenization ===		=== Retokenization ===

imported>Dgaiv at 00:09, 10 March 2014

2014-03-10T00:09:07Z

← Older revision		Revision as of 00:09, 10 March 2014
Line 1:		Line 1:
	Tokenization is the process of segmenting the input into [[nodes]], i.e., the tokens or processing units of the UNL framework.<br />		Tokenization is the process of segmenting the input into [[nodes]], i.e., the tokens or processing units of the UNL framework.<br />
	In [[UNLization]], tokenization corresponds to splitting the natural language input into lexical items, i.e., to recognize that the string "Barking dogs seldom bite" contains 7 tokens: [barking][ ][dogs][ ][seldom][ ][bite], if these entries are provided in the dictionary.<br />		In [[UNLization]], tokenization corresponds to splitting the natural language input into lexical items, i.e., to recognize that the string "Barking dogs seldom bite" contains 7 tokens: [barking][ ][dogs][ ][seldom][ ][bite], if these entries are provided in the [[dictionary]].<br />
	In [[NLization]], tokenization corresponds to splitting the UNL graph into UWs, relations and attributes, i.e., to recognize that, in the graph "agt(bite,dog.@pl)mod(dog.@pl,bark)tim(bite,seldom)", there are three relations ("agt", "mod" and "tim"), one attribute ("@pl") and 4 UWs ("bite", "dog", "bark" and "seldom").		In [[NLization]], tokenization corresponds to splitting the UNL graph into UWs, relations and attributes, i.e., to recognize that, in the graph "agt(bite,dog.@pl)mod(dog.@pl,bark)tim(bite,seldom)", there are three relations ("agt", "mod" and "tim"), one attribute ("@pl") and 4 UWs ("bite", "dog", "bark" and "seldom").

imported>Martins at 13:05, 8 November 2013

2013-11-08T13:05:50Z

← Older revision		Revision as of 13:05, 8 November 2013
Line 178:		Line 178:
	;Case 10		;Case 10
	:input: "abcde"		:input: "abcde"
	:disambiguation rule: ("/.{2,5}/")=0; (A2)(B2)=1; (A2)(B1)=1; (induces the sequence [A2][B1])		:disambiguation rule: ("/.{2,5}/")=0; (A2)(B2)=1; (A2)(B1)=1; (induces the sequence [A2][B2] and [A2][B1])
	:tokenization output: [a,A2][b,B2][c,C1][d,D1][e,E1] (the rule [A2][B2] has the ~~some~~ probability as [A2][B1] and comes first in the grammar)		:tokenization output: [a,A2][b,B2][c,C1][d,D1][e,E1] (the rule [A2][B2] has the same probability as [A2][B1] and comes first in the grammar)
	;Case 11		;Case 11
	:input: "abcde"		:input: "abcde"
	:disambiguation rule: ("/.{2,5}/")=0; (A2)(B2)=1; (A2)(B1)=2; (induces the sequence [A2][B1])		:disambiguation rule: ("/.{2,5}/")=0; (A2)(B2)=1; (A2)(B1)=2; (induces the sequence [A2][B2] and [A2][B1])
	:tokenization output: [a,A2][b,B1][c,C1][d,D1][e,E1] (the priority of the rule [A2][B1] is higher than [A2][B2])		:tokenization output: [a,A2][b,B1][c,C1][d,D1][e,E1] (the priority of the rule [A2][B1] is higher than [A2][B2])
	;Case 12		;Case 12

imported>Martins at 15:55, 1 October 2013

2013-10-01T15:55:51Z

← Older revision		Revision as of 15:55, 1 October 2013
Line 209:		Line 209:
	;Case 14		;Case 14
	:input: "abcde"		:input: "abcde"
	:disambiguation rule: ("/.{2,5}/")=0; (A2)(B2)=1;		:disambiguation rule: ("/.{2,5}/")=0; (A1)(B1)=0;
	:tokenization output: [a,A2][b,B2][c,C1][d,D1][e,E1] (the ~~same as case 11; compare with case 12~~ - ~~note that~~ the rule (B1,#FINAL)~~(C2) does not induce the choice of~~ B1~~; it applies only to C2~~, i.e., ~~it indicates that~~, ~~if B1~~, ~~then C2~~)		:tokenization output: [a,A1][b,B2][c,C1][d,D1][e,E1] (Since the sequence (A1)(B1) is prohibited by the D-rule, the system tries to replace first B1, which is the rightmost entry)
			:disambiguation rule: ("/.{2,5}/")=0; (A1)(B1,#FINAL)=0;
			:tokenization output: [a,A2][b,B1][c,C1][d,D1][e,E1] (Since the sequence (A1)(B1) is prohibited by the D-rule, the system tries to replace first A1, which is the rightmost entry without the attribute #FINAL)

	=== Retokenization ===		=== Retokenization ===

imported>Martins at 15:49, 1 October 2013

2013-10-01T15:49:20Z

← Older revision		Revision as of 15:49, 1 October 2013
Line 207:		Line 207:
	As the tokenization algorithm goes from left to right, the system will try to replace first the rightmost entries in case of blocking patterns. In order to prevent this, the attribute #FINAL should be introduced to the entries that are not expected to be replaced.		As the tokenization algorithm goes from left to right, the system will try to replace first the rightmost entries in case of blocking patterns. In order to prevent this, the attribute #FINAL should be introduced to the entries that are not expected to be replaced.

			;Case 14
	:input: "abcde"		:input: "abcde"
	:disambiguation rule: ("/.{2,5}/")=0; (A2)(B2)=1;		:disambiguation rule: ("/.{2,5}/")=0; (A2)(B2)=1;

imported>Martins at 15:48, 1 October 2013

2013-10-01T15:48:32Z

← Older revision		Revision as of 15:48, 1 October 2013
Line 203:		Line 203:

	=== <nowiki>#FINAL</nowiki> ===		=== <nowiki>#FINAL</nowiki> ===
	The attribute #FINAL is used to create context-sensitive rules, as described in [[D-rules##FINAL]].		The attribute #FINAL is used to create context-sensitive rules, as described in [[D-rules##FINAL\|D-rules]].

	As the tokenization algorithm goes from left to right, the system will try to replace first the rightmost entries in case of blocking patterns. In order to prevent this, the attribute #FINAL should be introduced to the entries that are not expected to be replaced.		As the tokenization algorithm goes from left to right, the system will try to replace first the rightmost entries in case of blocking patterns. In order to prevent this, the attribute #FINAL should be introduced to the entries that are not expected to be replaced.

imported>Martins at 15:48, 1 October 2013

2013-10-01T15:48:07Z

← Older revision		Revision as of 15:48, 1 October 2013
Line 203:		Line 203:

	=== <nowiki>#FINAL</nowiki> ===		=== <nowiki>#FINAL</nowiki> ===
	The attribute #FINAL is used to create context-sensitive rules, as described in [[D-rules]].		The attribute #FINAL is used to create context-sensitive rules, as described in [[D-rules##FINAL]].

	As the tokenization algorithm goes from left to right, the system will try to replace first the rightmost entries in case of blocking patterns. In order to prevent this, the attribute #FINAL should be introduced to the entries that are not expected to be replaced.		As the tokenization algorithm goes from left to right, the system will try to replace first the rightmost entries in case of blocking patterns. In order to prevent this, the attribute #FINAL should be introduced to the entries that are not expected to be replaced.

imported>Martins at 15:47, 1 October 2013

2013-10-01T15:47:34Z

← Older revision		Revision as of 15:47, 1 October 2013
Line 203:		Line 203:

	=== <nowiki>#FINAL</nowiki> ===		=== <nowiki>#FINAL</nowiki> ===
	The attribute #FINAL is used to create context-sensitive rules.		The attribute #FINAL is used to create context-sensitive rules, as described in [[D-rules]].

	~~According to tokenization algorithm,~~


			As the tokenization algorithm goes from left to right, the system will try to replace first the rightmost entries in case of blocking patterns. In order to prevent this, the attribute #FINAL should be introduced to the entries that are not expected to be replaced.

	:input: "abcde"		:input: "abcde"
	:disambiguation rule: ("/.{2,5}/")=0; (A2)(B2)=1; ~~(B1,#FINAL)(C2)=5; (if B1 is fixed, C2 must be chosen)~~		:disambiguation rule: ("/.{2,5}/")=0; (A2)(B2)=1;
	:tokenization output: [a,A2][b,B2][c,C1][d,D1][e,E1] (the same as case 11; compare with case 12 - note that the rule (B1,#FINAL)(C2) does not induce the choice of B1; it applies only to C2, i.e., it indicates that, if B1, then C2)		:tokenization output: [a,A2][b,B2][c,C1][d,D1][e,E1] (the same as case 11; compare with case 12 - note that the rule (B1,#FINAL)(C2) does not induce the choice of B1; it applies only to C2, i.e., it indicates that, if B1, then C2)


	=== Retokenization ===		=== Retokenization ===

imported>Martins at 15:40, 1 October 2013

2013-10-01T15:40:13Z

← Older revision		Revision as of 15:40, 1 October 2013
Line 202:		Line 202:
	:tokenization output: [ab][c][d][e] (the first combination of shortest temporary nodes from left to right after [a][b][c][d][e], which was blocked by the disambiguation rule. Note that [ab] is not in the dictionary and, therefore, will be treated as TEMP)		:tokenization output: [ab][c][d][e] (the first combination of shortest temporary nodes from left to right after [a][b][c][d][e], which was blocked by the disambiguation rule. Note that [ab] is not in the dictionary and, therefore, will be treated as TEMP)

	=== #FINAL ===		=== <nowiki>#FINAL</nowiki> ===
			The attribute #FINAL is used to create context-sensitive rules.

			According to tokenization algorithm,



			:input: "abcde"
			:disambiguation rule: ("/.{2,5}/")=0; (A2)(B2)=1; (B1,#FINAL)(C2)=5; (if B1 is fixed, C2 must be chosen)
			:tokenization output: [a,A2][b,B2][c,C1][d,D1][e,E1] (the same as case 11; compare with case 12 - note that the rule (B1,#FINAL)(C2) does not induce the choice of B1; it applies only to C2, i.e., it indicates that, if B1, then C2)

imported>Martins at 15:36, 1 October 2013

2013-10-01T15:36:43Z

← Older revision		Revision as of 15:36, 1 October 2013
Line 189:		Line 189:
	:tokenization output: [a,A1][b,B1][c,C2][d,D1][e,E1] (the priority of the rule [B1][C2] is higher than [A2][B2])		:tokenization output: [a,A1][b,B1][c,C2][d,D1][e,E1] (the priority of the rule [B1][C2] is higher than [A2][B2])
	The attribute #FINAL is used to indicate context		The attribute #FINAL is used to indicate context
	~~;Case 13~~
	~~:input: "abcde"~~
	~~:disambiguation rule: ("/.{2,5}/")=0; (A2)(B2)=1; (B1,#FINAL)(C2)=5; (if B1 is fixed, C2 must be chosen)~~
	:tokenization output: [a,A2][b,B2][c,C1][d,D1][e,E1] (the same as case 11; compare with case 12 - note that the rule (B1,#FINAL)(C2) does not induce the choice of B1; it applies only to C2, i.e., it indicates that, if B1, then C2)
	The tokenization algorithm backtracks in case of dead-ends.		The tokenization algorithm backtracks in case of dead-ends.
	In case of backtracking, the tokenization starts merging nodes from left to right, but considering their size, i.e., the shortest combinations appear first.		In case of backtracking, the tokenization starts merging nodes from left to right, but considering their size, i.e., the shortest combinations appear first.
	;Case 14		;Case 13
	:input: "abcde"		:input: "abcde"
	:dictionary:		:dictionary:
Line 205:		Line 201:
	:disambiguation rule: ("a")("b")("c")("d")("e")=0;		:disambiguation rule: ("a")("b")("c")("d")("e")=0;
	:tokenization output: [ab][c][d][e] (the first combination of shortest temporary nodes from left to right after [a][b][c][d][e], which was blocked by the disambiguation rule. Note that [ab] is not in the dictionary and, therefore, will be treated as TEMP)		:tokenization output: [ab][c][d][e] (the first combination of shortest temporary nodes from left to right after [a][b][c][d][e], which was blocked by the disambiguation rule. Note that [ab] is not in the dictionary and, therefore, will be treated as TEMP)

			=== #FINAL ===



	=== Retokenization ===		=== Retokenization ===