LACEhpc - Revision history

imported>Martins at 11:05, 16 September 2013

2013-09-16T11:05:13Z

← Older revision		Revision as of 11:05, 16 September 2013
Line 20:		Line 20:
	The raw corpus is presented in two distributions at [http://cadmos.undlfoundation.ch:8080/corpus/]:		The raw corpus is presented in two distributions at [http://cadmos.undlfoundation.ch:8080/corpus/]:
	*The experimental corpus contains 10K documents from 3 languages (English, French and Japanese) aligned at the document level		*The experimental corpus contains 10K documents from 3 languages (English, French and Japanese) aligned at the document level
	*The abridged corpus contains 100K documents from 10 languages (~~Chinese~~, English, French, German, Italian, Japanese, Polish, Portuguese, Russian and Spanish) aligned at the document level		*The abridged corpus contains 100K documents from 10 languages (Dutch, English, French, German, Italian, Japanese, Polish, Portuguese, Russian and Spanish) aligned at the document level.

	== N-grams ==		== N-grams ==

imported>Martins at 18:45, 11 July 2013

2013-07-11T18:45:27Z

← Older revision		Revision as of 18:45, 11 July 2013
Line 29:		Line 29:
	*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing frequency-filtered n-gram)		*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing frequency-filtered n-gram)
	*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)		*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)
	The n-grams may be downloaded from [http://cadmos.undlfoundation.ch:8080/n-grams/]		The latest release of the n-grams extracted in the project LACE<sup>hpc</sup> may be downloaded from [http://cadmos.undlfoundation.ch:8080/n-grams/]

	== Anchors ==		== Anchors ==

imported>Martins at 17:32, 11 July 2013

2013-07-11T17:32:21Z

← Older revision		Revision as of 17:32, 11 July 2013
Line 1:		Line 1:
	The project LACE<sup>hpc</sup> aims at designing and implementing efficient high-performance computing methods for extracting monolingual and multilingual resources from comparable non-parallel corpora.		The project LACE<sup>hpc</sup> is part of the project [[LACE]] and aims at designing and implementing efficient high-performance computing methods for extracting monolingual and multilingual resources from comparable non-parallel corpora.

	== Goals ==		== Goals ==

imported>Martins at 17:30, 11 July 2013

2013-07-11T17:30:35Z

← Older revision		Revision as of 17:30, 11 July 2013
Line 52:		Line 52:

	== Files ==		== Files ==
	;[http://www.undlfoundation.org/lace/docs/LACEhpc.pdf Original project]		*[http://www.undlfoundation.org/lace/docs/LACEhpc.pdf Original project]


	== Support ==		== Support ==

imported>Martins at 17:29, 11 July 2013

2013-07-11T17:29:48Z

← Older revision		Revision as of 17:29, 11 July 2013
Line 23:		Line 23:

	== N-grams ==		== N-grams ==
	''main article: [[N-gram]]''<br />		''main article: [[N-gram]]''<br /><br />
	The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:		The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:
	*0. raw data (n-grams extracted from the corpus)		*0. raw data (n-grams extracted from the corpus)

imported>Martins at 17:29, 11 July 2013

2013-07-11T17:29:36Z

← Older revision		Revision as of 17:29, 11 July 2013
Line 23:		Line 23:

	== N-grams ==		== N-grams ==
	''main article: [[N-gram]]''		''main article: [[N-gram]]''<br />
	The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:		The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:
	*0. raw data (n-grams extracted from the corpus)		*0. raw data (n-grams extracted from the corpus)

imported>Martins at 17:29, 11 July 2013

2013-07-11T17:29:25Z

← Older revision		Revision as of 17:29, 11 July 2013
Line 24:		Line 24:
	== N-grams ==		== N-grams ==
	''main article: [[N-gram]]''		''main article: [[N-gram]]''
	:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:		The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:
	:*0. raw data (n-grams extracted from the corpus)		*0. raw data (n-grams extracted from the corpus)
	:*1. frequency filtered (n-grams whose frequency is equal or higher than the ratio between tokens/types for all n-grams in the corpus)		*1. frequency filtered (n-grams whose frequency is equal or higher than the ratio between tokens/types for all n-grams in the corpus)
	:*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing frequency-filtered n-gram)		*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing frequency-filtered n-gram)
	:*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)		*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)
	The n-grams may be downloaded from [http://cadmos.undlfoundation.ch:8080/n-grams/]		The n-grams may be downloaded from [http://cadmos.undlfoundation.ch:8080/n-grams/]

imported>Martins at 17:29, 11 July 2013

2013-07-11T17:29:05Z

← Older revision		Revision as of 17:29, 11 July 2013
Line 24:		Line 24:
	== N-grams ==		== N-grams ==
	''main article: [[N-gram]]''		''main article: [[N-gram]]''

	~~;[http://cadmos.undlfoundation.ch:8080/n-grams/ N-grams]~~
	:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:		:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:
	:*0. raw data (n-grams extracted from the corpus)		:*0. raw data (n-grams extracted from the corpus)
Line 31:		Line 29:
	:*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing frequency-filtered n-gram)		:*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing frequency-filtered n-gram)
	:*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)		:*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)
			The n-grams may be downloaded from [http://cadmos.undlfoundation.ch:8080/n-grams/]

	== Anchors ==		== Anchors ==
	''main article: [[Anchor]]''		''main article: [[Anchor]]''

imported>Martins at 17:23, 11 July 2013

2013-07-11T17:23:53Z

← Older revision		Revision as of 17:23, 11 July 2013
Line 10:		Line 10:
	In addition to HPC-oriented algorithms, the project is expected to deliver several different monolingual and bilingual databases, as well as aligned corpora and translation memories, which are important assets for natural language processing and fundamental resources for research in Linguistics and Computational Linguistics.		In addition to HPC-oriented algorithms, the project is expected to deliver several different monolingual and bilingual databases, as well as aligned corpora and translation memories, which are important assets for natural language processing and fundamental resources for research in Linguistics and Computational Linguistics.

	~~== Methodology ==~~		== Corpus ==
	=== Corpus ===
	In order to extract the data, we have proposed the use of the [http://www.wikipedia.org Wikipedia] as our corpus.<br />		In order to extract the data, we have proposed the use of the [http://www.wikipedia.org Wikipedia] as our corpus.<br />
	The choice for the Wikipedia derives from five main reasons:		The choice for the Wikipedia derives from five main reasons:
Line 23:		Line 22:
	*The abridged corpus contains 100K documents from 10 languages (Chinese, English, French, German, Italian, Japanese, Polish, Portuguese, Russian and Spanish) aligned at the document level		*The abridged corpus contains 100K documents from 10 languages (Chinese, English, French, German, Italian, Japanese, Polish, Portuguese, Russian and Spanish) aligned at the document level

	== ~~Definitions~~ ==		== N-grams ==
	~~===~~ N-gram ~~===~~		''main article: [[N-gram]]''
	~~In the scope of the project LACE, an n-gram is a linear structure of n strings composed entirely of alphabetical characters or hyphen (i.e.,~~ [a..~~zA..Z-]) isolated by blank space, punctuation marks, end of sentence, and other signs such as ([.,;~~:~~!?()"<>]). Strings containing digits or any non-alphabetical characters (such as [@_#$%~~/~~]) were ignored<ref>This means that the input string “abc def g-hi jkl m1 234 nop qrs tu_vw” was said to have:~~
	*six 1-grams (“abc”, “def”, “g-hi”, “jkl” “nop”, “qrs”)		;[http://cadmos.undlfoundation.ch:8080/n-grams/ N-grams]
	*four 2-grams (“abc def”, “def g-hi”, “g-hi jkl”, “nop qrs”)		:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:
	*two 3-grams (“abc def g-hi”, “def g-hi jkl”)		:*0. raw data (n-grams extracted from the corpus)
	*one 4-gram (“abc def g-hi jkl”).		:*1. frequency filtered (n-grams whose frequency is equal or higher than the ratio between tokens/types for all n-grams in the corpus)
	~~The strings "m1", "234" and "tu_vw" were not considered valid and, therefore, any~~ n-grams ~~including them were excluded from the results.<~~/~~ref>.<br /><br />~~		:*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing frequency-filtered n-gram)
	~~In the scope of the project LACE, n~~-grams ~~are said to be "continuous" or "discontinuous":~~		:*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)
	*a '''continuous n-gram''' is an invariant sequence of n immediately adjacent items, i.e., without any other items in-between;		== Anchors ==
	*a '''discontinuous n-gram''' is an open pattern: ~~it is a continuous~~ n-~~gram where some items~~ are ~~variable, i.e., a sequence of x and y (x+y=n) items where x items come~~ in ~~the same position and are isolated by the same number of y in-between items. A discontinuous n-gram is valid if, for the same x, there are at least~~ two different ~~y, otherwise we consider it noise. A discontinuous~~ n-gram may have one or more discontinuities, but due to the necessity of defining its external boundaries, we limit the notion of discontinuity to the internal items of an n-gram. In our notation, discontinuities are represented by the place holder "."<ref>In the example above, there are two discontinuous 3-grams ~~(“abc . g-hi”, “def . jkl”)~~ and ~~three discontinuous 4-grams (“abc . . jkl”, “abc def . jkl”, “abc . g-hi jkl”).</ref> Given the precondition of external boundaries,~~ discontinuous n-grams ~~should meet the requirement: n>2~~.~~<br /><br />~~		''main article: [[Anchor]]''
	In the scope of the project LACE, N-grams are considered to be '''linguistically-relevant''' if they are frequent, non-redundant, of a certain length, and may figure as syntactic and semantic units, according to the following criteria:
	*Length:		== MWE ==
	*~~:In the context of LACE, we treated both continuous and discontinuous~~ n-grams ~~with up to 7 items, i.e., where 1 ≤ n ≤ 7.~~		''main article: [[MWE]]''
	*Frequency:
	*~~:In the context of LACE, we considered an~~ n-~~gram to be frequent in the corpus if its~~ frequency ~~of occurrence~~ is equal or higher than the ratio between tokens ~~and~~ types~~, where “tokens” is the total number of~~ n-grams in the corpus, and “types” is the number of distinct n-grams in the corpus. For instance: given a corpus with 5,000 occurrences of distinct 1,000 unigrams, a 1-gram is considered relevant if, and only if, it occurs 5 or more times.
	*Redundancy:
	*~~:In the context of LACE, we intend an~~ n-~~gram to~~ be ~~redundant if it is~~ subsumed by any other x-~~gram, where x ≥~~ n~~. In that sense, the 1~~-gram “a” is considered unique if, and only if, there is at least one context “x a” and at least one context “a y”, where “x a” and “a y” have not been defined as an n-gram according to the criteria above concerning length and frequency. ~~For instance,~~ the ~~items “Sri” and “Lanka” are not considered~~ to ~~be 1-grams because they cannot occur in isolation: they always appear as part of~~ the 2-~~gram “Sri Lanka” (i.e., there is no context in the corpus in which we have “Sri” but not “Lanka”). The same applies for discontinuous~~ n-grams: ~~the sequence “a . . d” is a 4-gram if it is not subsumed by the 4-gram “a b . d”, i.e., if there is at least one “a x . d” where x ≠ b.~~

	~~=== MWE (multiword expression) ===~~
	~~=== Anchor ===~~
	~~Anchors are~~

	== Participants ==		== Participants ==
Line 61:		Line 54:
	;[http://www.undlfoundation.org/lace/docs/LACEhpc.pdf Original project]		;[http://www.undlfoundation.org/lace/docs/LACEhpc.pdf Original project]

	~~;[http://cadmos.undlfoundation.ch:8080/n-grams/ N-grams]~~
	~~:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:~~
	~~:*0. raw data (n-grams extracted from the corpus)~~
	~~:*1. frequency filtered (n-grams whose frequency is equal or higher than the ratio between tokens/types for all n-grams in the corpus)~~
	~~:*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing frequency-filtered n-gram)~~
	~~:*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)~~

	== Support ==		== Support ==

imported>Martins at 17:17, 11 July 2013

2013-07-11T17:17:14Z

← Older revision		Revision as of 17:17, 11 July 2013
Line 30:		Line 30:
	*two 3-grams (“abc def g-hi”, “def g-hi jkl”)		*two 3-grams (“abc def g-hi”, “def g-hi jkl”)
	*one 4-gram (“abc def g-hi jkl”).		*one 4-gram (“abc def g-hi jkl”).
	The strings "m1", "234" and "tu_vw" were not considered valid and, therefore, any n-grams including them were excluded from the results.</ref>.<br />		The strings "m1", "234" and "tu_vw" were not considered valid and, therefore, any n-grams including them were excluded from the results.</ref>.<br /><br />
	In the scope of the project LACE, n-grams are said to be "continuous" or "discontinuous":		In the scope of the project LACE, n-grams are said to be "continuous" or "discontinuous":
	*a '''continuous n-gram''' is an invariant sequence of n immediately adjacent items, i.e., without any other items in-between;		*a '''continuous n-gram''' is an invariant sequence of n immediately adjacent items, i.e., without any other items in-between;
	*a '''discontinuous n-gram''' is an open pattern: it is a continuous n-gram where some items are variable, i.e., a sequence of x and y (x+y=n) items where x items come in the same position and are isolated by the same number of y in-between items. A discontinuous n-gram is valid if, for the same x, there are at least two different y, otherwise we consider it noise. A discontinuous n-gram may have one or more discontinuities, but due to the necessity of defining its external boundaries, we limit the notion of discontinuity to the internal items of an n-gram. In our notation, discontinuities are represented by the place holder "."<ref>In the example above, there are two discontinuous 3-grams (“abc . g-hi”, “def . jkl”) and three discontinuous 4-grams (“abc . . jkl”, “abc def . jkl”, “abc . g-hi jkl”).</ref> Given the precondition of external boundaries, discontinuous n-grams should meet the requirement: n>2.<br />		*a '''discontinuous n-gram''' is an open pattern: it is a continuous n-gram where some items are variable, i.e., a sequence of x and y (x+y=n) items where x items come in the same position and are isolated by the same number of y in-between items. A discontinuous n-gram is valid if, for the same x, there are at least two different y, otherwise we consider it noise. A discontinuous n-gram may have one or more discontinuities, but due to the necessity of defining its external boundaries, we limit the notion of discontinuity to the internal items of an n-gram. In our notation, discontinuities are represented by the place holder "."<ref>In the example above, there are two discontinuous 3-grams (“abc . g-hi”, “def . jkl”) and three discontinuous 4-grams (“abc . . jkl”, “abc def . jkl”, “abc . g-hi jkl”).</ref> Given the precondition of external boundaries, discontinuous n-grams should meet the requirement: n>2.<br /><br />
	In the scope of the project LACE, N-grams are considered to be '''linguistically-relevant''' if they are frequent, non-redundant, of a certain length, and may figure as syntactic and semantic units, according to the following criteria:		In the scope of the project LACE, N-grams are considered to be '''linguistically-relevant''' if they are frequent, non-redundant, of a certain length, and may figure as syntactic and semantic units, according to the following criteria:
	*Length:		*Length: