LACE - Revision history

imported>Martins at 13:12, 11 July 2013

2013-07-11T13:12:43Z

@@ Line 6: / Line 6: @@
 The project LACE aims at compiling, replicating and extending techniques that have been widely used in statistical natural language processing, and evaluating their results in UNL-based applications. As a long term enterprise, the Project has been divided in three subsidiary projects, devoted to three different types of corpus and involving, therefore, three different extraction strategies:
 *LACE<sup>pc</sup> - To extract data from parallel corpora (proceedings from the United Nations and from the European Parliament);
-*LACE<sup>hpc</sup> - To extract data from comparable semi-parallel corpora (Wikipedia) using high-performance computing; and
+*[[LACEhpc|LACE<sup>hpc</sup>]] - To extract data from comparable semi-parallel corpora (Wikipedia) using high-performance computing; and
 *LACE<sup>npc</sup> - To extract data from comparable non-parallel corpora (newspapers) using linguistically-motivated models of language automatic acquisition.

imported>Martins at 20:40, 26 February 2013

2013-02-26T20:40:08Z

← Older revision		Revision as of 20:40, 26 February 2013
Line 42:		Line 42:
	:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:		:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:
	:*0. raw data (n-grams extracted from the corpus)		:*0. raw data (n-grams extracted from the corpus)
	:*1. frequency filtered (n-grams whose frequency is equal or higher to the ratio tokens/types for all n-grams in the corpus)		:*1. frequency filtered (n-grams whose frequency is equal or higher than the ratio between tokens/types for all n-grams in the corpus)
	:*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing n-gram		:*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing frequency-filtered n-gram)
	:*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)		:*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)

imported>Martins at 20:39, 26 February 2013

2013-02-26T20:39:27Z

← Older revision		Revision as of 20:39, 26 February 2013
Line 40:		Line 40:
	:*The abridged corpus contains 100K documents from 10 languages (Chinese, English, French, German, Italian, Japanese, Polish, Portuguese, Russian and Spanish) aligned at the document level		:*The abridged corpus contains 100K documents from 10 languages (Chinese, English, French, German, Italian, Japanese, Polish, Portuguese, Russian and Spanish) aligned at the document level
	;[http://cadmos.undlfoundation.ch:8080/n-grams/ N-grams]		;[http://cadmos.undlfoundation.ch:8080/n-grams/ N-grams]
	:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different ~~sets~~:		:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different subsets:
	:*0. raw data (n-grams extracted from the corpus)		:*0. raw data (n-grams extracted from the corpus)
	:*1. frequency filtered (n-grams whose frequency is equal or higher to the ratio tokens/types for all n-grams in the corpus)		:*1. frequency filtered (n-grams whose frequency is equal or higher to the ratio tokens/types for all n-grams in the corpus)
	:*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing n-gram		:*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing n-gram
	:*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)		:*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)

imported>Martins at 20:39, 26 February 2013

2013-02-26T20:39:03Z

← Older revision		Revision as of 20:39, 26 February 2013
Line 39:		Line 39:
	:*The experimental corpus contains 10K documents from 3 languages (English, French and Japanese) aligned at the document level		:*The experimental corpus contains 10K documents from 3 languages (English, French and Japanese) aligned at the document level
	:*The abridged corpus contains 100K documents from 10 languages (Chinese, English, French, German, Italian, Japanese, Polish, Portuguese, Russian and Spanish) aligned at the document level		:*The abridged corpus contains 100K documents from 10 languages (Chinese, English, French, German, Italian, Japanese, Polish, Portuguese, Russian and Spanish) aligned at the document level
	~~:In order to test the algorithms we have also worked with a small sample of the full corpus aligned at the sentence level.~~
	;[http://cadmos.undlfoundation.ch:8080/n-grams/ N-grams]		;[http://cadmos.undlfoundation.ch:8080/n-grams/ N-grams]
	:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different sets:		:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different sets:

imported>Martins at 20:38, 26 February 2013

2013-02-26T20:38:41Z

← Older revision		Revision as of 20:38, 26 February 2013
Line 38:		Line 38:
	:The corpus is presented in two distributions:		:The corpus is presented in two distributions:
	:*The experimental corpus contains 10K documents from 3 languages (English, French and Japanese) aligned at the document level		:*The experimental corpus contains 10K documents from 3 languages (English, French and Japanese) aligned at the document level
	:*The abridged corpus contains 100K documents from 10 languages (English, French, Ja ) aligned at the document level		:*The abridged corpus contains 100K documents from 10 languages (Chinese, English, French, German, Italian, Japanese, Polish, Portuguese, Russian and Spanish) aligned at the document level
	:In order to test the algorithms we have also worked with a small sample of the full corpus aligned at the sentence level.		:In order to test the algorithms we have also worked with a small sample of the full corpus aligned at the sentence level.
	;[http://cadmos.undlfoundation.ch:8080/n-grams/ N-grams]		;[http://cadmos.undlfoundation.ch:8080/n-grams/ N-grams]

imported>Martins at 20:34, 26 February 2013

2013-02-26T20:34:25Z

← Older revision		Revision as of 20:34, 26 February 2013
Line 35:		Line 35:
	=== Files ===		=== Files ===
	;[http://www.undlfoundation.org/lace/docs/LACEhpc.pdf Original project]		;[http://www.undlfoundation.org/lace/docs/LACEhpc.pdf Original project]
	;[http://cadmos.undlfoundation.ch:8080/corpus/~~Corpus~~ Corpus (Wikipedia)]		;[http://cadmos.undlfoundation.ch:8080/corpus/ Corpus (Wikipedia)]
	:The corpus is presented in two distributions:		:The corpus is presented in two distributions:
	:*The experimental corpus contains 10K documents from 3 languages (English, French and Japanese) aligned at the document level		:*The experimental corpus contains 10K documents from 3 languages (English, French and Japanese) aligned at the document level

imported>Martins at 20:33, 26 February 2013

2013-02-26T20:33:58Z

← Older revision		Revision as of 20:33, 26 February 2013
Line 34:		Line 34:

	=== Files ===		=== Files ===
	*Project		;[http://www.undlfoundation.org/lace/docs/LACEhpc.pdf Original project]
	**[http://www.undlfoundation.org/lace/docs/LACEhpc.pdf Original project]		;[http://cadmos.undlfoundation.ch:8080/corpus/Corpus Corpus (Wikipedia)]
	*Corpus (Wikipedia)		:The corpus is presented in two distributions:
	**[http://cadmos.undlfoundation.ch:8080/corpus/~~full/ Full corpus~~]		:*The experimental corpus contains 10K documents from 3 languages (English, French and Japanese) aligned at the document level
	**:The ~~full~~ corpus is presented in two distributions:		:*The abridged corpus contains 100K documents from 10 languages (English, French, Ja ) aligned at the document level
	**experimental corpus (10K documents from 3 languages ~~= 30K documents~~);		:In order to test the algorithms we have also worked with a small sample of the full corpus aligned at the sentence level.
	**abridged corpus (100K from 10 languages ~~= 100K documents~~)		;[http://cadmos.undlfoundation.ch:8080/n-grams/ N-grams]
	**[http://cadmos.undlfoundation.ch:8080/~~corpus/aligned~~/ ~~Aligned corpus]~~		:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different sets:
	**:The aligned corpus is a very small part of the full corpus and was used to test the algorithms.		:*0. raw data (n-grams extracted from the corpus)
	*N-grams		:*1. frequency filtered (n-grams whose frequency is equal or higher to the ratio tokens/types for all n-grams in the corpus)
	*:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different sets:		:*2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing n-gram
	**:0. raw data (n-grams extracted from the corpus)		:*3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)
	**:1. frequency filtered (n-grams whose frequency is equal or higher to the ratio tokens/types for all n-grams in the corpus)
	**:2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing n-gram
	**:3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)
	**[http://cadmos.undlfoundation.ch:8080/n-grams/English/ English]
	**[http://cadmos.undlfoundation.ch:8080/n-grams/French/ French]
	**[http://cadmos.undlfoundation.ch:8080/n-grams/Portuguese/ Portuguese]

imported>Martins at 20:26, 26 February 2013

2013-02-26T20:26:54Z

← Older revision		Revision as of 20:26, 26 February 2013
Line 38:		Line 38:
	*Corpus (Wikipedia)		*Corpus (Wikipedia)
	**[http://cadmos.undlfoundation.ch:8080/corpus/full/ Full corpus]		**[http://cadmos.undlfoundation.ch:8080/corpus/full/ Full corpus]
	**:The full corpus is presented in two distributions: experimental (10K documents from 3 languages = 30K documents) ~~and~~ abridged (100K from 10 languages = 100K documents)		**:The full corpus is presented in two distributions:
			**experimental corpus (10K documents from 3 languages = 30K documents);
			**abridged corpus (100K from 10 languages = 100K documents)
	**[http://cadmos.undlfoundation.ch:8080/corpus/aligned/ Aligned corpus]		**[http://cadmos.undlfoundation.ch:8080/corpus/aligned/ Aligned corpus]
	**:The aligned corpus is a very small part of the full corpus and was used to test the algorithms.		**:The aligned corpus is a very small part of the full corpus and was used to test the algorithms.

imported>Martins at 20:25, 26 February 2013

2013-02-26T20:25:45Z

← Older revision		Revision as of 20:25, 26 February 2013
Line 38:		Line 38:
	*Corpus (Wikipedia)		*Corpus (Wikipedia)
	**[http://cadmos.undlfoundation.ch:8080/corpus/full/ Full corpus]		**[http://cadmos.undlfoundation.ch:8080/corpus/full/ Full corpus]
			**:The full corpus is presented in two distributions: experimental (10K documents from 3 languages = 30K documents) and abridged (100K from 10 languages = 100K documents)
	**[http://cadmos.undlfoundation.ch:8080/corpus/aligned/ Aligned corpus]		**[http://cadmos.undlfoundation.ch:8080/corpus/aligned/ Aligned corpus]
			**:The aligned corpus is a very small part of the full corpus and was used to test the algorithms.
	*N-grams		*N-grams
			*:The n-grams are presented in two different sets: continuous n-grams and discontinuous n-grams. Each set is further organized in four different sets:
			**:0. raw data (n-grams extracted from the corpus)
			**:1. frequency filtered (n-grams whose frequency is equal or higher to the ratio tokens/types for all n-grams in the corpus)
			**:2. redundancy filtered (frequency-filtered n-grams that cannot be subsumed by any other existing n-gram
			**:3. constituency scores (the results of applying constituency scores to the redundancy-filtered n-grams)
	**[http://cadmos.undlfoundation.ch:8080/n-grams/English/ English]		**[http://cadmos.undlfoundation.ch:8080/n-grams/English/ English]
	**[http://cadmos.undlfoundation.ch:8080/n-grams/French/ French]		**[http://cadmos.undlfoundation.ch:8080/n-grams/French/ French]
	**[http://cadmos.undlfoundation.ch:8080/n-grams/Portuguese/ Portuguese]		**[http://cadmos.undlfoundation.ch:8080/n-grams/Portuguese/ Portuguese]

imported>Martins at 20:18, 26 February 2013

2013-02-26T20:18:25Z

← Older revision		Revision as of 20:18, 26 February 2013
Line 37:		Line 37:
	**[http://www.undlfoundation.org/lace/docs/LACEhpc.pdf Original project]		**[http://www.undlfoundation.org/lace/docs/LACEhpc.pdf Original project]
	*Corpus (Wikipedia)		*Corpus (Wikipedia)
	**[http://cadmos.undlfoundation.ch:8080 corpus]		**[http://cadmos.undlfoundation.ch:8080/corpus/full/ Full corpus]
			**[http://cadmos.undlfoundation.ch:8080/corpus/aligned/ Aligned corpus]
	*N-grams		*N-grams
	**[http://cadmos.undlfoundation.ch:8080 n-grams]		**[http://cadmos.undlfoundation.ch:8080/n-grams/English/ English]
			**[http://cadmos.undlfoundation.ch:8080/n-grams/French/ French]
			**[http://cadmos.undlfoundation.ch:8080/n-grams/Portuguese/ Portuguese]