Frequent term distribution measures for dataset profiling

De Roeck, Anne; Sarkar, Avik and Garthwaite, Paul H. (2004). Frequent term distribution measures for dataset profiling. In: Fourth International Conference on Language Resources and Evaluation (LREC 04), 26-28 May 2004, Lisbon, Portugal.

URL:	http://mcs.open.ac.uk/anr29/PapersLinks/LREC04DeRo...
Google Scholar:	Look up in Google Scholar

Abstract

We motivate the need for dataset profiling in the context of evaluation, and show that textual datasets differ in ways that challenge assumptions about the applicability of techniques. We set out some criteria for useful profiling measures. We argue that distribution patterns of frequent words are useful in profiling genre, and report on a series of experiments with χ² based measures on the TIPSTER collection, and on textual intranet data. Findings show substantial differences in the distribution of very frequent terms across datasets.

Item Type:	Conference Item
Copyright Holders:	2004 The Authors
Academic Unit/Department:	Mathematics, Computing and Technology Mathematics, Computing and Technology > Computing Mathematics, Computing and Technology > Mathematics and Statistics
Interdisciplinary Research Centre:	Centre for Research in Computing (CRC)
Item ID:	22601
Depositing User:	Sarah Frain
Date Deposited:	17 Aug 2010 11:46
Last Modified:	02 Dec 2010 21:01
URI:	http://oro.open.ac.uk/id/eprint/22601

Actions (login may be required)

	View Item
RDF+XMLBibTeXRDF+N-TriplesJSONDublin CoreAtomOAI-ORE Resource Map (Atom Format)Simple MetadataReferMETSOAI-ORE Resource Map (RDF Format)HTML CitationASCII CitationMultiline CSVRefMan RIS Format (UTF-8)OpenURL ContextObjectEndNoteMODSOpenURL ContextObject in SpanMPEG-21 DIDLEP3 XMLRefMan RIS FormatRDF+N3Eprints Application Profile	Public: Report issue / request change