Frequent term distribution measures for dataset profiling

De Roeck, Anne; Sarkar, Avik and Garthwaite, Paul H. (2004). Frequent term distribution measures for dataset profiling. In: Proceedings of the Fourth International Conference on Language Resources and Evaluation (Lino, M. T.; Xavier, M. F.; Ferreira, F.; Costa, R. and Silva, R. eds.), ELRA, Paris, pp. 1647–1650.

URL: http://mcs.open.ac.uk/anr29/PapersLinks/LREC04DeRo...

Abstract

We motivate the need for dataset profiling in the context of evaluation, and show that textual datasets differ in ways that challenge assumptions about the applicability of techniques. We set out some criteria for useful profiling measures. We argue that distribution patterns of frequent words are useful in profiling genre, and report on a series of experiments with χ² based measures on the TIPSTER collection, and on textual intranet data. Findings show substantial differences in the distribution of very frequent terms across datasets.

Viewing alternatives

Item Actions

Export

You can export this page using these formats