Web数据反馈的搭配抽取方法

林建方¹, 牛成², 李生¹, 郑德权¹

1.哈尔滨工业大学语言语音教育部-微软重点实验室;2.微软亚洲研究院



摘要:

为了提高搭配(Collocation)抽取的精度,提出一种新的互联网数据的搭配抽取方法.传统的搭配抽取统计方法都是基于语料库的,常受到语料库规模的影响和制约,而在互联网数据中蕴含着丰富的知识和信息,基于Web的词汇相关性度量方法,充分利用搭配在谷歌中的页面数模拟其对应语料库的词频数,并分别选取共现频率、互信息、卡方检验3种经典统计关联度量方法.实验结果表明召回率、精确率均好于对应的基于语料库的方法,这说明互联网中大量数据应用于自然语言处理各种任务的可行性.

关键词:  搭配  共现频率  互信息  卡方检验  语料库  Web

DOI：10.11918/j.issn.0367-6234.2010.02.023

分类号:TP391.1

基金项目:国家自然科学基金重点资助项目(60736044);国家科技发展计划探索类资助项目(2006AA01Z150)



Automatic collocation extraction using web feedback data

LIN Jian-fang¹, NIU Cheng², LI Sheng¹, ZHENG De-quan¹

1.MOE-MS Key Laboratory of Natural Language Processing and Speech,Harbin Institute of Technology,Harbin 150001,China;2.Microsoft Research Asia,Beijing 100080,China)

Abstract:

To improve the precison of collocation extraction,this paper proposes a new method based on Internet data.For the constraint by the corpus scale for traditional collocation extraction approach based on linguistic corpus,we acquire collocations from Web,which contains plenty of information and knowledge.Three classical association measures of co-occurrence frequency,mutual information and χ2-test are used to automatically extract the collocation.Based on the experimental results,the benchmarks show that the performance of this new Web-based approach is superior to that of traditional approach in both precision and recall.Thus the data from Internet may be applied in many NLP applications.

Key words:  collocation  co-occurrence frequency  mutual information  χ2-test  corpora  web