摘要:在大型IT企业中，尤其像Google或者百度，代码搜索已是软件开发过程中不可或缺且频繁的活动，其通过借鉴或复用已有代码，加速开发过程的速度.多年以来，已有大量的研究人员关注代码搜索，且设计出很多优秀的工具.但是已有的研究和工具主要是在小规模或者编程语言单一的代码数据集上，没有从企业实际搜索需求出发，且对用户的查询输入也有所限制，尚缺少一套针对企业级海量代码的检索与管理技术方案.提出了一套企业级海量数据代码搜索引擎的方案和系统实现，面向开发过程中用户最直接的需求，通过离线分析与在线分析，完成对海量代码库的索引构建与检索.其中，离线分析负责代码相关数据的获取与分析、构建索引集群.在线过程负责变换用户的query、对搜索的结果进行高级排序、生成摘要.本系统部署在百度代码库上，为数十TB级的Git代码库构建了索引，平均一次检索时间在1s之内.在百度推出应用以来，访问量逐步增加，现每周平均用户有数千人，每周查询平均有数万次，广受百度工程师好评.



Abstract:In large IT companies, especially like Google or Baidu, code search is an indispensable and frequent activity in the software development process, which speeds up the development process by learning or reusing existing code. Over the years, a large number of researchers have focused on code search and designed many excellent tools. However, the existing research and tools are mainly on a small-scale or single programming language code data set, not from the actual requirement of industries, and the user's query input is also limited; there is still a lack of a set of industrial-scale massive code retrieval and management technology solutions. This study proposes a code search engine solution and system implementation based on industrial-scale massive data, oriented to the most direct needs of users in the development process, through offline analysis and online analysis, complete the index construction and retrieval of massive code base. Among them, offline analysis is responsible for the acquisition and analysis of code-related data and building an index cluster. The online process is responsible for transforming the user's query, sorting the results of the search, and generating a summary. The system is deployed on the Baidu code base, and the index is built for dozens of TB-level Git code bases. The average retrieval time is within 1s. Since the launch of Baidu's application, the number of visits has gradually increased. There are thousands of users per week and tens of thousands of times searching. The system is widely praised by Baidu engineers.



PDF全文下载地址:

http://jos.org.cn/jos/article/pdf/5718