(本文)

The Web has become a new communication medium with historical, cultural, and social significance. Many different organizations, governments, groups, and individuals are increasingly and continually publishing and distributing their information on the Web. Collectively, this makes the Web highly dynamic: web pages are being created, updated, and deleted rapidly. According to recent statistics, seven million new web pages are being created daily; and the average life span of a web page is approximately 44-75 days. It is clear that if we do not preserve the Web, we will permanently lose some important information.

In recent years, many organizations have started Web archiving projects with the aim to preserve the Web information. A notable large-scale Web archiving project is the Internet Archive, which has begun archiving the global Web from 1996. As of July 2007, the Internet Archive has collected web pages corresponding to about 95 billion URLs (1.75 PB of raw data). Other Web archive projects are led by the national libraries of many different countries such as Australia, Sweden, United Kingdom, and Japan. The national Web archiving project is aiming at long-term preservation and access of Web information related to a specific country.

The borderlessness of the Web poses difficulties for comprehensive archiving of the Web of a country (national web archiving). Comprehensive Web archiving usually relies on a web crawler to automatically download a large number of web pages quickly. In the construction of a national web archive, where the primary goal is to comprehensively collect as much as possible web pages related to a specific country, two conventional web crawling methods are usually used i.e. (1) a domain name based restriction, and (2) geographical location based restriction crawling .

In domain name based restriction crawling, the crawler will be configured to limit crawl scope to only web pages from some selected domain names. For example, one of possible crawling methods for building a Thai web archive is to restrict the crawl within Country Code Top-Level Domain (ccTLD) for Thailand, i.e. ".th" domain name. In geographical location based restriction crawling, the crawler will be configured to download only web pages from servers whose physical locations are within a specific country. In this case, building of a Thai web archive can be done by limit the crawl to only web pages belonging to servers whose physical locations are inside Thailand.

This dissertation studies the challenges and issues faced in collecting language specific web pages and building of Thai web archive. Due to a wide varieties of languages and highly variable degree of cohesiveness of same-language web pages in the Web, conventional crawling methods for building web archives (i.e. domain name based restriction and geographical location based restriction crawling methods) are not suitable for the construction of a comprehensive, high-precision language specific web archive. A more realistic and suitable solution might be a language specific crawler.

In this work, we propose and design a method for crawling language specific web pages without any constraints on domain names and locations of web servers. A language specific crawler is implemented and uses in the building of our Thai web archive. Then, we conduct comprehensive link-based and content-based analyses on the Thai web archive derived from our Thai web crawls.

In link-based analysis, we analyze the characteristics and various statistical properties of the Thai Web graphs associated with Thai web snapshots in the archive. The study can be divided into three levels of abstractions: page-level, host-level, and community level link analysis. We also discuss and suggest some interesting applications of the statistics of Thai Web graph e.g. comparison of spam diffusion rates between different Web sub-regions (such as Thailand vs. Japan), and the discovery of new crawl seeds by studying the linguistic purity of web communities. In content-based analysis of the Thai web archive, we will focus on the detection of semantically meaningful socio-topical web keywords and their evolution patterns.

Following, we will give a detailed outline of the dissertation.

The dissertation starts by studying how we can efficiently collect web pages written in a specific language from the borderless Web space. Towards this goal, we first identify hyperlink patterns that frequently lead to Thai web pages by analyzing characteristics and graphical structure of a large Thai web snapshot. Based on the guidelines derived from linguistic analysis of the hyperlink patterns of the Thai Web mentioned earlier, we devise page-level and server-level language specific web crawling methods for Thai web crawling. Because web crawling consumes resources of remote servers, it is socially, economically, and ethically inappropriate to test a crawler on the real Web. To overcome this difficulty, we have designed and implemented a trace-driven web crawling simulator which utilize large real crawl log to simulate the real Web on a single local machine.

The evaluation of the proposed language specific crawling methods is done on the crawling simulator. The simulation-based evaluation results show higher performance of our proposed methods than conventional crawling strategies. The crawling method with the highest precision and coverage is the server-level strategy. We then implement our language specific web crawling method on a language specific crawler which will be used in the building of Thai web archive.

Unlike previous works on web archiving whose primary concerns are long-term access and preservation of the Web information, this work focuses on deriving values from the archives. The remaining parts of the dissertation deal with analysis and mining of the Thai web snapshots, and discuss how we can utilize the obtained statistical properties in future crawls, web archive managements, and spam detection. We analyze and mine the Thai web snapshots stored in the Thai web archive using both link-based and content-based techniques.

In link-based analysis and mining, we study several statistical properties of the Thai Web graph such as degree distribution, connectivity, and large-scale structure. The study can be divided into three levels of abstractions: page-level, host-level, and community level link analysis. For each level of abstraction, we try, as much as possible, to compare the derived characteristics of the Thai Web graph with other sub-regions of the Web. We also apply a web community extraction algorithm to the Thai web snapshots. We study these web communities in many aspects such as comparison with a real-world Web directory, linguistic purity of web community, and the evolution of some socially significant Thai web communities.

The statistical properties derived from link-based analysis of Thai Web graph can be used as a feedback for improving crawling strategy, managing of the web archives, and developing novel link-based algorithmic tools. Regarding the utilization of statistical results obtained from our link-based analysis of the Thai Web, we discuss some interesting applications of our statistics e.g. (1) degree distribution analysis for spam detection, and (2) linguistic purity of web community for crawl seeds expansion.

In content-based analysis and mining, we study the evolution of Thai web keywords and explore its relationship with real-world social events. As the Web is now being inundated by hyperlinked information issued by many organizations around the globe. Current events and trends that are happening in the real world may be detected from the Web. We first study the statistical characteristics of socio-topical web keywords sampled from Thai web archive. The socio-topical web keyword is a keyword relating to some topics of interest in a real-world society. We propose a method for extracting these socio-topical keywords from a series of web snapshots. Our proposed method relies on the correlation between link-based and contend-based characteristics of meaningful topical web keywords. By studying the evolution patterns of the extracted socio-topical keywords, it is possible to detect an event and/or trend which were/are happening in the real world.

Finally, the dissertation ends with a summary of main results and a discussion of the future work and remaining open problems

本論文は「 Research on Language Specific Crawling and Building of Thai Web Archive(言語特定クローリングとタイウェブアーカイブの構築に関する研究)」と題し、英文8章から構成されている。ウェブ上から特定の言語で表記されたページを効率よく取り込む(クローリング)方式を提案し、ウェブの実データを用いた実験を行い、提案する方式の有効性を論じると共に、タイ語で記述されたウェブアーカイブを構築、特定言語ウェブの特性について論じている。

第1章は、「Introduction (序章)」であり、本論文の背景および目的について概観し、本論文の構成を述べている。

第2章は、「Related Work(関連研究)」と題し、ウェブアーカイビング、目的に特化したクローリング、ウェブ構造の解析手法に関し、関連研究をまとめている。

第3章は、「Language Specific Crawling (言語特定クローリング)」と題し、特定言語で記述されたページのみを効率良く収集するための手法を提案している。本提案手法では、特定言語で記述されたページ探索戦略として、該当言語ページ内のリンク先の言語種別およびリンクホップ数、特定言語が使われているドメインのサーバ情報を利用することで、単純な幅優先探索、あるいは、特定言語ページへのリンクのみを利用する場合に比べ短時間に効率よく当該言語ページの収集が可能であることを示している。提案方式を詳述すると共に、当該手法の正しさ、および効率について、ウェブから収集されたデータセットを用いたシミュレーション実験を行い、その有効性を示している。

第4章は、「Thai Web Archive(タイウェブアーカイブ)」と題し、第3章で提案した言語特定クローリングを実装し、実際に2007年に5回のタイウェブスナップショットの収集を行い、その結果得られたタイウェブアーカイブの基礎データを示している。

第5章は、「Thai Web Community(タイウェブコミュニティ)」と題し、タイウェブスナップショットからコミュニティを抽出し、コミュニティ内のタイ言語で記述されたページ比率を解析すると共に、コミュニティ内のリンク情報からクローリングされていないページの特性を解析し、タイに関連する内容でありながらタイ言語以外で記述されたページの収集が可能であることを示している。その結果、提案している言語特定クローリングに当該言語のコミュニティ解析情報を加え、当該言語を用いる国、地域に関連したページの収集について議論、検討を行っている。

第6章は、「Characteristics of Thai Web(タイウェブの特徴)」と題し、収集したタイウェブスナップショットを用い、タイウェブグラフのリンク構造解析を行い、その形状を明らかにしている。また、日本語など他言語のウェブグラフのリンク構造と比較を行い、言語間の構造の差異を論じている。さらに、ウェブアーカイブのコンテンツ解析として、タイウェブアーカイブから社会的に話題となり得る語を含むページを抜き出し、その特性について議論、検討を行っている。

第7章「Conclusions and Future Work(結論)」では、本論文の成果と今後の課題について総括している。

以上これを要するに、本論文は、言語特定クローリング手法に関し、ページ内リンク情報、特定言語ドメイン内のサーバ情報に着目することにより、当該言語ページの収集の効率化を図るものであり、シミュレーション解析により有効性を明らかにすると共に、実ウェブからタイウェブアーカイブを構築し、ウェブコミュニティ解析を用いることで、未収集のページのみならず当該言語以外で記述されている関連ページをも発見可能であることを明らかにしており、電子情報学上貢献するところが少なくない。

よって本論文は博士(情報理工学)の学位請求論文として合格と認められる。