Facial expressions are playing an important role in communication between people. Related research from behavioral science provides strong evidence that facial expressions are tightly related to emotions. There are six basic emotions that are communicated by facial expressions, i.e. joy, sadness, surprise, anger, fear and disgust. It is a challenge to enable machines to automatically detect and classify facial expressions in order to adapt to the current emotional state of users. This will lead to improved quality of human-machine interfaces in the future. New technologies such as e.g. the Internet are recently gaining more and more popularity as a channel for human communication. One can say we are on the way to a ubiquitous network society. Therefore social rules have to be applied to networked user environments. In this context it is very important to consider the emotional component in networked communities. Current solutions for networked communities do mainly not reflect this issue. There are existing solutions for communicating emotions in the case that a networked community has been already established. However, as in real-life emotions are very important for even establishing a communication or community respectively. Currently there exists no solution that takes the emotional component for the process of community building in networked environments into account. The research presented here in this thesis was motivated by this situation. As a result of this work a user grouping mechanism has been developed, that provides users in a commonly shared context with the means to find other, so far unknown users based on a so called Emotion History. The new method enables people to find and connect to others that are of similar type or character as oneself. The benefit of this new method for user grouping is that a pre-selection of users can be made prior establishing a communication channel. This pre-selection is based on similar emotions derived from facial expressions and previously experienced in a shared context of a networked community. The Emotion History stores knowledge about previously experienced emotions of joy and surprise. Both emotions are derived directly from the intensities of facial expressions. A similarity measure is applied to the Emotion Histories of different users in order to provide a suggestion, whether to people will match or not based on the stored knowledge of emotions.

In the last decades much research has been done in the area of automatic facial expression recognition. The common approach is to detect facial features, track them over time and classify changes of the features in order to identify a certain expression. A necessary constraint is, to separate the feature changes caused by rigid head movement from those caused by the actual change in the human face. More specifically it is important to compensate in-plane and out-of plane head rotations in order to provide the classification task with reliable and stable features. Since this research focused on classification of facial expressions, this issue became an important matter. As a solution and third contribution of the work presented here, the Adaptive Depth Map algorithm is proposed as a solution for the above problem. The main goal of the algorithm is to provide a compensation of feature position displacement caused by the surface structure of the face, i.e. different depth of feature points. This natural property of a human face causes the problem of moving features during out-of plane head rotations. This displacement is obviously not caused by changes in facial expressions and has to be therefore compensated. The compensation term is derived from observed displacement of features in the camera's image plane by referencing to a stored frontal face view. Each feature will be assigned with a compensation term in a dynamic process. This term will be used further to compensate out-of-plane head rotations for each feature. The benefit of this approach is that no camera calibration is necessary and furthermore it is adaptive to new users, i.e. user independent. Moreover the Adaptive Depth Map algorithm is easy to implement and has it advantages over other approaches such as e.g. using adaptive customized 3D head models in cases were the head rotation parameters are known.

The Facial Action Coding System developed by Ekman and Friesen allows a systematic and detailed description of facial mimics via so called action units. One feature of these action units is that they encode different levels of intensity. In this research a set of neural network classifiers has been developed, that allows in contrast to other related works in the area an explicit training of these intensity levels. Each classifier outputs an intensity value accordingly to the trained shape of facial features. This allows a dynamic analysis of facial expressions which is a crucial requirement for detecting blends of different emotions displayed in a human's face.

During the research presented in this thesis the BROAFERENCE frame work has been developed and implemented as a flexible platform for media distribution in networked communities. It served as a major tool for creating a networked community and provides a set of realized methods for facial expression analysis, shared context creation and user grouping. Several experiments of this research have been conducted on top of the BROAFERENCE framework.

　本論文は、「BROAFERENCE−A Study on Automatic Support for Building Emotion Oriented Networked Communities(ブローファレンス−感性共有情報を用いたネットワークコミュニティ構築自動支援に関する研究)」と題し、通信放送融合時代の新しいサービスを提供するものであり、TV番組を視聴するユーザ同士を感性情報に基づいてグループ化し、新しいコミュニティーを生成することにより、放送と通信双方の魅力を相乗的に高める技術を提案している。

　ユビキタスネットワーク社会の発達により、いつでもどこでも仮想空間上で他の人と会える。それゆえ、リアルな生活での社会ルールは仮想空間にも拡張されるべきであり、例えば、将来の通信システムでは、感情をきちんと扱えるようにする必要もあろう。本論文はこのような背景から、通信放送融合システムにおいて自動的に抽出された顔表情と視線情報を用いて新サービスを生成することを目的とし、具体的には、BROAFERENCEと呼ぶ新しいサービスを提案している。

　BROAFERENCEは、放送と通信の融合を目指した新しいサービスであり、双方向放送が当たり前になる時代を前提に、双方向放送型接続に加え、視聴者同士をも双方向で会議接続するシステムである。この際、感性視聴質測定システムを研究開発し、ユーザ(TV視聴者)の顔表情や視線情報から感情情報を自動測定して放送局にフィードバックすするとともに、この時の感情情報からユーザコミュニティーを自動生成し、ユーザ間の会話を楽しみながらTV視聴することを可能とする。同一TV番組を見て同じような感情や視線を有する視聴者同士ならばその感性が近く、会話も盛り上がりやすいからである。

　本論文では、以上のような概要のもと、下記の章によって構成されている。

　まず第1章は、感情とは何かについて定義し、感情と顔表情の関係について述べ、続く第2章は、既存研究として、顔表情の判定ならびに分類、ネットワークコミュニティー生成における感情の扱い、視線追尾などに関する既存研究について整理している。

　第3章は、「Adaptive Depth Map Algorithm」と題し、これまでの顔表情検出技術の問題点について論じ、従来では実現できなかった精度で顔表情検出・分類を行うための基礎となる適応的Depth Map手法を提案している。

　第4章は、「Action Unit Detection with Artificial Neural Networks」と題し、FACS(Facial Action Coding System)に基づく顔の特徴点検出手法とそれを用いた顔表情の分類手法について、ニューラルネットワークを用いた手法について提案している。

　第5章は、「User Grouping」と題し、自動測定された顔表情の種類・程度を個々のユーザ毎に蓄積し、その結果を用いてマッチングさせるための手法を提案している。

　第6章は、「Broaference Framework」と題し、第3,4,5章で提案した手法を融合して新しい通信放送融合型マルチメディアサービスを実現するためのフレームワークを提案する。また、「感情情報」という個人情報保護のためのセキュリティー技術について論じている。

　第7章は、「Other Application」と題し、先述したBROAFERENCE以外への応用として、4つのアプリケーション提案している。

　第8章は、「Conclusions and Future Works」と題し、結論と今後の課題について言及している。

　以上のように、本論文では、TV視聴者の顔情報から感情情報を自動抽出し、その結果を放送局にフィードバックするとともに、感情情報に基づいてユーザコミュニティーを自動生成し、ユーザ間の会話を楽しみながらTV視聴することを可能とするものであり、その成果は実世界への応用を含め画像処理工学分野に寄与するところ大である。

　よって本論文は博士(工学)の学位請求論文として合格と認められる。