Gesture perception or action recognition is receiving growing attentions due to its ap-plications in smart surveillance, sign language interpretation, advanced user interface and intelligent robot control. As compared to static image recognition, action recogni-tion usually requires handling overwhelmingly large amount of data because a whole set of video sequences must be analyzed. Moreover, if action recognition is subject to cluttered background, results are often degraded significantly. Therefore, it sometimes requires taking additional measures, i.e., tracking windows or background estimation on the frame-basis. In some cases, it is desirable to build the recognition system di-rectly in the VLSI hardware such as ASICs (application specific integrated circuits) or FPGAs in order to achieve real-time performance. Therefore several constraints need to be further imposed on the algorithms. One important requirement is that the back-ground elimination should be incorporated to the system so that video sequences can be taken as direct input. Another constraint is that computation in the system should be simple enough to be implemented on VLSI circuits either by analog or digital technology.

Among various algorithms, recognition based on local features is receiving great attention due to its robustness to space and time variation. In order to apply local fea-tures for recognition, interest points containing essential information of the movement are usually detected by spatio-temporal detectors, which are inspired by object track-ing and object recognition. Once interest points are detected, motion descriptors are extracted from the video and used for building models representing certain motions. Beside the algorithms that apply machine-learning methods for motion analysis, bio-inspired hierarchical models based on both local features and associate memory prin-ciples show promising advantages for recognition tasks. One of the latest models for action recognition extended the similar structures for established object recognition system, and good results are reported. Nevertheless, the system applied only spatial patches in the processing for lower layers, therefore may not fully take advantages of the temporal relationship in the video sequences.

Inspired by this work as well as our previous efforts on VLSI based recognition sys-tem applying associate memory principles, a VLSI-hardware-friendly action recogni-tion algorithm using spatio-temporal motion-field patches has been introduced in this thesis. The system employs a hierarchical two-layer structure so that the robust recog-nition can be achieved gradually. At the lower level, primary features called motion field maps that represent local features such as speed and direction are calculated from video sequences, further blurred by max filters. At the higher level, a collection of so-called template/prototype patches are used to recognize query actions by com-paring local features in the query videos with those prototypes. In addition, in order to design a system for real-time performance, we intentionally simplify all the calcula-tions into summation operations or Boolean operations so that the algorithm can be directly implemented on ultra high-speed VLSI chips without much effort. Our pro-posed system is at first developed for the application of gesture perception and prom-ising results were reported compared to our previous researches based on global fea-tures.

As an improvement, we have further proposed an enhanced processing to estimate motion field maps based on so-called essential directional edge displacement map. As the results, clean motion field maps can be calculated at the lower level. In addition, we simply the computations for feature vectors generation, by introducing more hardware friendly updating scheme. The proposed method not only reduce the com-putational cost significantly compared to our previous system but also fully take ad-vantages of parallel processing for hardware implementation.

For most researches applying local features, descriptors play an important role in achieving high accuracy recognition. However, for most of researches, pixels within the descriptors are directly concatenated and Principle component analysis (PCA) is usually carried out for dimension reduction, which are not only unsuitable for hard-ware implementation but also inefficient to some extent. Based on our previous sys-tems developed for face recognition. We have extended the face descriptor to the spatio-temporal form by coding temporal information along with spatial information. In addition, another effective descriptor coded for local maximum was also proposed. We show that the choices of the descriptors have significant influence to the recogni-tion result.

Finally, once feature vectors for a given video is extracted, classifiers will be applied so that query videos can be labeled from the information of learning samples (already labeled samples). Among various classification models, sparse representation classification (SPC) becomes popular due to its powerfulness in face recognition. Re-cently, an extension of SPC called Fisher discriminant dictionary learning (FDDL) has been also proposed in which a structured dictionary is learned instead of raw vectors from learning samples. Because the success of the algorithm for face recognition, we employed it in our action recognition system and compared the recognition results with kNN. We show that FDDL is an effective classification method for our local feature based recognition system.

本論文は,"A Hierarchical Action Recognition System Based on Spatio-Temporal Local Motion Feature Descriptors (和訳:ローカルな動きの時空間特徴表現を用いた階層的行動認識システム)"と題し,ビデオシーケンス中の人物の行動認識に関し,動きフィールドの時間変化をベースとして,ローカルな動きの時空間特徴表現を導入するとともに,これを用いて階層的な行動認識システムを構築する研究の成果を纏めたもので,全文6章よりなり英文で書かれている.

第1章は,序論であり,本研究の背景について議論するとともに,本論文の構成について述べている.

第2章は,"A System for Gesture Perception Using Compact Spatio-Temporal Motion Filed Patches"と題し,本研究で構築した行動認識システム全体の構成について述べている.人間の脳における階層的な動き認識のプロセスにヒントを得て,動画像の各フレームより方向性エッジを抽出,これに基づいて動き場を求めるとともに,この動き場の時間変化から行動を認識する構成となっている.静止している背景の情報を消去して動きのある部分のみに着目するため,時間的に変化するエッジフラッグだけを残し,その総量が所定の値となるまで積分したDifferential Edge Displacement (DED) マップを作成するが,これによって動作のスピードに依存しない認識アルゴリズムを実現している.そして,時間的に連続するDEDマップのブロックマッチングから動きフィールドを検出し,行動を動きフィールドの時間シーケンスとしてとらえる.さらに,こうして得られた,空間軸(x,y)及び時間軸(t)上に広がる動きフィールドの3次元データを,dx,dy,dtの微小3次元領域のpatchの集合体としてとらえるが,本研究では,patch内の動きフィールドの各方向成分を,空間軸上,時間軸上に射影することで,コンパクトなベクトルに変換する方法を提案している.さまざまな種類の動作からpatchをランダムに多数サンプリングし,K-measn法で典型的なパターンを500個選び出し,これを行動パターン表現の基底として用いる.そして一つの行動シーケンスは,その中に含まれるすべてのパッチと500の典型パターンとの類似度を求め,各パターンの最大類似度を要素とするベクトルとして表現される.このベクトル表現では,動作シーケンス内の時空間情報がすべて消去されているため,大変ロバストな認識ができる.腕を回して単純なパターンを描くという比較的簡単な動作に対し,nearest neighbor分類器で90%以上の正答率を得ており,隠れマルコフモデルを用いたもっと複雑なシステムよりも良い結果となっている.微小3次元領域patchの時空間軸上への射影によるコンパクトなベクトル表現は,本研究の重要な成果である.

第3章は,"Complex Action Recognition Employing Essential Directional Edge Displacement (E-DED) Maps and Updating Feature Calculations"と題し,前章で確立したシステムの性能向上と,もっと複雑な問題への適用の結果について述べている.認識の基となる動きフィールド検出の精度向上のため,その元となるDEDマップに対し,よりノイズの少ないEssential DED(E-DED)マップを導入するとともに,得られた動きフィールドにblurフィルターによるボカシを施すことにより,特徴的な動きのみハイライトされるようにした.その結果,前章より複雑な動作を含むWeisemann human databaseや,さらに困難なKTH human databaseに対し高い正答率を得た.これは実用的に重要な成果である.

第4章は,"Spatio-temporal Motion Field Descriptors for the Hierarchical Action Recognition System"と題し,本アルゴリズムをVLSIハードウェア上へ効率よく実装するための新たな微小3次元領域patchの表現方法を提案している.パッチを時空間軸上で等間隔に分割したセルに分け,セル内の各動き方向成分の最大値を用いるMPM表現法,及び合計値を用いるAPM表現法の二つを提案,第1章での表現よりコンパクトで認識性能も高いことが示されている.Weisemann databaseでは,提供されている背景情報を用いることなく先行研究よりも良い結果が得られている.

第5章は,"Fisher Discrimination Dictionary Learning (FDDL) via Sparse Representation for Action Recognition"と題し,モデルの学習過程に関し,異なる方法についての検討結果を述べている.顔認識で良い結果の得られているSparse Representation法にFDDL法を組み合わせたシステムでは,Weisemann databaseに対し100%の認識率が得られた.しかしKTH databaseでは,第3章の結果には及ばなかった.

第6章は結論である.

以上要するに本論文は,ビデオシーケンスから抽出した動きフィールドの時間変化をベースに,この中のローカルな時空間微小領域の動きをコンパクトな特徴ベクトルで表現し,これを用いて階層的な行動認識システムを構築するとともにその有用性を実験的に示した研究の成果を纏めたもので,電子工学の発展に寄与するところが少なくない.

よって本論文は博士(工学)の学位請求論文として合格と認められる.