The out-the-window view provides the main information pilots need to control the aircraft during the final phases of landing. Pilots learn how to use this information through extensive practicing, and little is known about the -mostly subconscious- perception and decision making processes playing a role. In my research I try to reveal which visual cues (i.e., characteristics of the visual scene) pilots use for their navigation and control decisions, and how they use them.

Outline

This thesis consists of 3 main parts. The first part consists of critical reviews of visual cues and pilot models found in literature. After introducing the general principles of visual depth and motion perception, I discuss the applicability of previously suggested cues to the final approach and landing of airplanes. Various control modeling approaches are reviewed, and especially applications to human operator modeling and airplane pilot modeling in specific are considered.

The second part deals with the generation of fuzzy-neural network models to identify human pilot control behavior. Using data from real and simulated landings by professional airline pilots, small networks were trained to map the relation between visual cue inputs and the pilot's longitudinal aircraft control as output. The gamma-operator, know for its applicability to human decision making, was used as a neuron transfer function to create a transparent model which can be interpreted in terms of a fuzzy logic controller.

In the third part I introduce and analyze the hypothesis that the apparent motion of the runway sidelines is an important cue for timing the initiation of the flare maneuver. Several pilots took part in additional simulator experiments to test this hypothesis under different approach conditions.

Background

The focus of this research is on the so called `flare' maneuver for mid-sized jet airliners, although the preceding `glide phase' (Fig. 1) is also included to some extent. The flare maneuver is considered the most difficult standard maneuver in airplane operations, while its timely initiation and proper execution are indispensable for a safe and soft landing. A few seconds before touchdown, the pilot pulls the control column to pitch up and change the 3 degree glide approach path to a shallower path. Apart from reducing the sinkrate -and thus the impact with the ground- the higher pitch of the aircraft also ensures it will touch down on the main gear first, before the nose gear touches down.

Although training literature and pilots generally state that the flare should be initiated at a certain altitude, it is also known that the timing should be changed according to the sinkrate (which depends on e.g. wind and aircraft weight). Flare-related decisions are made in close proximity to the ground, and information has to be integrated and processed so quickly that the pilot has no time to read all instruments. Therefore he obtains the necessary information mostly from visual cues. Since visual perception and situational awareness are subconscious processes, even highly skilled pilots cannot explain their technique in words. This means that students have to learn by experience from extensive (simulator) practicing, with only marginal feedback.

The study of visual cues is not only important for pilot training and developing training aids, but also for a variety of other applications, such as the development of augmented or synthetic vision systems, the improvement of simulator fidelity, and the awareness of possible visual illusions.

Materials and Methods

In the literature review I consider several viewpoints and present findings from medical, psychophysical, perception, and engineering reports, as well as guidelines found in pilot training publications. General introductions to pictorial cues, optical flow theory, and time-to-contact are provided. Visual cues suggested by other researchers are inventoried, and their applicability to control in the final phases of aircraft landing is investigated through an analysis of cue availability, perceptual thresholds, and visual illusions. The question of whether binocular cues could be used by pilots is extensively discussed in a special section. The review of literature on modeling techniques provides the reader with a little background regarding existing research on pilot modeling. Although most of this research was done back in the late 1950s-70s and applies linear feedback control modeling strategies, I also present some other methodologies that recently caught more attention.

Whereas most existing research involves simplified laboratory experiments and uses aircraft states as model inputs, I obtained data from full flight training simulators with professional pilots and from a few real flights, and use visual cues as model inputs. As mentioned before, the pilot obtains most information for the final approach and landing control from the visual scene, since there is no time to read all instruments and integrate this information. The visual cues are therefore the real inputs to the pilot, and consequently they should be the input parameters if the model is to be used for analysis of the human operator himself. Simulated landings in a Dornier Do-228-200 turboprop and in a Boeing 767 type airplane form the main data basis for this research. Landings were performed with clear visibility and under various wind and loading conditions.

For the general analysis of visual cue use, I apply a neural network modeling approach. However, instead of the commonly used sigmoid transfer functions, I use the gamma-operator proposed by Zimmermann and Zysno [``Decisions and evaluations by hierarchical aggregation of information''. Fuzzy sets and systems, 1983; 10(3):243-260]. The gamma-operator allows for partial compensation and can `interpolate' between the common AND and OR logic connectives. The neural network framework allows the parameters of the gamma-operator to be trained from data through error back-propagation learning. The gamma-operator was chosen as it matches human decision making characteristics well, and its application makes the resulting network (model) more transparent. The model structure and parameters can easily be put into a linguistic form, and are therefore well suited for applications in pilot training and feedback generation.

In the third part of this research (i.e., testing the hypothesis that the apparent motion of the runway sidelines is an important flare cue) I analyze the proposed cue mathematically and compare it to other cues proposed in literature. Statistical analyses such as ANOVA and coefficients of variation analysis are applied to approach data obtained in dedicated simulator experiments. Additionally, eye-tracking experiments were done to find out where the pilot is fixating during the final approach and landing. Various cockpit instruments as well as various positions in the out-the-window scene were selected for classification of the fixation data.

Results

The literature study resulted in three lists of visual cues which are considered important in the glide phase, the flare phase, and for the timing of the flare initiation. Especially for the glide phase, there seems to be a strong agreement that the `H-distance' (the angle in the vertical plane between the aimpoint and the horizon; dH in Fig. 2c) is important for keeping the glide path. Also the principle that the aimpoint should coincide with the center of expansion of the optical flow is often noted, as well as the importance of the height of the horizon with respect to an aircraft-fixed reference for inner loop pitch attitude control. For flare control (touch down control), many possible cues were identified from literature, including the runway sidelines, the focus of expansion, optical flow (through a wide field of view), the height of the horizon and the time-to-contact. For flare initiation, mainly the angle between the runway sidelines (theta, a cue for altitude; Fig. 2) and the time-to-contact have been considered in literature so far.

The study of literature on binocular cues showed that pilots of mid-sized airliners most likely cannot use stereopsis for depth perception in the final approach, although it may still benefit some last-second adjustments in touch down control. Stereopsis may still be useful for landing small aircraft, close range operations such as mid-air refueling and taxing, and for helicopter operations. Other interesting findings from this investigation include the importance of a wide field of view and the notion that the effective use of monocular cues may be trained.

The gamma-network pilot models showed that the H-distance is the main cue during the glide phase, which is supported by literature. A more interesting result was that, the apparent motion of the runway sidelines (theta-dot; Fig. 2b) was identified as the main visual cue for the timing of the flare initiation. This cue was not found in literature, while literature did mention the importance of a combination of altitude and sinkrate, two aircraft states directly influencing this visual cue. Application of the gamma-network modeling technique to aircraft states -rather than to visual cues- also identified altitude and sinkrate as the main factors influencing the pilot's flare initiation timing.

The surprising result of this general modeling required further investigation of the apparent motion of the runway sidelines as a visual cue to flare initiation timing. A mathematical analysis of the landing scene geometry revealed that theta-dot is a function of the runway width, altitude, and sinkrate. Altitude and sinkrate are two factors of known importance to the flare, since an earlier flare (i.e., at higher altitude) is needed in an approach with excessive sinkrate to obtain a gentle touchdown. The fact that the runway width also plays a role in this cue could cause visual illusions when landing on runways of non-standard width. Such illusions are actually reported by pilots, and it is therefore still possible that pilots indeed rely on the apparent motion of the runway sidelines.

In tailored experiments using full flight training simulators, professional airline pilots landed a Boeing 767 under various wind and gross-weight conditions. These conditions resulted in various nominal approach sinkrates and subsequently influenced the proposed visual cue, while the typical visual cues of a 3 degree glideslope were maintained.

The strong winds appeared challenging for the pilots, and some data obtained under extreme headwind conditions may not be representative due to the exceptionally low pitch angles required in these approaches. Data from the captain-rank pilot supported the hypothesis that the theta-dot cue is used to time the flare initiation, although the use of other or additional cues is suggested in the extreme headwind cases. Although the vertical time to contact tau(z) (altitude divided by sinkrate), also matched the data well, its visual counterpart tau(w), which is based on the apparent runway width, showed poor correspondence. Moreover, eye-mark experiments showed that pilots are not paying attention to this cue. The one novice pilot (co-pilot-rank) studied in this research appeared to use the automated voice callouts of radio altitude for timing the flare maneuver. In comparison with a captain pilot's control inputs during the flare phase, the co-pilot's control was clearly of higher amplitude and frequency, which suggests that incorporating the sinkrate in the flare timing decision takes away the need for relatively aggressive touchdown control.

Using the eye-tracking device it was found that the pilots' fixations are mostly on the instrument panel (glideslope, altitude and speed indicators) up to about 1km before touchdown (below 200ft). When looking outside, the pilot mostly fixates on the runway threshold or the aimpoint markers. In the following phase to touchdown, over 70% of time is spent looking at the out-the-window scene, and mostly at the end of the runway or the horizon. Extreme winds and the absence of motion simulation were found to increase instrument use, while the variety of instruments checked also changed in some cases. When asked to minimize instrument use, pilots keep checking the airspeed indicator like before, while use of the artificial horizon (flight director/attitude) display reduced greatly. Use of the PAPI and the sinkrate and glideslope indicators becomes prominent in visual flight.

Although the results for the extreme headwind cases remain inconclusive, the majority of the data supports the hypothesis that pilots use the apparent movement of the runway sidelines is used as a visual cue to time the initiation of the flare maneuver. It is likely that this cue is perceived foveally when the pilot looks at the far runway end and the horizon, and probably also through optical flow in the periphery of the pilot's visual field.

Figure 1: The final approach to landing consists of the 'Glide' phase, where the pilot has to track a 3 degree glideslope, the 'Flare' phase, where the pilot pitches up to arrest sinkrate and prepares for landing softly on the main gear, and possibly a 'Float' phase where the pilot needs to perform some final 'touchdown control'. Ideally, a smooth flare ends with the touchdown, and no float phase is present. (Pitch angles are exaggerated.)

Figure 2: Change of the visual scene for (a) forward motion, (b) downward motion and (c) glideslope tracking. The sequences show a constant speed motion at constant time intervals (within a sequence). θ is the angle between the runway sidelines. dH is the H-distance. The rightmost pictures combine all sequence geometries.

修士(工学)Jorg Onno Entzinger 提出の論文はAnalysis of Visual Cues for Human Pilot Control in the Final Approach and Landing「パイロットの着陸操縦におけるビジュアルキューに関する研究」と題し、6章からなっている。

旅客機の操縦は高度に自動化されているが、現在でも離着陸の大多数はパイロットのマニュアル操縦であり、特に着陸には難度の高い操縦技術が要求されている。過大な沈下速度による着陸は、乗客へ不快感を与えるのみならず、機材への構造的ダメージや、乗員への身体的影響を及ぼすため、エアラインでは操縦訓練や、操縦マニュアルの整備を進めている。ただし、接地間際の引き起こし操作時には、パイロットは計器類を確認する余裕がなく、コックピットからの視覚情報(ビジュアルキュー)をもとに操縦を行っているため、操縦技能の習得は難しく、効果的な訓練法の開発が望まれている。本論文は、こうした状況を背景として、パイロットがビジュアルキューをどのように操縦に反映しているかを分析するために、ビジュアルキューを入力、パイロットの操作を出力とするパイロットモデルをニューロ・ファジィネットワークを用いて構築するとともに、フレアタイミングの決定法に関する研究をおこなっている。

第1章は序論で、研究の背景を整理するとともに、着陸操縦法を説明し、最後に本研究の応用の可能性を整理している。

第2章では、ビジュアルキューに関する文献調査結果を整理している。過去の研究では両眼を使用するステレオ視と、単眼による奥行き視の医学、心理学、航空工学の研究がなされ、旅客機のビジュアルキューには両眼ステレオ視の利用には限界があり、単眼のオプティカルフローが有効に利用されていることが示されている。

第3章では、制御工学によるパイロット操縦のモデル化に関して過去の研究が概観されている。線形、準線形理論に基づく数学モデル化はパイロットにとって理解が困難であり非線形な操作の表現に限界があるため、ファジィ理論やニューラルネットワークの研究が進められていることが示されている。本研究では、パイロットの普遍的な操縦パターンを分析するために、比較的単純なニューロ・ファジィネットワークを採用することが説明されている。

第4章では、ニューロ・ファジィネットワークを利用したパイロットの着陸操縦分析法が提案され、その分析結果が示されている。ニューロ・ファジィネットワークはファジィな論理結合子を用いて入出力関係をネットワークとしてモデル化するもので、入力にはタッチダウンマーカーの位置と間隔、水平線の位置、滑走路両サイドラインの角度、接地予想時間が、出力にはエレベーター操舵量が用いられている。ビジネスジェット、小型旅客機、中型旅客機に関するフライトシミュレータおよび実機飛行試験により、入出力データを取得し、それらを再現するニューロ・ファジィネットワークモデルを3度の経路角を維持するアプローチフェーズ、接地間際に沈下率を連続的に低減させる引き起こし(フレア)フェーズ、およびその中間のフレアタイミングフェーズに分けて構築している。その結果、アプローチフェーズではタッチダウンマーカーの位置が、フレアフェーズでは滑走路両サイドラインの角度と水平線位置の変化率が、フレアタイミングフェーズでは滑走路両サイドラインの角度変化率が主要なビジュアルキューであることが導かれている。特に、フレアタイミングが、滑走路両サイドライン角度変化率に依存するという結果は、新たな知見である。

第5章では、フレアタイミングに関係するビジュアルキューに関する仮説の検証を試みている。最初に滑走路両サイドライン角度変化率の幾何学的な解析から、滑走路両サイドラインの角度変化率が、高度と沈下率の関数となることを導き、フレアタイミングを決定するビジュアルキューに相応しいことを考察している。その後、経路角が同じ場合、沈下率が機体重量や風によって変化することに着目し、機体重量と風を変えた場合の着陸操縦分析を5名のパイロットに関してフライトシミュレータを用いて実施している。その結果、1名の副機長はいずれのケースでも一定高度でフレアを開始しているが、他の4名の機長は高度ではなく、滑走路両サイドラインの角度変化率が一定になるようにフレアを開始していることを明らかにした。この結果は、熟練したパイロットは、滑走路両サイドラインの角度変化率というビシュアルキューを活用してフレアタイミングを決めていることを裏付けている。また、補足的にアイマークカメラによる視線の計測結果も行っている。これによれば、パイロットはフレアフェーズでは計器を確認する時間が減少し、モーション無しに比べ、モーション有りのシミュレータを操縦するほどビジュアルキューの依存度が増すことが示され、ビジュアルキューの重要性が確認されている。

第6章では、本研究の成果をまとめると同時に、さらなる研究課題について述べている。

以上、要するに、本論文は、ビジュアルキューに基づくパイロット着陸操縦のニューロ・ファジィネットワークによる分析手法を提案し、異なる機体の操縦分析から滑走路両サイドラインの角度変化率がフレアタイミングを決めるという仮説を導き、複数のパイロットに対する異なる飛行環境による着陸操縦からその仮説の裏付けを行った。これらの成果は、航空工学上貢献するところが大きい。

よって本論文は博士(工学)の学位請求論文として合格であると認められる。