In this thesis, we study the thermodynamic phase transitions and critical phenomena in a certain class of classical frustrated spin systems with continuous spin symmetry. The main goal is to gain a unified knowledge on this class of systems. The most important common feature of this class of systems is that the system comprises two equivalent sublattices and the symmetry of the characteristic frustrated exchange coupling precludes the inter-sublattice effective coupling of the bilinear structure. Consequently, the effect of the higher order biquadratic exchange coupling, which is usually dominated by that of the bilinear coupling in conventional systems, becomes prominent. As a result, the phase transitions in this class of systems significantly differ from those in the conventional unfrustrated systems.

In Chapter 2, by constructing an appropriate effective GLW Hamiltonian for N-component vector fields and performing the mean-field and RG analysis, we studied several generic aspects of this class of frustrated systems. The two-sublattice structure with the ferro-quadrupolar biquadratic coupling in principle allows the Ising-like intermediate phase. In this phase, only an Ising-like composite order parameter exhibits long-range ordering while the individual spins remain disordered. Because of the composite structure of the Ising-like order parameter, the Z2 symmetry breaking transition temperature cannot be lower than that of the O(N) magnetic transition where spins exhibit long-range ordering. We then examined a mean-field theory that can account for such an intermediate phase. We found that the theory predicts a multicritical point in the vicinity of the bicritical point where the Z2 and O(N) transition merge. While the Z2 symmetry breaking transition is always of second order in this approximation, the O(N) transition changes itself from of second order to first order before it merges with the Z2 transition.

In the subsequent RG treatment, we examined the stability of the conventional O(N) fixed point corresponding to decoupled sublattices using scaling analysis. The precise numerical estimation of the critical exponents available in literature allowed us to obtain an accurate conclusion on this fixed point: the XY and Heisenberg decoupled fixed points are unstable against the perturbation of the inter-sublattice quadrupolar coupling. We then reviewed the RG treatment by Aharony and we investigated the more global structure of the RG flow diagram by the one-loop approximation. No stable fixed point was found for the XY and Heisenberg spin systems. Therefore, one possibility is the fluctuation-induced first-order phase transition. However, this statement needs certain care because the RG flows near the O(N) decoupled fixed point obtained by the one-loop approximation contradict the results obtained by the nonperturbative scaling arguments. The latter is more reliable because higher order contribution is taken into account.

In Chapter 3, we discussed the Heisenberg model in a magnetic field on the body-centered tetragonal (BCT) lattice. The system is XY-like because of the magnetic field. A quasi-2D spin-dimer compound BaCuSi2O6 is an experimental realization of the BCT antiferromagnet when a magnetic field with strength comparable to the singlet-triplet excitation gap is applied. The main motivation of the present work is to clarify the relation between the experimental observation of the XY-like transition at T > 0 and the potential consequence of the previous prediction of the "bond ordering" in this compound. Using the spin-wave approximation and Monte Carlo simulation, we investigated the finite-T transition in detail. We confirmed the collinear two-sublattice state stabilized by the order-by-disorder mechanism. In that phase, O(2) spin symmetry and Z2 lattice symmetry are broken. In particular, we found that the effect of the inter-sublattice bilinear coupling is completely suppressed (prohibited by symmetry) while the biquadratic coupling is allowed and present. Therefore, the system provides a realization of the effective GLW Hamiltonian with N=2. Most importantly, in light of the previous RG treatment by Aharony, our observation indicated that the BCT antiferromagnet such as BaCuSi2O6 may undergo hidden crossover behavior on the verge of the XY-like ordering. This crossover is induced by the inter-sublattice biquadratic coupling. The RPA argument suggested that the two kinds of symmetry breaking (i.e. XY-like and Ising-like) take place simultaneously at a single transition in most cases of the XY spin systems. The results of Monte Carlo simulation partially succeeded in giving numerical supports for the above notions: the collinear ordered state and the single phase transition with the simultaneous Ising-like and XY-like orderings. However, we found that the scaling behavior around the transition temperature is strongly dominated by the decoupled 3D XY fixed point, which was shown to be unstable by the previous RG argument. The crossover theory can give us a reasonable account for such an observation of transient critical behavior: it is most likely due to the system size much smaller than the characteristic length for the crossover behavior. However, the crossover theory cannot provide information on the resulting true asymptotic behavior deviating from the 3D XY-like behavior on the very verge of the phase transition.

The unsettled problem of the crossover from the decoupled 3D XY model was re-investigated in Chapter 4. We explored the true asymptotic behavior resulting from the crossover from the 3D XY decoupled fixed point by introducing another effective lattice model suitable for large-scale simulation and performing systematic finite-size scaling analysis, referred to as the Monte Carlo renormalization group. The effective model that we introduced has the same symmetry structure as the frustrated systems but it is unfrustrated in itself allowing us to employ a variant of the efficient cluster algorithm. Using the Monte Carlo renormalization group analysis, we found that the RG flow near the 3D XY fixed point shows systematic deviations from this fixed point and reaches the region where we confirmed the first-order transition. Because we did not find any sign of a stable fixed point or a separatrix, we concluded that our observations provide a strong numerical evidence of the first-order transition in the frustrated XY spin systems. The absence of stable fixed point is qualitatively consistent with the one-loop treatment of the corresponding effective GLW Hamiltonian. Therefore, our numerical results in tern suggest that in spite of the incorrect features contradictory to the results of the non-perturbative scaling arguments, the one-loop approximation nevertheless captures the correct physical picture, namely the fluctuation-induced first-order transition.

The crossover region where the true weak discontinuous nature appears is tiny as long as the frustrated inter-layer coupling is small in comparison with the intra-layer exchange. This is indeed the case of the quasi-2D compound BaCuSi2O6. Therefore, the thermodynamic behavior will be dominated by the 3D XY decoupled fixed point in a broad region near the transition and the true discontinuous nature of the transition could easily be beyond the experimental precision in most cases. This is our conclusion as to the relation between the potential effect of the bond ordering and the experimental observation of the 3D XY-like criticality.

In Chapter 5, we investigated the Heisenberg spin systems. Among the same class of frustrated systems, the discovery of the high-Tc superconductivity in ferropnictides has given rise to a recurring interest on the stacked J1-J2 Heisenberg model because it has provided a simple understanding of the structural and spin-density-wave (SDW) order in these compounds. In particular, as argued by Xu et al., the J1-J2 Heisenberg model can naturally explain the experimental observation that the structural transition temperature in ferropnictides is commonly either the same as the Neel temperature, as in the 3D 122-type undoped compounds, or higher than the Neel temperature as in the quasi-2D 1111-type compounds. By means of Monte Carlo simulation on an effective model called the Ising-O(3) model we found that this class of Heisenberg spin systems in d=3 has the Ising ordered phase where the Heisenberg spins remain disordered in a moderate quasi-2D region. The region where the O(3)-symmetric Ising ordered phase exists was quantitatively clarified. The important difference from the XY spin systems, where we concluded such an intermediate phase is almost unlikely in d=3, is related to the fragileness of spin ordering of the Heisenberg spin systems. Compared to the Ising-like ordering, the ordering of the Heisenberg spins is more sensitive to the magnitude of the inter-layer coupling. Our finite-size scaling analysis suggests that the transitions are of second order and in the Ising and Heisenberg universality classes. So far, the other subtle features discussed using the mean-field theory have not been reproduced but need further investigation. In the 3D region with sufficiently large inter-layer coupling the first-order transition directly bridges the paramagnetic and lowest-T ordered phases, in agreement with the absence of stable fixed points in the one-loop RG flow diagram obtained by Aharony. Finally, in d=2 the system undergoes a finite-temperature transition in the Ising universality class where the O(3) spins remain disordered, as expected from its symmetry, range of interaction and the dimensionality.

By comparing the results of the Ising-O(3) model with experiments on ferropnictides, we found that our results provide a qualitative explanation on the experimental observations, namely the observed sequence and the orders of transitions, in terms of the magnitude of the inter-layer coupling. Firstly, the separate second-order or weakly first-order structural and SDW transitions in the 1111 systems, for which density functional calculation suggest the strong two dimensionality, may be understood in terms of the Ising-O(3) model in the quasi-2D region where the transitions are separate and of second order. Also, the first-order transition in the more 3D 122 parent systems, where the lattice distortion and the SDW ordering simultaneously occur, may be understood in terms of the Ising-O(3) model in the region where the inter-layer coupling is sufficiently large. Although the local-moment model is most likely inadequate for microscopic description of the metallic ferropnictides, the phase diagram that qualitatively reproduces the experiments leads us to expect that the J1-J2 Heisenberg model and the related systems serve as a good starting point for describing the universal properties of ferropnictides.

古典スピン系は、統計力学の最も基本的な模型の一つとして長年研究され、相転移や臨界現象など多くの重要な概念の形成に重要な役割を果たした。しかし、全ての相互作用エネルギーを同時に最小にすることができない、すなわちフラストレーションがある模型については、多くの基本的な問題が未解明のままである。また、近年の物質開発の進展によって、興味深い古典スピン系の問題が提起されることも数々ある。本論文は、このような背景のもと、フラストレーションのある連続スピン系における相転移と臨界現象を数値的および解析的手法によって理論的に研究したものであり、6章からなる。

第1章では、研究の動機として、臨界現象の理論的研究と、フラストレーションのある磁性体の実験的研究を簡単に紹介している。第2章では、連続対称性を持つ2次元反強磁性スピン系が積層した3次元系で、面間の結合にフラストレーションがある場合について議論している。この系の臨界現象を記述する有効的な場の理論は、以前に別の問題についてAharonyによって導かれた場の理論と同一であり、その際に与えられたくりこみ群の解析が本論文の対象にも適用できることを指摘している。

第3章では、第2章で議論した系の具体例として、面間の結合にフラストレーションがある積層正方格子反強磁性古典XY模型を論じている。これは、最近注目されているスピンダイマー系BaCuSi2O6の磁場誘起相転移を記述する模型として導入されている。面内ではフラストレーションがないため、この模型の基底状態は各面内のネール秩序状態で与えられる。このとき、面間相互作用は完全にキャンセルするため、ネール秩序の向きは面ごとに自由である。しかし、有限温度では、ゆらぎの効果により、各面は2つおきに同じネール秩序を持ち、隣り合う面のネール秩序は平行または反平行のどちらかに揃う傾向がある。これが全体に波及すれば、イジング的な対称性の自発的破れを伴う相になる。このイジング秩序と、ネール秩序は異なる温度で発現する可能性があり、興味がもたれる。しかし、本章では、モンテカルロシミュレーションによって、誤差の範囲内でこの2つの転移温度が一致することを示した。また、臨界指数は3次元XY模型のものと一致し、偶数番目の面と奇数番目の相が独立にXY転移すると言う描像を支持する結果となった。第2章のくりこみ群の議論によれば、このような臨界現象はくりこみ群の不安定な固定点に対応し実際の系では観測されないはずであり、本章の結論と一見矛盾する。これについて著者は、シミュレーションが可能な系のサイズでは、有限サイズ効果のために不安定な固定点の効果が観測されると言う解釈を与えている。

第4章では、第3章で問題となった不安定な固定点に対応する臨界現象について、より詳しく調べている。原理的には、十分大きなサイズの系のシミュレーションを行えば、不安定な固定点から離れて真の臨界現象を観測できるはずである。しかし、元のフラストレートしたスピン系の直接シミュレーションでは、計算可能な系のサイズが限られており検証が困難である。そこで、著者は、元の模型に代えて、臨界現象を記述する有効的な場の理論のモンテカルロシミュレーションを格子上で行った。モンテカルロくりこみ群の解析により、系が実際に不安定な固定点から離れる様子が確認され、熱力学的極限では非常に弱い1次転移を示すことが示唆された。

第5章では、鉄ニクタイド物質に関連して議論されているフラストレートした古典ハイゼンベルグ模型の有効模型として、イジングスピンとハイゼンベルグスピンが結合した擬2次元古典スピン系を導入している。2次元XY模型と異なり、2次元ハイゼンベルグ模型は、温度が正である限り相関関数が指数関数的に減衰すると考えられている。これを反映して、面間相互作用が弱い極限では、イジング転移が有限の温度で起きる一方、ハイゼンベルグスピンの秩序化の転移温度はゼロに漸近する。すなわち、面間相互作用が十分弱ければ、イジングスピンとハイゼンベルグスピンの秩序化は異なる温度で起こり、イジングスピンのみ秩序化した中間相が存在することが予想される。著者は、大規模なモンテカルロシミュレーションにより、面間相互作用が面内相互作用の約2%以下では実際に中間相が存在することを示す結果を得た。

第6章では全体のまとめと考察を行なっている。

以上のように、本論文では、フラストレーションのある古典スピン系で有効的にイジング自由度が出現する場合について、解析的な理論とモンテカルロシミュレーションを組み合わせることによって詳細に調べている。特に、元の模型の直接シミュレーションでは調べることが困難な現象について、有効模型を導出してその大規模なシミュレーションを行い、また理論的な知見と組み合わせることによって深い理解を得ることに成功している。本論文で展開された手法は、本論文で取り扱った模型のみならず、広い範囲のフラストレーション系の研究にも有効であることが期待できる。

なお、本論文は、指導教員である川島直輝教授他との共同研究に基づいているが、本人の寄与は主体的で十分であると認められる。

よって、論文審査委員会は全員一致で博士(理学)の学位授与が適当であると認めた。