On effect of surface tension in the Rayleigh–Taylor problem of stratified viscoelastic fluids

Abstract

In this article, we investigate the effect of surface tension in the Rayleigh–Taylor (RT) problem of stratified incompressible viscoelastic fluids. We prove that there exists an unstable solution to the linearized stratified RT problem with a largest growth rate Λ under the instability condition (i.e., the surface tension coefficient ϑ is less than a threshold $\vartheta _{c}$). Moreover, for this instability condition, the largest growth rate $\varLambda _{\vartheta }$ decreases from a positive constant to 0, when ϑ increases from 0 to $\vartheta _{c}$, which mathematically verifies that the internal surface tension can constrain the growth of the RT instability during the linear stage.

1 Introduction

It is well known that the equilibrium state of the heavier fluid on top of the lighter one under the gravity is unstable to sustain a small disturbance. In this process, the unstable disturbance will grow and lead to a release of potential energy. Since Rayleigh [40] and then Taylor [43] first studied this phenomenon, we call it the Rayleigh–Taylor (RT) instability. In the last decades, it has been also widely investigated how the RT instability evolves under the effects of other physical factors, such as elasticity [4, 11, 27, 29, 42, 46], rotation [3, 6, 45], internal surface tension [9, 16, 24, 48], magnetic fields [5, 17, 19,20,21,22,23, 25, 26], and so on. We also refer to the other related mathematical problems [1, 12, 13, 30, 32,33,34, 36, 38, 41]. In this article, we consider the effect of surface tension on the linear RT instability for stratified viscoelastic fluids defined on a horizontally periodic domain in the presence of a uniform gravitational field. Before stating our main results, we shall introduce the relevant mathematical progress in the stratified RT problem in detail.

To begin with, let us recall the RT problem of stratified viscoelastic incompressible fluids in an infinity layer domain [27]:

$$ \textstyle\begin{cases} \rho _{\pm }\partial _{t}v_{\pm }+\rho _{\pm }v_{\pm }\cdot \nabla v_{ \pm }+\operatorname{div}\mathcal{S}_{\pm }(p^{g}_{\pm },v_{\pm },U_{\pm })=0 &\text{in }\varOmega _{\pm }(t), \\ \partial _{t}U_{\pm }+v_{\pm }\cdot \nabla U_{\pm }=\nabla v_{\pm }U _{\pm } &\text{in } \varOmega _{\pm }(t), \\ \operatorname{div}{v_{\pm }}=0 &\text{in } \varOmega _{\pm }(t), \\ d_{t}+v_{1}\partial _{1}d+v_{2}\partial _{2}d=v_{3}&\text{on }\varSigma (t), \\ [\!\![v_{\pm }]\!\!]=0,\quad \quad [\!\![\mathcal{S}_{\pm }(p ^{g}_{\pm },v_{\pm },U_{\pm })\nu -gd\rho _{\pm }\nu ]\!\!]= \vartheta \mathcal{C}\nu &\text{on }\varSigma (t), \\ v_{\pm }=0 &\text{on }\varSigma _{\pm }, \\ (v_{\pm },U_{\pm })| _{t=0}=(v^{0}_{\pm },U^{0}_{\pm })&\text{in } \varOmega _{\pm }(0), \\ d| _{t=0}=d^{0}&\text{on }\varSigma (0). \end{cases} $$

(1.1)

Next we further explain some notations in the above (stratified) VRT problem (1.1).

The notations $f_{+}$ and $f_{-}$ in (1.1) denote the values of the quantity f in the upper and lower fluids, respectively. The superscript T denotes the transposition, and the notation $f^{0}$ denotes the initial data of f. In this paper, we consider that the domain Ω occupied by the two fluids is horizontal periodic, and thus we denote

$$ \varOmega :=\bigl\{ (x_{h},x_{3})\in \mathbb{R}^{3}\mid x_{h}:=(x _{1},x_{2}) \in \mathbb{T}, -h_{-}< x_{3}< h_{+}\bigr\} \quad \text{with } h _{-},h_{+}>0, $$

(1.2)

where $\mathbb{T}:=\mathbb{T}_{1}\times \mathbb{T}_{2}$, $\mathbb{T} _{i}=2\pi L_{i}(\mathbb{R}/\mathbb{Z})$, and $2\pi L_{i}$ ($i=1,2$) are the periodicity lengths. For each given $t>0$, $d:=d(x_{h},t): \mathbb{T}\mapsto (-h_{-},h_{+})$ is a height function of a point at the interface of stratified viscoelastic fluids. $\varSigma (t)$ is a set of interface, and it is defined as follows:

$$ \varSigma (t):=\bigl\{ (x_{h},x_{3})\mid x_{h}\in \mathbb{T}, x _{3}:=d(x_{h},t)\bigr\} . $$

(1.3)

Moreover, we also have the following expressions:

$$ \begin{aligned} & \varSigma _{+}:=\mathbb{T}\times \{h_{+}\}, \qquad \varSigma _{-}:=\mathbb{T}\times \{-h_{-}\}, \\ &\varOmega _{+}(t):=\bigl\{ (x _{h},x_{3})\mid x_{h}\in \mathbb{T}, d(x_{h},t)< x_{3}< h_{+} \bigr\} , \\ & \varOmega _{-}(t):=\bigl\{ (x_{h},x_{3})\mid x_{h}\in \mathbb{T}, -h_{-}< x_{3}< d(x _{h},t)\bigr\} , \\ &\varOmega (t):=\varOmega _{+}(t)\cup \varOmega _{-}(t). \end{aligned} $$

For given $t>0$, $v_{\pm }(x,t):\varOmega _{\pm }(t)\mapsto \mathbb{R} ^{3}$, $p_{\pm }(x,t):\varOmega _{\pm }(t)\mapsto \mathbb{R}$, and $U_{\pm }(x,t):\varOmega _{\pm }(t)\mapsto \mathbb{R}^{9}$ are the velocities, the pressures, and the deformation tensor (a $3\times 3$ matrix-valued function) of fluids. Moreover, the stress tensors enjoy the following expression:

$$ \mathcal{S}_{\pm }\bigl(p^{g}_{\pm },v_{\pm },U_{\pm } \bigr):=p^{g}_{\pm } I- \mu _{\pm }\mathbb{D}v_{\pm }- \kappa _{\pm }\rho _{\pm }\bigl(U_{\pm }U_{ \pm }^{T}-I \bigr), $$

(1.4)

where $p^{g}_{\pm }:=p_{\pm } +g\rho _{\pm }x_{3}$, $\mathbb{D}v_{ \pm }:=\nabla v_{\pm }+\nabla v^{\mathrm{T}}_{\pm }$ and I denotes the $3\times 3$ identity matrix. $\rho _{\pm }$ are the density constants, $U_{\pm }$ are the deformation tensor (a $3\times 3$ matrix-valued function), and the constants $\mu _{\pm }$ and $\kappa _{\pm }$ denote the shear viscosity coefficients and the elasticity coefficients of the two fluids, resp. g and ϑ represent the gravitational constant and the surface tension coefficient, resp. For a function f defined on $\varOmega (t)$, we define $[\!\![f_{\pm }]\!\!]:=f_{+}| _{ \varSigma (t)}-f_{-}| _{\varSigma (t)}$, where $f_{\pm }| _{\varSigma (t)}$ are the traces of the quantities $f_{\pm }$ on $\varSigma (t)$. ν is the unit outer normal vector at boundary $\varSigma (t)$ of $\varOmega _{-}(t)$, and $\mathcal{C}$ is the twice of mean curvature of the internal surface $\varSigma (t)$, i.e.,

$$ \mathcal{C}:=\frac{\Delta _{h}d+(\partial _{1}d)^{2}\partial ^{2}_{2}d+( \partial _{2}d)^{2}\partial ^{2}_{1}d-2\partial _{1}d\partial _{2}d \partial _{1}\partial _{2}d}{(1+(\partial _{1}d)^{2}+(\partial _{2}d)^{2})^{3/2}}. $$

Finally, we briefly explain the physical meaning of each identity in (1.1). The equations (1.1)₁–(1.1)₂ describe the motion of the upper heavier and lower lighter fluids driven by the gravitational field along the negative $x_{3}$-direction, which occupy the two time-dependent disjoint open subsets $\varOmega _{+}(t)$ and $\varOmega _{-}(t)$ at time t, respectively. We call (1.1)₁ the momentum equation and (1.1)₂ the deformation equation. Since the fluids are incompressible, we naturally pose the divergence-free condition (1.1)₃. The two fluids interact with each other by the motion equation of a free interface (1.1)₄ and the interfacial jump conditions in (1.1)₅. The first jump condition in (1.1)₅ represents that the velocity is continuous across the interface. The second jump condition in (1.1)₅ represents that the jump in the normal stress is proportional to the mean curvature of the surface multiplied by the normal to the surface [31, 50]. The non-slip boundary condition of the velocities on both upper and lower fixed flat boundaries are described by (1.1)₆, and (1.1)₇–(1.1)₈ represent the initial status of the two fluids.

Problem (1.1) enjoys an equilibrium state (or rest) solution: $(v,d,U,p^{g})=(0,\bar{d},I,\bar{p}^{g})$, where $\bar{d}\in (-h_{-},h _{+})$. We should point out that $\bar{p}^{g}$ can be uniquely computed out by hydrostatics, which depends on the variable $x_{3}$ and $\rho _{\pm }$, and is continuous with respect to $x_{3}\in (-h_{-},h _{+})$. Without loss of generality, we assume that $\bar{d}=0$ in this article. If d̄ is not zero, we can adjust the $x_{3}$ co-ordinate to make $\bar{d}=0$. Thus d can be regarded as the displacement away from the plane

$$ \varSigma :=\mathbb{T}\times \{0\}. $$

In order to simplify the representation of problem (1.1), we introduce the indicator function

$$ \chi _{\varOmega _{\pm }(t)}:= \textstyle\begin{cases} 1,&x\in \varOmega _{\pm }(t),\\ 0,&x\in \varOmega _{\pm }^{c}(t), \end{cases} $$

and denote

$$ \begin{aligned} &\rho =\rho _{+}\chi _{\varOmega _{+}(t)}+\rho _{-}\chi _{\varOmega _{-}(t)}, \quad\quad \mu =\mu _{+}\chi _{\varOmega _{+}(t)}+\mu _{-}\chi _{\varOmega _{-}(t)}, \quad\quad \kappa = \kappa _{+}\chi _{\varOmega _{+}(t)}+\kappa _{-} \chi _{\varOmega _{-}(t)}, \\ &v=v_{+}\chi _{\varOmega _{+}(t)}+v_{-} \chi _{\varOmega _{-}(t)}, \quad\quad U=U_{+}\chi _{\varOmega _{+}(t)}+U_{-} \chi _{\varOmega _{-}(t)},\quad\quad p=p_{+}\chi _{\varOmega _{+}(t)}+p_{-} \chi _{\varOmega _{-}(t)}, \\ &v^{0}=v_{+}^{0}\chi _{\varOmega _{+}(0)}+v_{-} ^{0}\chi _{\varOmega _{-}(0)},\quad\quad U^{0}=U^{0}_{+} \chi _{\varOmega _{+}(0)}+U ^{0}_{-}\chi _{\varOmega _{-}(0)}, \\ &\mathcal{S}\bigl(p^{g},v,U\bigr):=p^{g}I- \mu \mathbb{D}v- \kappa \rho \bigl(UU^{T}-I\bigr). \end{aligned} $$

Now, we denote the perturbation quantity around the equilibrium state $(0,0,I,\bar{p}^{g})$ by

$$ v=v-0, \quad\quad d=d-0, \quad\quad V=U-I, \quad \text{and} \quad \sigma =p^{g}-\bar{p}^{g}. $$

Then we have a VRT problem in a perturbation form:

$$ \textstyle\begin{cases} \rho v_{t}+\rho v\cdot \nabla v+\operatorname{div}\mathcal{S}(\sigma ,v,V+I)=0 &\text{in } \varOmega (t), \\ V_{t}+v\cdot \nabla V=\nabla v(V+I) & \text{in } \varOmega (t), \\ \operatorname{div}{v}=0 & \text{in }\varOmega (t), \\ d_{t}+v_{1}\partial _{1}d+v_{2}\partial _{2}d=v_{3} & \text{on }\varSigma (t), \\ [\!\![v ]\!\!]=0,\quad\quad [\!\![\mathcal{S}(\sigma ,v,V+I)-g \rho dI]\!\!]\nu =\vartheta \mathcal{C}\nu &\text{on } \varSigma (t), \\ v=0 & \text{on }\varSigma ^{+}_{-}, \\ (v,V)| _{t=0}=(v^{0},V^{0}) &\text{in }\varOmega (0), \\ d| _{t=0}=d^{0}&\text{on }\varSigma (0), \end{cases} $$

(1.5)

where $\varSigma ^{+}_{-}:=\varSigma _{-}\cup \varSigma _{+}$, and we omitted the subscript ± in the above problem for simplicity. Thus a zero solution is an equilibrium-state solution of the above perturbation VRT problem.

It is well known that the movement of the free interface $\varSigma (t)$ and the subsequent change of the domains $\varOmega _{\pm }(t)$ in Eulerian coordinates will result in severe mathematical difficulties. In order to circumvent such difficulties, we shall adopt the transformation method of Lagrangian coordinates so that the interface and the domains stay fixed in time. In addition, the VRT problem in Lagrangian coordinate has better mathematical structure.

To this end, we define the fixed Lagrangian domains $\varOmega _{+}:= \mathbb{T}\times (0,h_{+})$ and $\varOmega _{-}:=\mathbb{T}\times (-h_{-},0)$, and assume that there exist invertible mappings

$$ \zeta ^{0}_{\pm }:\varOmega _{\pm }\rightarrow \varOmega _{\pm }(0), $$

such that

$$ \begin{aligned} \varSigma (0)=\zeta ^{0}_{\pm }(\varSigma ), \quad\quad \varSigma _{+}= \zeta ^{0}_{+}(\varSigma _{+}), \quad\quad \varSigma _{-}=\zeta ^{0}_{-}(\varSigma _{-}) \end{aligned} $$

(1.6)

and

$$ \begin{aligned} \det \bigl(\nabla \zeta ^{0}_{\pm }\bigr)=1. \end{aligned} $$

(1.7)

The first condition in (1.6) means that the initial interface $\varSigma (0)$ is parameterized by the mapping $\zeta ^{0}_{\pm }$ restricted to Σ, while the latter two conditions in (1.6) mean that $\zeta ^{0}_{\pm }$ map the fixed upper and lower boundaries into themselves. Define the flow maps $\zeta _{\pm }$ as the solutions to

$$ \textstyle\begin{cases} \partial _{t}\zeta _{\pm }(y,t)=v_{\pm }(\zeta _{\pm }(y,t),t)&\text{in }\varOmega _{\pm }, \\ \zeta _{\pm }(y,0)=\zeta ^{0}_{\pm }(y)&\text{in }\varOmega _{\pm }. \end{cases} $$

We denote the Eulerian coordinates by $(x,t)$ with $x=\zeta (y,t)$, whereas the fixed $(y,t)\in \varOmega \times \mathbb{R}^{+}$stand for the Lagrangian coordinates. Here we remark that $\varOmega := \varOmega _{+}\cup \varOmega _{-}$.

In order to switch back and forth from Lagrangian to Eulerian coordinates, we assume that $\zeta _{\pm }(\cdot ,t)$ are invertible and $\varOmega _{\pm }(t)=\zeta _{\pm }(\varOmega _{\pm },t)$, and since $v_{\pm }$ and $\zeta ^{0}_{\pm }$ are all continuous across Σ, we have $\varSigma (t)=\zeta _{\pm }(\varSigma ,t)$. In other words, the Eulerian domains of upper and lower fluids are the image of $\varOmega _{\pm }$ under the mappings $\zeta _{\pm }$, and the free interface is the image of Σ under the mappings $\zeta _{\pm }( \cdot ,t)$. In view of the non-slip boundary condition $v_{\pm }| _{ \varSigma _{\pm }}=0$, we have

$$ y=\zeta _{\pm }(y,t) \quad \text{on }\varSigma _{\pm }. $$

In addition, by the incompressible condition, we have

$$ \begin{aligned} \det (\nabla \zeta _{\pm })=1 \quad \text{in }\varOmega _{\pm } \end{aligned} $$

(1.8)

as well as initial condition (1.7), see [35, Proposition 1.4].

Now, setting $\zeta =\zeta _{+}\chi _{\varOmega _{+}(t)}+\zeta _{-} \chi _{\varOmega _{-}(t)}$, $\eta =\zeta -y$, and the Lagrangian unknowns

$$ (u,\tilde{U},q) (y,t)=(v,U,\sigma ) \bigl(\zeta (y,t),t\bigr)\quad \text{for }(y,t) \in \varOmega \times \mathbb{R}^{+}, $$

and then we can see that in Lagrangian coordinates the evolution equations for u and q read as follows:

$$ \textstyle\begin{cases} \eta _{t}=u &\text{in }\varOmega , \\ \rho u_{t}+\operatorname{div}_{\mathcal{A}}\mathcal{S}_{\mathcal{A}}(q,u, \eta )=0 &\text{in }\varOmega , \\ \operatorname{div}_{\mathcal{A}}u=0&\text{in }\varOmega , \\ [\!\![\eta ]\!\!]=[\!\![u]\!\!]=0, \quad\quad [\!\![\mathcal{S}_{\mathcal{A}}(q,u,\eta )-g\rho \eta _{3}I]\!\!]\vec{n}=\vartheta H\vec{n} &\text{on } \varSigma , \\ (\eta ,u)=0 &\text{on } \varSigma ^{+}_{-}, \\ (\eta ,u)| _{t=0}=(\eta ^{0},u^{0})&\text{in } \varOmega , \end{cases} $$

(1.9)

where we have denoted

$$\begin{aligned} &\mathcal{S}_{\mathcal{A}}(q,u,\eta ):=qI-\mu \mathbb{D}_{\mathcal{A}}u- \kappa \rho \bigl(\mathbb{D}\eta +\nabla \eta \nabla \eta ^{T}\bigr), \quad\quad \mathbb{D}_{\mathcal{A}}u:=\nabla _{\mathcal{A}}u+\nabla _{\mathcal{A}}u ^{\mathrm{T}}, \\ &H:=\bigl( \vert \partial _{1}\zeta \vert ^{2}\partial _{2}^{2}\zeta -2(\partial _{1} \zeta \cdot \partial _{2}\zeta )\partial _{1}\partial _{2}\zeta + \vert \partial _{2}\zeta \vert ^{2}\partial _{1}^{2}\zeta \bigr)\cdot \vec{n}/\bigl( \vert \partial _{1}\zeta \vert ^{2} \vert \partial _{2}\zeta \vert ^{2}- \vert \partial _{1}\zeta \partial _{2}\zeta \vert ^{2}\bigr), \\ &\vec{n}:=\mathcal{A}e_{3}/ \vert \mathcal{A}e_{3} \vert ,\quad\quad \tilde{U}(y,t):= \nabla \zeta (y,t). \end{aligned}$$

In what follows, we call problem (1.9) the transformed stratified VRT problem, and η the displacement function of particle (labeled by y).

Next, we further introduce the notations involving $\mathcal{A}$. The matrix $\mathcal{A}:=(\mathcal{A}_{ij})_{3\times 3}$ is defined via

$$ \mathcal{A}^{\mathrm{T}}=(\nabla \zeta )^{-1}:=(\partial _{j}\zeta _{i})^{-1} _{3\times 3}, $$

where the subscript T denotes the transposition, and $\partial _{j}$ denotes the partial derivative with respect to the jth components of variables y. The differential operator $\nabla _{\mathcal{A}}$ is defined by

$$ \nabla _{\mathcal{A}}w:=(\nabla _{\mathcal{A}}w_{1},\nabla _{\mathcal{A}}w _{2},\nabla _{\mathcal{A}}w_{3})^{\mathrm{T}} \quad \text{and} \quad \nabla _{\mathcal{A}}w_{i}:=( \mathcal{A}_{1k}\partial _{k}w_{i}, \mathcal{A}_{2k}\partial _{k}w_{i}, \mathcal{A}_{3k}\partial _{k}w_{i})^{ \mathrm{T}} $$

for a vector function $w:=(w_{1},w_{2},w_{3})$, and the differential operator $\operatorname{div}_{\mathcal{A}}$ is defined by

$$ \begin{aligned} \operatorname{div}_{\mathcal{A}} \bigl(f^{1},f^{2},f^{3}\bigr):=\bigl( \operatorname{div}_{ \mathcal{A}}f^{1},\operatorname{div}_{\mathcal{A}}f^{2}, \operatorname{div}_{ \mathcal{A}}f^{3}\bigr)^{\mathrm{T}} \quad \text{and} \quad \operatorname{div}_{\mathcal{A}}f^{i}:= \mathcal{A}_{lk}\partial _{k}f_{l} ^{i} \end{aligned} $$

for a vector function $f^{i}:=(f_{1}^{i},f_{2}^{i},f_{3}^{i})^{ \mathrm{T}}$. It should be noted that we have used the Einstein convention of summation over repeated indices. In addition, we define $\Delta _{\mathcal{A}}X:=\operatorname{div}_{\mathcal{A}} \nabla _{ \mathcal{A}}X$.

Finally, we introduce some properties of $\mathcal{A}$. In view of the definition of $\mathcal{A}$ and (1.8), we see that

$$ \begin{aligned} \mathcal{A}=\bigl( \mathcal{A}^{\ast }_{ij}\bigr)_{3\times 3}, \end{aligned} $$

(1.10)

where $\mathcal{A}^{\ast }_{ij}$ is the algebraic complement minor of $(i,j)$th entry $(\partial _{j}\zeta _{i})_{3\times 3}$. Moreover, it is easy to check that $\mathcal{A}e_{3}=\partial _{1}\zeta \times \partial _{2}\zeta$. In addition, we have

$$ \begin{aligned} \partial _{k} \mathcal{A}^{\ast }_{ik}=0 \quad \text{or} \quad \partial _{k}\mathcal{A}_{ik}=0, \end{aligned} $$

(1.11)

and

$$ \begin{aligned} \mathcal{A}_{ij}\partial _{j}\zeta _{l}=\mathcal{A}_{ji}\partial _{l} \zeta _{j}=\delta _{il}, \end{aligned} $$

(1.12)

where $\delta _{il}=1$ for $i=l$ and $\delta _{il}=0$ for $i\neq l$.

We assume that $(u,\eta )$ is very small, then the small terms of second order (i.e., the nonlinear terms) in (1.9) could be neglected, and we thus obtain the following linearized stratified VRT problem:

$$ \textstyle\begin{cases} \eta _{t}=u&\text{in }\varOmega , \\ \rho u_{t}+\nabla q-\mu \Delta u=\kappa \rho \operatorname{div}\mathbb{D} \eta &\text{in }\varOmega , \\ \operatorname{div}{u}=0&\text{in }\varOmega , \\ [\!\![\eta ]\!\!]=[\!\![u]\!\!]=0 &\text{on } \varSigma , \\ [\!\![(q-g\rho \eta _{3})I-\mathbb{D}(\mu u+\kappa \rho \eta ) ]\!\!]e_{3} =\vartheta \Delta _{h}\eta _{3}e_{3} &\text{on } \varSigma , \\ (\eta ,u)=0 &\text{on }\varSigma ^{+}_{-}, \\ (\eta ,u)| _{t=0}=(\eta ^{0},u^{0}) &\text{in }\varOmega , \end{cases} $$

(1.13)

where $\Delta _{h}:=\partial _{1}^{2}+\partial _{2}^{2}$. The linearized problem is convenient to analyze in order to have an insight into the physical and mathematical mechanisms of the stratified VRT problem.

In 1953, Bellman–Pennington [2] first analyzed the inhibition of RT instability by surface tension, where the study was based on a linearized two-dimensional (2D) motion equation of stratified incompressible inviscid fluids defined on the domain $\mathbb{T}_{1}\times (-h_{-},h_{+})$ (i.e., $\mu =0$ in the corresponding 2D case of (1.13)). Moreover, they precisely proved that there exists a threshold $g[\!\![\rho ]\!\!]L _{1}^{2}$ of surface tension coefficient for linear stability and instability of RT problem. In other words, the linear 2D stratified incompressible inviscid fluids are stable when $\vartheta >g[\!\![\rho ]\!\!]L_{1}^{2}$, and vice versa. Similarly, in the case of three dimensions, Guo and Tice proved that $\vartheta _{\mathrm{c}}:= { g[\!\![\rho ]\!\!]\max \{L_{1}^{2},L_{2}^{2}\} }$ is a critical value of surface tension coefficient for stability and instability in the linearized stratified compressible viscous fluids defined on Ω [9].

Next, we further introduce some mathematical progress for the nonlinear case. First, by a Henry instability method [39], Prüess and Simonett first proved that the equilibria solution of RT problem for stratified incompressible viscid fluids defined on the domain $\mathbb{R}^{3}$ is unstable. Later, Wang, Tice, and Kim proved that, under the case of $\vartheta >\vartheta _{\mathbb{T}}$ or $\vartheta \in [0, \vartheta _{\mathbb{T}})$ [47, 48], the equilibria solution of RT problem for stratified incompressible viscous fluids defined on Ω is stable, resp. unstable. In addition, the same results of stability and instability have been further obtained by Jang, Wang, and Tice under the corresponding compressible case [15, 16]. More recently, under the cylindrical domain with finite height [49], Wilke also proved that the value $\vartheta _{\mathrm{c}}$ is a threshold for the stability and instability of stratified viscous fluids (with heavier fluid over lighter fluid). Finally, as documented in [14, 18], the results of nonlinear RT instability in inhomogeneous fluid (without interface), by the classical bootstrap instability method, were obtained for inviscid and viscous cases.

2 Preliminary

2.1 Simplified notations

Before stating our main results for the linearized stratified VRT problem in detail, we shall introduce some simplified notations used throughout this paper.

(1) Basic notations: $I_{T}:=(0,T)$. $\mathbb{R} ^{+}:=(0,\infty )$, $\mathbb{R}^{+}_{0}:=[0,\infty )$, and $\overline{ \varOmega }:=\mathbb{R}^{2}\times [-h_{-},h_{+}]$. $\nabla _{h}:=( \partial _{1},\partial _{2})^{T}$ and $f_{h}:=(f_{1},f_{2})$. $\partial _{h}^{\alpha }$ denote $\partial _{1}^{\alpha _{1}}\partial _{2}^{\alpha _{2}}$ for some multiindex of order $\alpha :=(\alpha _{1}, \alpha _{2})$ and $\partial _{h}^{j}$ denotes $\partial _{h}^{\alpha }$ for any α satisfying $\vert \alpha \vert =\alpha _{1}+\alpha _{2}=j$. The jth difference quotient of size h is $D_{j}^{h}w:=(w(y+he_{j})-w(y))/h$ for $j=1$ and 2, and $D_{h}^{h}w:=(D _{1}^{h}w_{1},D_{2}^{h}w_{2})$, where $\vert h \vert \in (0,1)$, and w is defined on Ω and a locally summable function. ℜf, resp. ℑf, denotes the real, resp. imaginary, part of the complex function f. $a\lesssim b$ means that $a\leqslant cb$ for some constant $c>0$, where the positive constant c may depend on the domain Ω, and known parameters such as g, $\rho _{\pm }$, $\mu _{\pm }$, and ϑ may vary from line to line.

(2) Simplified notations of Sobolev spaces:

$$ \begin{aligned} &L^{p}:=L^{p}(\varOmega )=W^{0,p}(\varOmega ), \qquad W^{i,2}:=W^{i,2}( \varOmega ), \qquad {H}^{i}:=W^{i,2} , \\ &H^{\infty }:=\bigcap_{j=1}^{\infty }H^{j}, \qquad \underline{H}^{i}:=\biggl\{ w\in H^{i}\Bigm| \int _{\varOmega }w\,\mathrm{d}y=0\biggr\} , \\ &H ^{1}_{\sigma }:=\bigl\{ w\in {H}^{1}(\varOmega ) \vert w \vert _{\varSigma _{-}^{+}}=0\text{ in the sense of trace}, \operatorname{div}w=0\bigr\} , \\ &H^{-1}_{\sigma }\text{ is the dual space of }H^{1}_{\sigma }, \qquad H_{\sigma }^{i}:=H_{\sigma }^{1} \cap H^{i}, \\ &H_{\sigma ,\varSigma }^{1}:= \bigl\{ w\in {H}_{\sigma }^{1} \mid w_{3}|_{\varSigma }\in H^{1}(\mathbb{T}) \bigr\} , \\ &H^{1}_{\sigma ,3}:= \bigl\{ w\in H^{1}_{\sigma ,\vartheta } \mid w_{3} \neq 0\text{ on }\varSigma \bigr\} , \quad\quad H_{\sigma , \vartheta }^{1}= \textstyle\begin{cases} H_{\sigma ,\varSigma }^{1} & \text{if }\vartheta \neq 0, \\ H_{\sigma }^{1} &\text{if }\vartheta = 0, \end{cases}\displaystyle \\ & \mathcal{A}: = \bigl\{ w\in {H}_{\sigma ,\vartheta }^{1} \mid \Vert \sqrt{ {\rho }}w \Vert ^{2}_{L^{2}}=1 \bigr\} , \end{aligned} $$

where $1< p\leqslant \infty$, and $i\geqslant 0$ is an integer. Sometimes, we denote $\mathcal{A}$ by $\mathcal{A}_{\vartheta }$ to emphasize the dependence of ϑ. Moreover, to prove the existence of unstable classical solutions of a linearized VRT problem, we shall introduce a function space

$$ H_{\sigma , \vartheta }^{1,k}:= \textstyle\begin{cases} \{w\in H_{\sigma ,\vartheta }^{1} \mid \nabla _{\mathrm{h}}^{j} w \in H^{1}\text{ and }w_{3}| _{\varSigma }\in H^{k+1}(\mathbb{T})\text{ for } j\leqslant k \} & \text{if }\vartheta \neq 0, \\ \{w\in H_{\sigma }^{1} \mid \nabla _{\mathrm{h}}^{j}w \in H^{1} \text{ for } j\leqslant k \}&\text{if }\vartheta = 0, \end{cases} $$

where $k\geqslant 0$ is an integer. Besides, it should be noted that $H_{\sigma ,\vartheta }^{1,0}=H_{\sigma , \vartheta }^{1}$.

(3) Simplified norms: $\Vert \cdot \Vert _{i} := \Vert \cdot \Vert _{W^{i,2}}$, $\vert \cdot \vert _{s} := \Vert \cdot| _{\varSigma } \Vert _{H^{s}(\mathbb{T})}$, where s is a real number, and i is a non-negative integer.

(4) Functionals:

$$ \mathcal{E}(w):= \vartheta \vert \nabla _{\mathrm{h}}w_{3} \vert _{0}^{2}-g[\!\![\rho ]\!\!]\vert w_{3} \vert _{0}^{2}+ \Vert \sqrt{\kappa \rho } \mathbb{D}w \Vert ^{2}_{0}/2 $$

and

$$ \mathcal{F}(w,s):= -\bigl(\mathcal{E}(w) +s \Vert \sqrt{\mu } \mathbb{D}w \Vert ^{2}_{0}/2\bigr). $$

2.2 Preliminary lemmas

In this subsection, we mainly introduce some preliminary lemmas, which will be used later.

Lemma 2.1

Existence theory of a stratified (steady) Stokes problem (see [48, Theorem 3.1]): Let $k\geqslant 0$, $f^{\mathrm{S},1} \in H^{k}$ and $f^{\mathrm{S},2}\in H^{k+1/2}$, then there exists a unique solution $(u,q)\in H^{k+2}\times \underline{H}^{k+1}$ satisfying

$$\begin{aligned} \textstyle\begin{cases} \nabla q-\mu \Delta u =f^{\mathrm{S},1}&\textit{in } \varOmega , \\ [\!\![u ]\!\!]=0,\quad\quad [\!\![(q I-\mathbb{D}u )e_{3} ]\!\!]=f^{\mathrm{S},2} &\textit{on }\varSigma , \\ u=0 &\textit{on }\varSigma _{-}^{+}. \end{cases}\displaystyle \end{aligned}$$

(2.1)

Moreover,

$$ \Vert u \Vert _{\mathrm{S},k} \lesssim \bigl\Vert f^{\mathrm{S},1} \bigr\Vert _{k} + \bigl\vert f^{ \mathrm{S},2} \bigr\vert _{k+1/2} . $$

(2.2)

Lemma 2.2

Difference quotients and weak derivatives: Let D be Ω or $\mathbb{T}$.

• (1) Suppose $1\leqslant p<\infty$ and $w\in W^{1,p}(D)$. Then $\Vert D^{h}_{\mathrm{h}} w \Vert _{L^{p}(D) }\lesssim \Vert \nabla _{\mathrm{h}}w \Vert _{L^{p}(D)}$.

• (2) Assume $1< p<\infty$, $w\in L^{p}(D)$, and there exists a constant c such that $\Vert D^{h}_{\mathrm{h}} w \Vert _{L^{p}(D)}\leqslant c$. Then $\nabla _{\mathrm{h}} w\in L^{p}(D)$ satisfies $\Vert \nabla _{\mathrm{h}}w \Vert _{L^{p}(D)}\leqslant c$ and $D^{-h_{k}}_{ \mathrm{h}} w \rightharpoonup \nabla _{\mathrm{h}}w$ in $L^{p}(D)$ for some subsequence $-h_{k}\to 0$.

Proof

Following the argument of [7, Theorem 3] and using the periodicity of w, we can easily get the desired conclusions. □

Lemma 2.3

Friedrichs’s inequality (see [37, Lemma 1.42]): Let $1\leqslant p<\infty$ and D be a bounded Lipschitz domain. Let a set $\varGamma \subset \partial D$ be measurable with respect to the $(N-1)$-dimensional measure ${\mu }:=\mathrm{meas}_{N-1}$ defined on ∂D, and let $\mathrm{meas}_{N-1}(\varGamma )>0$. Then

$$ \Vert w \Vert _{W^{1,p}(D)}\lesssim \Vert \nabla w \Vert _{L^{p}(D)} $$

for all $w\in W^{1,p}(D)$ satisfying that the trace of w on Γ is equal to 0 a.e. with respect to the $(N-1)$-dimensional measure μ.

Remark 2.1

By Friedrichs’s inequality and the fact

$$ \Vert \nabla w \Vert _{0}^{2}= \Vert \mathbb{D}w \Vert _{0}^{2}/2 \quad \text{for any }w \in H_{\sigma }^{1}, $$

(2.3)

we get Korn’s inequality

$$ \Vert w \Vert ^{2}_{1} \lesssim \Vert \mathbb{D}w \Vert _{0}^{2} \quad \text{for any }w \in H_{\sigma }^{1}. $$

(2.4)

Lemma 2.4

Trace estimates:

$$\begin{aligned} & \vert w \vert _{0} \leqslant \Vert w \Vert _{1} \quad \textit{for any }w\in H^{1}_{\sigma }, \end{aligned}$$

(2.5)

$$\begin{aligned} & \vert w \vert _{0} \leqslant \sqrt{h_{\pm }/2} \Vert \mathbb{D} w \Vert _{L^{2}( \varOmega _{\pm })}/2 \quad \textit{for any }w\in H^{1}_{\sigma }. \end{aligned}$$

(2.6)

Proof

See [28, Lemma 9.7] for (2.5). Since $C_{\sigma }^{\infty }:=C^{\infty }_{0}(\mathbb{R}^{2}\times (-h_{-},h _{+}))\cap H_{\sigma }^{1}$ is dense in $H^{1}_{\sigma }$, it suffices to prove that (2.6) holds for any $w\in C_{\sigma } ^{\infty }$ by (2.5).

First, let ŵ be the horizontal Fourier transformed function of $w\in C_{\sigma }^{\infty }$, and

$$ \varphi (\xi ,y_{3})=\mathrm{i} \hat{w}_{1}(\xi ,y_{3}), \qquad \theta (\xi ,y_{3})=\mathrm{i} \hat{w}_{2}(\xi ,y_{3}), \qquad \psi (\xi ,y_{3})= \hat{w}_{3}(\xi ,y_{3}). $$

Then

$$ \xi _{1}\varphi +\xi _{2}\theta +\psi '=0 $$

(2.7)

and $\psi (\cdot , y_{3})\in H^{2}_{0}(-h_{-},h_{+})$, because of $\operatorname{div} {w}=0$ and $w| _{\varSigma _{-}^{+}}=0$. In addition,

$$ \widehat{\nabla {w}}=(\widehat{\partial _{i} w_{j}})= \begin{pmatrix} \xi _{1}\varphi & \xi _{2}\varphi & -\mathrm{i}\varphi ' \\ \xi _{1}\theta & \xi _{2}\theta &-\mathrm{i}\theta ' \\ \mathrm{i}\xi _{1}\psi & \mathrm{i}\xi _{2}\psi & \psi ' \end{pmatrix}. $$

Next, we can deduce from (2.7) that

$$ \psi (0,y_{3})=0\quad \text{for }\xi =0. $$

(2.8)

By (2.8) and the Fubini and Parseval theorems, one has

$$ \vert w_{3} \vert ^{2}_{0} = \frac{1}{4\pi ^{2} L_{1}L_{2}} \sum_{\xi \in (L^{-1}_{1}\mathbb{Z}\times L^{-1}_{2}\mathbb{Z})\backslash \{0\}} \bigl\vert \psi (\xi ,0) \bigr\vert ^{2} $$

(2.9)

and

$$\begin{aligned} \frac{1}{2} \Vert \mathbb{D} {w} \Vert ^{2}_{L^{2}(\varOmega _{-})}&= \frac{1}{8 \pi ^{2} L_{1}L_{2}} \sum_{\xi \in (L^{-1}_{1}\mathbb{Z}\times L^{-1}_{2}\mathbb{Z})\backslash \{0\}} \sum _{1\leqslant i,j\leqslant 3} \int _{-h_{-}}^{0} \vert \widehat{\partial _{i} {w}_{j}}+\widehat{\partial _{j} {w}_{i}} \vert ^{2} \,\mathrm{d}y_{3} \\ & =\frac{1}{4\pi ^{2} L_{1}L_{2}} \sum_{\xi \in (L^{-1}_{1}\mathbb{Z}\times L^{-1}_{2}\mathbb{Z})\backslash \{0\}}M_{1}^{\xi }( \varphi ,\theta ,\psi ) \\ &\quad{} + \frac{1}{4\pi ^{2} L_{1}L_{2}} \int _{-h_{-}}^{0} \bigl( \bigl\vert \varphi '(0,y_{3}) \bigr\vert ^{2}+ \bigl\vert \theta '(0,y_{3}) \bigr\vert ^{2} \bigr)\, \mathrm{d}y _{3}, \end{aligned}$$

(2.10)

where

$$ \begin{aligned} M_{1}^{\xi }( \varphi ,\theta , \psi ):={} & \int _{-h_{-}}^{0} \bigl( \vert \xi \vert ^{2}\bigl( \vert \varphi \vert ^{2}+ \vert \theta \vert ^{2}+ \vert \psi \vert ^{2}\bigr) \\ &{} +2\Re {\psi }'' \Re {\psi } + 2\Im {\psi }'' \Im {\psi }+ \bigl\vert \varphi ' \bigr\vert ^{2}+ \bigl\vert \theta ' \bigr\vert ^{2}+3 \bigl\vert \psi ' \bigr\vert ^{2} \bigr)\,\mathrm{d}y_{3}. \end{aligned} $$

Using (2.7), we have

$$ \begin{aligned} & \bigl\vert \psi ' \bigr\vert ^{2} = \xi _{1}^{2} \vert \varphi \vert ^{2}+\xi _{2}^{2} \vert \theta \vert ^{2}+ 2 \xi _{1}\xi _{2}(\Re {\varphi } \Re { \theta } + \Im {\varphi } \Im {\theta }) \leqslant \vert \xi \vert ^{2}\bigl( \vert \varphi \vert ^{2}+ \vert \theta \vert ^{2}\bigr), \\ & \bigl\vert \psi '' \bigr\vert ^{2} \leqslant \vert \xi \vert ^{2}\bigl( \bigl\vert \varphi ' \bigr\vert ^{2}+ \bigl\vert \theta ' \bigr\vert ^{2}\bigr) , \end{aligned} $$

which imply that

$$ \begin{aligned} \int _{-h_{-}}^{0} \bigl(4 \bigl\vert {\psi }' \bigl\vert ^{2}+ \bigr\vert \bigr\vert \xi \bigl\vert {\psi }+ {\psi }''/ \vert \xi \vert \bigr\vert ^{2}\bigr) \,\mathrm{d}y_{3}\leqslant M_{1}^{\xi }(\varphi ,\theta , \psi ) \end{aligned} $$

(2.11)

for given $\xi \in (L^{-1}_{1}\mathbb{Z}\times L^{-1}_{2}\mathbb{Z}) \backslash \{0\}$. Employing (2.9)–(2.11) and the relation

$$ \phi ^{2}(0)\leqslant h_{-} \bigl\Vert \phi ' \bigr\Vert _{L^{2}(-h_{-},0)}^{2} \quad \text{for any }\phi \in H^{1}_{0}(-h_{-},h_{+}), $$

we can obtain

$$\begin{aligned} \vert w_{3} \vert ^{2}_{0} ={} & \frac{1}{4\pi ^{2} L_{1}L_{2}} \sum_{\xi \in (L^{-1}_{1}\mathbb{Z}\times L^{-1}_{2}\mathbb{Z})\backslash \{0\}} \bigl\vert \psi (\xi ,0) \bigr\vert ^{2} \\ \leqslant{} & \frac{h_{-}}{16 \pi ^{2} L_{1}L_{2}} \sum_{\xi \in (L^{-1}_{1}\mathbb{Z}\times L^{-1}_{2}\mathbb{Z})\backslash \{0\}} \int _{-h_{-}}^{0} \bigl(4 \bigl\vert {\psi }' \bigr\vert ^{2}+\bigl( \vert \xi \vert {\psi }+ {\psi }''/ \vert \xi \vert \bigr)^{2} \bigr) \,\mathrm{d}y_{3} \\ \leqslant{} & \frac{h_{-}}{16 \pi ^{2} L_{1}L_{2}} \sum_{\xi \in (L^{-1}_{1}\mathbb{Z}\times L^{-1}_{2}\mathbb{Z})\backslash \{0\}} M_{1}^{\xi }( \varphi ,\theta ,\psi ) \leqslant {h_{-}} \Vert \mathbb{D} w \Vert ^{2}_{L^{2}(\varOmega _{-})}/8. \end{aligned}$$

(2.12)

Similarly, we also get

$$ \vert w_{3} \vert ^{2}_{0} \leqslant {h_{+}} \Vert \mathbb{D} w \Vert ^{2}_{L^{2}(\varOmega _{+})} /8, $$

which, together with (2.12), yields the desired conclusion. This completes the proof. □

Remark 2.2

From the derivation of (2.6), we easily see that

$$ \Vert \partial _{3} w_{3} \Vert _{L^{2}(\varOmega _{\pm })}^{2} \leqslant \Vert \mathbb{D} w \Vert _{L^{2}(\varOmega _{\pm })}^{2}/8\quad \text{for any }w\in H^{1} _{\sigma }. $$

(2.13)

Lemma 2.5

Negative trace estimate:

$$ \vert u_{3} \vert _{-1/2} \lesssim \Vert u \Vert _{0}+ \Vert \operatorname{div}u \Vert _{0} \quad \textit{for any }u:=(u_{1},u_{2},u_{3}) \in H_{0}^{1}. $$

(2.14)

Proof

Estimate (2.14) can be derived by integration by parts and an inverse trace theorem [37, Lemma 1.47]. □

Lemma 2.6

Let X be a given Banach space with dual $X^{*}$, and let u and w be two functions belonging to $L^{1}((a,b),X)$. Then the following two conditions are equivalent:

• (1) For each test function $\phi \in C_{0}^{\infty }(a,b)$,

  $$ \int _{a}^{b} u(t)\phi '(t)\, \mathrm{d}t=- \int _{a}^{b} w(t)\phi (t) \,\mathrm{d}t. $$

• (2) For each $\eta \in X^{*}$,

  $$ \frac{\mathrm{d}}{\mathrm{d}t}\langle u,\eta\rangle _{X\times X^{*}}=\langle w,\eta\rangle _{X \times X^{*}}, $$

  in the scalar distribution sense, on $(a,b)$, where $\langle \cdot ,\cdot\rangle _{X \times X^{*}}$ denotes the dual pair between X and $X^{*}$.

Proof

See Lemma 1.1 in Chap. 3 in [44]. □

3 Main results

In this paper, we investigate the effect of surface tension on the linear RT instability by the linearized motion (1.13). First of all, we exploit the modified variational method of PDEs and existence theory of stratified (steady) Stokes problem to prove the existence of solutions with a largest growth rate $\varLambda _{\vartheta }$ for (1.13) under the instability condition $\vartheta \in [0, \vartheta _{c})$. And then, we find a new upper bound for $\varLambda _{\vartheta }$:

$$ \varLambda _{\vartheta }\leqslant m:=\min \{m_{1},m_{2} \}, $$

(3.1)

where

$$\begin{aligned} &m_{1}:=\frac{ (\vartheta _{\mathrm{c}}-\vartheta ) }{4\vartheta _{c}}\biggl(g [\!\![\rho ]\!\!]\min \biggl\{ \frac{h_{+}}{\mu _{+}},\frac{ {h}_{-}}{\mu _{-}} \biggr\} -4\max \biggl\{ \frac{\kappa _{+}\rho _{+}}{ \mu _{+}},\frac{\kappa _{-}\rho _{-}}{\mu _{-}} \biggr\} \biggr), \\ &m_{2}:=\biggl(\frac{(g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}- \vartheta ))^{2} }{4\vartheta _{\mathrm{c}}^{2} \max \{\rho _{+}\mu _{+} ,\rho _{-}\mu _{-}\}}\biggr)^{\frac{1}{3}}. \end{aligned}$$

In addition, we see from (3.1) that

$$ \varLambda _{\vartheta }\to 0\quad \text{as } \vartheta \to \vartheta _{ \mathrm{c}}. $$

(3.2)

In classical Rayleigh–Taylor (RT) experiments [8, 10], it has been shown that the phenomenon of that surface tension during the linear stage can restrain the instability growth, and the growth is exponential in time. Obviously, this phenomenon can be verified mathematically by the convergence behavior (3.2). Next, we shall give the definition of the largest growth rate of RT instability in the linearized stratified VRT problem.

Definition 3.1

We call $\varLambda >0$ the largest growth rate of RT instability in the linearized stratified VRT problem (1.13) if it satisfies the following two conditions:

• (1) For any strong solution $(\eta , u)\in C^{0}([0,T),H^{3} \cap H^{2}_{\sigma })\cap L^{2}(I_{T},H^{3}\cap H^{3}_{\sigma })$ of the linearized stratified VRT problem with q enjoying the regularity $q\in C^{0}([0,T),H^{1})\cap L^{2}(I_{T},H^{2})$, then we have, for any $t\in [ 0,T)$,

  $$\begin{aligned} & \bigl\Vert (\eta ,u) \bigr\Vert _{1}^{2}+ \Vert u_{t} \Vert ^{2}_{0 }+ \int _{0}^{t} \bigl\Vert u(s) \bigr\Vert ^{2}_{1}\,\mathrm{d}s\lesssim e^{2\varLambda t}\bigl( \bigl\Vert \eta ^{0} \bigr\Vert _{3}^{2}+ \bigl\Vert u ^{0} \bigr\Vert _{2}^{2}\bigr) . \end{aligned}$$

  (3.3)

• (2) There exists a strong solution $(\eta , u)$ of the linearized stratified VRT problem in the form

  $$ (\eta , u):=e^{\varLambda t}( \tilde{\eta }, \tilde{u} ), $$

  where $( \tilde{\eta }, \tilde{u})\in H^{2}$.

Now, let us state the first main result on the existence of largest growth rate in the linearized stratified VRT problem.

Theorem 3.1

Let $g>0$, $\rho >0$, $\kappa >0$, and $\mu >0$ be given. Then, for any given

$$ \vartheta \in [0, \vartheta _{\mathrm{c}}), $$

(3.4)

there exists an unstable solution

$$ ( \eta , u,q):=e^{\varLambda t}( w/\varLambda ,w,\beta ) $$

to the linearized stratified VRT problem (1.13), where $(w,\beta )\in H^{\infty }$ solves the boundary value problem:

$$ \textstyle\begin{cases} \varLambda ^{2}\rho w + \varLambda (\nabla \beta -\mu \Delta w)=\kappa \rho \operatorname{div}\mathbb{D}w &\textit{in }\varOmega , \\ \operatorname{div}w=0 & \textit{in }\varOmega , \\ [\!\![w ]\!\!]= 0 &\textit{on }\varSigma , \\ [\!\![(\varLambda \beta - g \rho w_{3})I-\mathbb{D}((\varLambda \mu + \kappa \rho )w) ]\!\!]e_{3}= \vartheta \Delta _{\mathrm{h}}w_{3} e _{3} &\textit{on }\varSigma , \\ w = {0}& \textit{on }\varSigma _{-}^{+} \end{cases} $$

(3.5)

with a largest growth rate $\varLambda >0$ satisfying

$$ \varLambda ^{2}= \sup_{\varpi \in \mathcal{A} }\mathcal{F}(\varpi , \varLambda )=\mathcal{F}(w, \varLambda ) . $$

(3.6)

Moreover,

$$\begin{aligned} & w_{3}\neq 0, \partial _{3} w_{3}\neq 0, \operatorname{div}_{\mathrm{h}}w _{\mathrm{h}}\neq 0\quad \textit{in }\varOmega , \quad\quad \vert w_{3} \vert \neq 0\quad \textit{on } \varSigma . \end{aligned}$$

(3.7)

Next we briefly introduce how to prove Theorem 3.1 by the modified variational method of PDEs and regularity theory of stratified (steady) Stokes problem. The detailed proof will be given in Sect. 4.1.

First, we assume a growing mode ansatz to the linearized problem:

$$ \eta (x,t)=\tilde{\eta }( x)e^{\varLambda t},\quad\quad u(x,t)=w( x)e^{\varLambda t}, \quad\quad q(x,t)=\beta ( x)e^{\varLambda t} $$

for some $\varLambda >0$. Substituting this ansatz into the linearized stratified VRT problem (1.13), we can get a spectrum problem

$$ \textstyle\begin{cases} \varLambda \tilde{\eta }=w &\text{in }\varOmega , \\ \varLambda \rho w+\nabla \beta -\mu \Delta w=\kappa \rho \operatorname{div} \mathbb{D}\tilde{\eta } &\text{in }\varOmega , \\ \operatorname{div}w=0 & \text{in }\varOmega , \\ [\!\![\tilde{\eta } ]\!\!]=[\!\![w ]\!\!]= 0 &\text{on }\varSigma , \\ [\!\![(\beta - g \rho \tilde{\eta }_{3})I -\mathbb{D}(\mu w+ \kappa \rho \tilde{\eta })]\!\!]e_{3}= \vartheta \Delta _{ \mathrm{h}}\tilde{\eta }_{3} e_{3} &\text{on }\varSigma , \\ (\tilde{\eta },w) = {0}& \text{on }\varSigma _{-}^{+} , \end{cases} $$

and then eliminating η̃ by using the first equation, we arrive at boundary value problem (3.5) for w and β. Multiplying (3.5)₁ by w in $L^{2}$, and using the formula of integral by parts and conditions (3.5)₂–(3.5)₅, we have

$$ \varLambda ^{2} \Vert \sqrt{\rho }w \Vert ^{2}_{0}= -\mathcal{E}(w)- \Vert \sqrt{ \varLambda \mu }\mathbb{D}w \Vert ^{2}_{0}/2, $$

where we have defined that

$$ \mathcal{E}(w):= \vartheta \vert \nabla _{\mathrm{h}}w_{3} \vert _{0}^{2}-g[\!\![\rho ]\!\!]\vert w_{3} \vert _{0}^{2}+ \Vert \sqrt{\kappa \rho } \mathbb{D}w \Vert ^{2}_{0}/2. $$

From the view point of energy, if

$$ \mathcal{E}(w)< 0 \quad \text{for some } w\in H^{1}_{\sigma }, $$

(3.8)

the linearized stratified VRT problem may be unstable, otherwise stable. Obviously, the above instability condition (3.8) is equivalent to

$$ C_{\vartheta }:=\sup_{w\in H_{\sigma ,3}^{1}} \frac{g[\!\![\rho ]\!\!]\vert w_{3} \vert ^{2}_{0}- \Vert \sqrt{\kappa \rho }\mathbb{D}w \Vert _{0} ^{2}/2}{\vartheta \vert \nabla _{h}w_{3} \vert _{0}^{2}}>1. $$

(3.9)

In this article, we assume that ϑ is a constant, then we derive from (3.9) that

$$ \vartheta < \vartheta _{\mathrm{c}}:=\sup_{\omega \in H_{\sigma ,3} ^{1}} \frac{g[\!\![\rho ]\!\!]\vert w_{3} \vert ^{2}_{0}- \Vert \sqrt{ \kappa \rho }\mathbb{D}w \Vert _{0}^{2}/2}{ \vert \nabla _{h}w_{3} \vert ^{2}_{0}}. $$

(3.10)

Under (3.10), the linearized stratified VRT problem is obviously unstable, if there exists a solution $(w,\beta )$ to boundary value problem (3.5) with $\varLambda >0$.

To look for the unstable solution, we use a modified variational method of PDEs, and thus modify (3.5) as follows:

$$ \textstyle\begin{cases} \alpha \rho w+s(\nabla \beta -\mu \Delta w)=\kappa \rho \operatorname{div} \mathbb{D}w &\text{in }\varOmega , \\ \operatorname{div}w=0 & \text{in }\varOmega , \\ [\!\![w ]\!\!]= 0 &\text{on }\varSigma , \\ [\!\![(s\beta - g \rho w_{3})I -\mathbb{D}((s\mu +\kappa \rho )w) ]\!\!]e_{3}= \vartheta \Delta _{\mathrm{h}}w_{3} e_{3} &\text{on }\varSigma , \\ w = {0}& \text{on }\varSigma _{-}^{+}, \end{cases} $$

(3.11)

where $s>0$ is a parameter. In order to emphasize the dependence of s upon α and ϑ, we will write $\alpha (s,\vartheta )=\alpha$.

By modified problem (3.11), we can find that it enjoys the following variational identity:

$$ \alpha (s,\vartheta ) \Vert \sqrt{ {\rho }}w \Vert _{0}^{2}= \mathcal{F}(w,s). $$

Thus, by a standard variational approach, there is a maximizer $w\in \mathcal{A}$ of the functional $\mathcal{F}$ defined on $\mathcal{A}$; furthermore, w is just a weak solution to (3.11) with α defined by the relation

$$ \alpha ( s ,\vartheta ) =\sup_{w\in \mathcal{A} } \mathcal{F}(w,s ) \in \mathbb{R}, $$

(3.12)

see Proposition 4.1. Next we further use the method of difference quotients and the existence theory of the stratified (steady) Stokes problem to improve the regularity of the weak solution, and thus we can prove that $(w,\beta )\in H^{\infty }$ is a classical solution to boundary value problem (3.11), see Proposition 4.2.

In view of instability condition (3.4) and the definition of $\alpha (s ,\vartheta )$, we can infer that, for given ϑ, the function $\alpha (s,\cdot )$ on the variable s enjoys some good properties (see Proposition 4.3), which imply that there exists Λ satisfying the fixed-point relation

$$ \varLambda =\sqrt{\alpha (\varLambda ,\cdot )} \in (0, \mathcal{M}_{\vartheta }). $$

(3.13)

After that we obtain a nontrivial solution $(w,\beta )\in H^{\infty }$ to (3.5) with Λ defined by (3.13), and therefore the linear instability follows. Furthermore, Λ is the largest growth rate of RT instability in the linearized stratified VRT problem (see Proposition 4.4), and thus we get Theorem 3.1.

Next, we turn to introducing the second main result on the properties of largest growth rate constructed by (3.13).

Theorem 3.2

The largest growth rate $\varLambda _{\vartheta }:=\varLambda$ in Theorem 3.1 enjoys the estimate (3.1). Moreover,

$$ \varLambda _{\vartheta }\textit{ strictly decreases and is continuous with respect to }\vartheta \in [0,\vartheta _{\mathrm{c}}). $$

(3.14)

In particular, we have $\varLambda _{\vartheta }\rightarrow 0$ as $\vartheta \rightarrow \vartheta _{\mathrm{c}}$.

Here we briefly introduce the idea of its proof. We can find that, for fixed s, $\alpha (\cdot ,\vartheta )$ defined by (3.12) is continuous with respect to ϑ and strictly decreases (see Proposition 4.5). Therefore, by some analysis based on the definition of continuity and the fixed-point relation (3.13), we can show that $\varLambda _{\vartheta }:=\varLambda$ also inherits the continuity and monotonicity of $\alpha (\cdot ,\vartheta )$. Finally, we derive (3.1) from (3.6) naturally by some estimate techniques. A more detailed proof of Theorem 3.2 will be presented in Sect. 4.2.

4 Proof of main theorems

4.1 The result of linear instability

This subsection is devoted to the proof of Theorem 3.1. First of all, we will use modified variational method to construct unstable solutions for the linearized stratified VRT problem. Guo and Tice firstly use this method for constructing unstable solutions to a class of ordinary differential equations arising from a linearized RT instability problem [9]. In this paper, we will directly apply Guo and Tice’s modified variational method to the partial differential equations (3.5), and thus obtain a linear instability result of the VRT problem by further using an existence theory of stratified Stokes problem. Next, we begin to prove Theorem 3.1 by four steps.

(1) Existence of weak solutions to the modified problem

We investigate the existence of weak solutions to the following modified problem:

$$ \textstyle\begin{cases} \alpha (s,\vartheta ) {\rho } w+s(\nabla \beta -\mu \Delta w)=\kappa \rho \operatorname{div}\mathbb{D}w &\text{in }\varOmega , \\ \operatorname{div}w=0 &\text{in }\varOmega , \\ [\!\![w ]\!\!]=0 &\text{on }\varSigma , \\ [\!\![(s\beta -g\rho w_{3}) I-\mathbb{D}((s\mu +\kappa \rho )w) ]\!\!]e_{3}= \vartheta \Delta _{\mathrm{h}}w_{3} e_{3}& \text{on }\varSigma , \\ w = {0}& \text{on }\varSigma _{-}^{+}, \end{cases} $$

(4.1)

where $s>0$ is any given. In order to prove the existence of weak solutions of the above problem, we first consider the variational problem of the functional $\mathcal{F}(\varpi ,s )$:

$$ \alpha (s,\vartheta ): =\sup_{\varpi \in \mathcal{A}} \mathcal{F}( \varpi ,s ) $$

(4.2)

for given $s>0$, where we have defined that

$$ \mathcal{F}(\varpi ,s):= -\bigl(\mathcal{E}(\varpi )+s \Vert \sqrt{\mu } \mathbb{D}\varpi \Vert _{0}^{2}/2\bigr). $$

And we sometimes denote $\alpha (s,\vartheta )$ and $\mathcal{F}( \varpi ,s)$ by α (or $\alpha (s)$) and $\mathcal{F}(\varpi )$ for simplicity, resp. Then we have the following conclusions.

Proposition 4.1

Let $s>0$ be any given.

• (1) In variational problem (4.2), $\mathcal{F}(\varpi )$ achieves its supremum on $\mathcal{A}$.

• (2) Let w be a maximizer and $\alpha := \sup_{\varpi \in \mathcal{A}}\mathcal{F}(\varpi )$, then w is a weak solution of boundary problem (4.1) with given α.

Proof

Noting that

$$ \vert v \vert _{0}^{2} \lesssim \Vert v \Vert _{0} \Vert \partial _{3} v \Vert _{0} \quad \text{for any }v \in H_{0}^{1}, $$

(4.3)

thus, by Young’s inequality and Korn’s inequality (2.4), we know that $\{\mathcal{F}(\varpi )\}_{\varpi \in \mathcal{A}}$ has an upper bound for any $\varpi \in \mathcal{A}$. Hence there is a maximizing sequence $\{w^{n}\}_{n=1}^{\infty }\subset \mathcal{A}$, which satisfies $\alpha =\lim_{n\to \infty } \mathcal{F}(w_{n})$. In addition, making use of (4.3), the fact $\Vert \sqrt{ {\rho }}w^{n} \Vert _{0}=1$, trace estimate (2.6) and Young’s and Korn’s inequalities, we get $\Vert w^{n} \Vert _{1}+\vartheta \vert \nabla _{\mathrm{h}}w^{n}_{3} \vert _{0} \leqslant c_{1}$ for some constant $c_{1}$, which is independent of n. Therefore, by (4.3) and the well-known Rellich–Kondrachov compactness theorem, there exist a subsequence, still labeled by $w^{n}$, and a function $w\in \mathcal{A}$ such that

$$ \begin{aligned} & w^{n}\rightharpoonup w\quad \text{in }H^{1}_{\sigma }, \quad\quad w^{n}\to w\quad \text{in }L^{2}, \quad\quad w^{n}\big| _{y_{3}=0}\to w| _{y_{3}=0}\quad \text{in }L^{2}( \mathbb{T}), \\ & w^{n}_{3}\big| _{y_{3}=0}\rightharpoonup w_{3}| _{y_{3}=0}\quad \text{in }H^{1}( \mathbb{T})\text{ if }\vartheta \neq 0. \end{aligned} $$

Using the above convergence results and the lower semicontinuity of weak convergence, we obtain

$$ -\alpha =\liminf_{n\to \infty }\bigl(-\mathcal{F}\bigl(w^{n} \bigr)\bigr) \geqslant - \mathcal{F}(w) \geqslant -\alpha . $$

Hence, w is a maximum point of the functional $\mathcal{F}(\varpi )$ with respect to $\varpi \in \mathcal{A}$.

Obviously, w constructed above is also a maximum point of the functional $\mathcal{F}(\varpi )/ \Vert \sqrt{\rho }\varpi \Vert ^{2}_{0}$ with respect to $\varpi \in H_{\sigma , \vartheta }^{1}$. Furthermore $\alpha = \mathcal{F}(w)/ \Vert \sqrt{\rho }w \Vert ^{2}_{0}$. Therefore, for any given $\varphi \in H^{1}_{\sigma ,\vartheta }$, the point $t=0$ is the maximum point of the function

$$ I(t):=\mathcal{F}(w+t\varphi )- \int \alpha {\rho } \vert w+t\varphi \vert ^{2} \, \mathrm{d}y\in C^{1}(\mathbb{R}). $$

Then, by computing out $I'(0)=0$, we have the following weak form:

$$\begin{aligned} & \frac{1}{2} \int (s\mu +\kappa \rho ) \mathbb{D} w:\mathbb{D}\varphi \,\mathrm{d}y + \vartheta \int _{\varSigma }\nabla _{\mathrm{h}}w_{3}\cdot \nabla _{\mathrm{h}}\varphi _{3} \,\mathrm{d}y_{\mathrm{h}} \\ & \quad = g[\!\![\rho ]\!\!]\int _{\varSigma }w_{3}\varphi _{3} \, \mathrm{d}y_{\mathrm{h}}- \alpha \int \rho w\cdot \varphi \,\mathrm{d}y. \end{aligned}$$

(4.4)

Notice the fact, for any $f^{1}, f^{2} \in H^{1}$, and any matrices $A, B\in \mathbb{R}^{3\times 3}$,

$$\begin{aligned} \frac{1}{2} \int \mathbb{D}_{A}f^{1}:\mathbb{D}_{B}f^{2} \,\mathrm{d}y= \int \mathbb{D}_{A}f^{1}:\nabla _{B}f^{2}\,\mathrm{d}y, \end{aligned}$$

(4.5)

thus (4.4) is equivalent to

$$\begin{aligned} & \int (s\mu +\kappa \rho ) \mathbb{D} w:\nabla \varphi \,\mathrm{d}y+ \vartheta \int _{\varSigma }\nabla _{\mathrm{h}}w_{3}\cdot \nabla _{ \mathrm{h}}\varphi _{3}\,\mathrm{d}y_{\mathrm{h}} = g[\!\![\rho ]\!\!]\int _{\varSigma }w_{3}\varphi _{3}\, \mathrm{d}y_{\mathrm{h}}- \alpha \int \rho w\cdot \varphi \,\mathrm{d}y. \end{aligned}$$

This shows that w is a weak solution of modified problem (4.1). □

(2) Improving the regularity of weak solution

By Proposition 4.1, we know that boundary value problem (4.1) admits a weak solution $w\in H^{1}_{\sigma ,\vartheta }$. Next, we will further improve the regularity of w.

Proposition 4.2

Let w be a weak solution of boundary value problem (4.1), then $w\in H^{\infty }$.

Proof

First of all, we shall establish the following preliminary conclusion:

For any $i\geqslant 0$ , we have

$$ w \in H^{1,i}_{\sigma ,\vartheta } $$

(4.6)

and

$$\begin{aligned} &\frac{1}{2} \int (s\mu +\kappa \rho ) \mathbb{D} \partial _{\mathrm{h}} ^{i} w :\mathbb{D}\varphi \,\mathrm{d}y + \vartheta \int _{\varSigma } \nabla _{\mathrm{h}} \partial _{\mathrm{h}}^{i} w_{3}\cdot \nabla _{\mathrm{h}} \varphi _{3}\,\mathrm{d}y_{\mathrm{h}} \\ &\quad = g[\!\![\rho ]\!\!]\int _{\varSigma }\partial _{\mathrm{h}} ^{i} w_{3}\varphi _{3}\,\mathrm{d}y_{\mathrm{h}}-\alpha \int \rho \partial _{\mathrm{h}}^{i} w\cdot \varphi \, \mathrm{d}y. \end{aligned}$$

(4.7)

By induction, it is obvious to see that the above assertion can reduce to verifying the following recurrence relation:

For given $i\geqslant 0$ and any $\varphi \in H^{1}_{\sigma , \vartheta }$, if $w \in H^{1,i}_{\sigma ,\vartheta }$ satisfies ( 4.7 ), then

$$ w \in H^{1,i+1}_{\sigma ,\vartheta } $$

(4.8)

and w satisfies

$$\begin{aligned} &\frac{1}{2} \int (s\mu +\kappa \rho ) \mathbb{D} \partial _{\mathrm{h}} ^{i+1} w :\mathbb{D} \varphi \,\mathrm{d}y+ \vartheta \int _{\varSigma } \nabla _{\mathrm{h}} \partial _{\mathrm{h}}^{i+1} w_{3}\cdot \nabla _{\mathrm{h}} \varphi _{3}\,\mathrm{d}y_{\mathrm{h}} \\ & \quad = g[\!\![\rho ]\!\!]\int _{\varSigma }\partial _{\mathrm{h}} ^{i+1} w_{3}\varphi _{3} \,\mathrm{d}y_{\mathrm{h}}-\alpha \int \rho \partial _{\mathrm{h}}^{i+1} w\cdot \varphi \, \mathrm{d}y. \end{aligned}$$

(4.9)

Next, we use the method of difference quotients to verify the above recurrence relation.

Now, for any $\varphi \in H^{1}_{\sigma ,\vartheta }$, we assume that $w \in H^{1,i}_{\sigma ,\vartheta }$ satisfies (4.7). Noting that $\partial _{\mathrm{h}}^{i}w \in H^{1}_{\sigma ,\vartheta }$, we can deduce from (4.7) that, for $j=1$ and 2,

$$\begin{aligned} & \frac{1}{2} \int (s\mu +\kappa \rho ) \mathbb{D} \partial _{ \mathrm{h}}^{i} w : \mathbb{D} D_{j}^{h} \varphi \,\mathrm{d}y+ \vartheta \int _{\varSigma }\nabla _{\mathrm{h}} \partial _{\mathrm{h}}^{i} w_{3} \cdot \nabla _{\mathrm{h}} D_{j}^{h} \varphi _{3} \,\mathrm{d}y_{ \mathrm{h}} \\ &\quad = g[\!\![\rho ]\!\!]\int _{\varSigma }\partial _{\mathrm{h}} ^{i} w _{3} D_{j}^{h} \varphi _{3}\, \mathrm{d}y_{\mathrm{h}}-\alpha \int \rho \partial _{\mathrm{h}}^{i} w\cdot D_{j}^{h} \varphi \,\mathrm{d}y \end{aligned}$$

and

$$\begin{aligned} & \frac{1}{2} \int (s\mu +\kappa \rho ) \mathbb{D} \partial _{ \mathrm{h}}^{i} w : \mathbb{D} D_{j}^{-h} D_{j}^{h} \partial _{ \mathrm{h}}^{i} w \,\mathrm{d}y+ \vartheta \int _{\varSigma }\nabla _{ \mathrm{h}}\partial _{\mathrm{h}}^{i} w_{3}\cdot \nabla _{\mathrm{h}} D _{j}^{-h} D_{j}^{h}\partial _{\mathrm{h}}^{i} w_{3} \,\mathrm{d}y_{ \mathrm{h}} \\ &\quad = g[\!\![\rho ]\!\!]\int _{\varSigma }\partial _{\mathrm{h}} ^{i} w_{3} D_{j}^{-h} D_{j}^{h} \partial _{\mathrm{h}}^{i} w_{3} \,\mathrm{d}y_{\mathrm{h}}- \alpha \int \rho \partial _{\mathrm{h}}^{i} w \cdot D_{j}^{-h} D_{j}^{h} \partial _{\mathrm{h}}^{i} w \,\mathrm{d}y, \end{aligned}$$

which yield that

$$\begin{aligned} &\frac{1}{2} \int (s\mu +\kappa \rho ) \mathbb{D} D_{j}^{-h} \partial _{\mathrm{h}}^{i} w : \mathbb{D} \varphi \,\mathrm{d}y + \vartheta \int _{\varSigma }\nabla _{\mathrm{h}} D_{j}^{-h} \partial _{\mathrm{h}} ^{i} w_{3}\cdot \nabla _{\mathrm{h}} \varphi _{3} \,\mathrm{d}y_{ \mathrm{h}} \\ &\quad = g[\!\![\rho ]\!\!]\int _{\varSigma }D_{j}^{-h} \partial _{\mathrm{h}}^{i} w_{3} \varphi _{3}\, \mathrm{d}y_{\mathrm{h}}- \alpha \int \rho D_{j}^{-h}\partial _{\mathrm{h}}^{i} w\cdot \varphi \,\mathrm{d}y \end{aligned}$$

(4.10)

and

$$\begin{aligned} & \bigl\Vert \sqrt{s\mu +\kappa \rho }\mathbb{D} D_{j}^{h} \partial _{ \mathrm{h}}^{i} w \bigr\Vert ^{2}_{0}/2+ \vartheta \bigl\vert D_{j}^{h}\nabla _{\mathrm{h}} \partial _{\mathrm{h}}^{i} w_{3} \bigr\vert ^{2}_{0} \\ & \quad \lesssim g[\!\![\rho ]\!\!]\bigl\vert D_{j}^{h} \partial _{\mathrm{h}} ^{i} w_{3} \bigr\vert ^{2}_{0} + \vert \alpha \vert \bigl\Vert \sqrt{ \rho } D_{j}^{h} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{0}, \end{aligned}$$

(4.11)

resp.

By Korn’s inequality, it is easy to get that

$$ \bigl\Vert D_{j}^{h} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{1}\lesssim \bigl\Vert \sqrt{s \mu + \kappa \rho }\mathbb{D} D_{j}^{h} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{0}, $$

therefore, using (4.3), Young’s inequality, and the first conclusion in Lemma 2.2, we further deduce from (4.11) that

$$\begin{aligned} \bigl\Vert D^{h}_{\mathrm{h}} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{1}+ \vartheta \bigl\vert D^{h}_{\mathrm{h}}\nabla _{\mathrm{h}} \partial _{\mathrm{h}}^{i} w _{3} \bigr\vert ^{2}_{0} \lesssim \bigl\Vert D^{h}_{\mathrm{h}} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{0} \lesssim \bigl\Vert \nabla _{\mathrm{h}} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{0}\lesssim 1. \end{aligned}$$

Thus, using (4.3), trace estimate (2.6), and the second conclusion in Lemma 2.2, we can find that there exists a subsequence of $\{-h\}_{h\in \mathbb{R}}$ (still denoted by −h) such that

$$ \textstyle\begin{cases} D^{-h}_{\mathrm{h}} \partial _{\mathrm{h}}^{i} w \rightharpoonup \nabla _{\mathrm{h}}\partial _{\mathrm{h}}^{i} w \quad \text{in }H^{1}_{ \sigma }, \quad\quad D^{-h}_{\mathrm{h}} \partial _{\mathrm{h}}^{i} w \to \nabla _{\mathrm{h}}\partial _{\mathrm{h}}^{i} w \quad \text{in }L^{2}, \\ D^{-h}_{\mathrm{h}} \partial _{\mathrm{h}}^{i} w| _{y_{3}=0} \to \nabla _{\mathrm{h}}\partial _{\mathrm{h}}^{i} w| _{y_{3}=0} \quad \text{in }L^{2}(\mathbb{T}), \\ D^{-h}_{\mathrm{h}}\partial _{\mathrm{h}}^{i} w_{3}| _{\varSigma }\rightharpoonup \nabla _{\mathrm{h}}\partial _{\mathrm{h}}^{i} w_{3}| _{\varSigma }\quad \text{in }H^{1}(\mathbb{T}) \text{ if }\vartheta \neq 0. \end{cases} $$

(4.12)

Exploiting the regularity of w in (4.12) and the fact $w\in H_{\sigma ,\vartheta }^{1,i}$, we get (4.8). Moreover, using the limit results in (4.12), we can deduce (4.9) from (4.10). This means that we have completed the proof of the recurrence relation, and thus (4.6) holds.

Now, with (4.6) in hand, we consider a stratified Stokes problem:

$$ \textstyle\begin{cases} s \nabla \beta ^{k} -(s\mu +\kappa \rho ) \Delta \omega ^{k} =-\alpha \rho \partial _{\mathrm{h}}^{k} w &\text{in } \varOmega , \\ \operatorname{div}\omega ^{k}=0&\text{in } \varOmega , \\ [\!\![\omega ^{k} ]\!\!]=0, \quad\quad [\!\![( s \beta ^{k} I - (s \mu +\kappa \rho )\mathbb{D}\omega ^{k})e_{3} ]\!\!]= \partial _{\mathrm{h}}^{k}\mathcal{L}^{1}&\text{on }\varSigma , \\ \omega ^{k} =0 &\text{on }\varSigma _{-}^{+}, \end{cases} $$

(4.13)

where $k\geqslant 0$ is a given integer, and we have defined that

$$ \begin{aligned} & \mathcal{L}^{1} :=g [\!\![\rho ]\!\!]w_{3}e_{3} + \vartheta \Delta _{\mathrm{h}}w_{3} e_{3}. \end{aligned} $$

Because of regularity (4.6) of w, we see that $\partial _{\mathrm{h}}^{k}w\in L^{2}$ and $\partial _{\mathrm{h}}^{k} \mathcal{L}^{1} \in H^{1}(\mathbb{T})$. Using the existence theory of stratified Stokes problem (see Lemma 2.1), there is a unique strong solution $(\omega ^{k},\beta ^{k})\in H^{2}\times \underline{H}^{1}$ of the above problem (4.13).

Multiplying (4.13)₁ by $\varphi \in H^{1}_{ \sigma ,\vartheta }$ in $L^{2}$ (i.e., taking the inner product in $L^{2}$), and using (4.13)₂–(4.13)₄ and the integration by parts, we arrive at

$$\begin{aligned} &\frac{1}{2} \int (s\mu +\kappa \rho )\mathbb{D}\omega ^{k}:\mathbb{D} \varphi \,\mathrm{d}y \\ & \quad = g[\!\![\rho ]\!\!]\int _{\varSigma }\partial _{\mathrm{h}}^{k} w_{3}\varphi _{3} \,\mathrm{d}y_{\mathrm{h}} - \int _{\varSigma }\vartheta \partial _{\mathrm{h}}^{k} \nabla _{\mathrm{h}}w_{3}\cdot \nabla _{\mathrm{h}}\varphi _{3}\,\mathrm{d}y_{\mathrm{h}} - \int \alpha \rho \partial _{\mathrm{h}}^{k} w\varphi \, \mathrm{d}y. \end{aligned}$$

(4.14)

And then, subtracting the two identities (4.7) and (4.14) yields that

$$ \int (s\mu +\kappa \rho )\mathbb{D}\bigl(\partial _{\mathrm{h}}^{k} w-\omega ^{k}\bigr):\mathbb{D} \varphi \,\mathrm{d}y=0. $$

Taking $\varphi := \partial _{\mathrm{h}}^{k} w-\omega ^{k}\in H^{1} _{\sigma ,\vartheta }$ in the above identity, and using Korn’s inequality, we see that $\omega ^{k}= \partial _{\mathrm{h}}^{k} w$. Thus we immediately find that

$$ \partial _{\mathrm{h}}^{k} w\in H^{2} \quad \text{for any }k\geqslant 0, $$

(4.15)

which implies $\partial _{\mathrm{h}}^{k}w\in H^{1}$, and $\partial _{\mathrm{h}}^{k}\mathcal{L}^{1}\in H^{2}(\mathbb{T})$ for any $k\geqslant 0$. Therefore, applying the stratified Stokes estimate (2.2) to (4.13), we obtain

$$ \partial _{\mathrm{h}}^{k} w\in H^{3} \quad \text{for any }k\geqslant 0. $$

(4.16)

By induction, it is obvious to see that we can easily follow the improving regularity method from (4.15) to (4.16) to deduce that $w\in H^{\infty }$. Moreover, we have $\beta :=\beta ^{0}\in H^{\infty }$; furthermore, $\beta ^{k}$ in (4.13) is equal to $\partial ^{k} _{\mathrm{h}}\beta$.

Finally, recalling the embedding $H^{k+2}\hookrightarrow C^{0}(\overline{ \varOmega })$ for any $k\geqslant 0$, it is easy to see that $(w,\beta )$ constructed above is indeed a classical solution to modified problem (4.1). □

(3) Some properties of the function $\alpha (s)$

Now we devote to the derivation of some properties of the function $\alpha (s)$, which can make sure the existence of fixed point of $\sqrt{\alpha (s)}$ in $\mathbb{R}^{+}$.

Proposition 4.3

For given $\vartheta \in \mathbb{R}^{+}_{0}$, we have

$$\begin{aligned} &\alpha (s_{2}) < \alpha (s_{1})\quad \textit{for any }s_{2} >s_{1}>0, \end{aligned}$$

(4.17)

$$\begin{aligned} & \alpha (s) \in C^{0,1}_{\mathrm{loc}}\bigl( \mathbb{R}^{+}\bigr), \end{aligned}$$

(4.18)

$$\begin{aligned} & \alpha (s)>0\quad \textit{on some interval }(0,c_{2})\textit{ for } \vartheta \in [0, \vartheta _{\mathrm{c}}), \end{aligned}$$

(4.19)

$$\begin{aligned} &\alpha (s)< 0\quad \textit{on some interval }(c_{3},\infty ). \end{aligned}$$

(4.20)

Proof

First, we verify (4.17). For given $s_{2}>s_{1}$, there is $v^{s_{2}}\in \mathcal{A}$ such that $\alpha (s_{2}) = \mathcal{F}(v ^{s_{2}},s_{2})$. Hence, by the fact $\Vert \sqrt{\rho }v^{s_{2}} \Vert _{0}=1$ and Korn’s inequality,

$$ \alpha (s_{1})\geqslant \mathcal{F}\bigl(v^{s_{2}},s_{1} \bigr) = \alpha (s_{2}) + ( s_{2}- {s_{1}}) \bigl\Vert \sqrt{\mu }\mathbb{D}v^{s_{2}} \bigr\Vert _{0}^{2}/2 > \alpha (s_{2}), $$

which yields (4.17).

Then we turn to proving (4.18). Choosing a bounded interval $[c_{4},c_{5}]\subset (0,\infty )$, then, for any $s\in [c _{4},c_{5}]$, there is a function $v^{s}$ satisfying $\alpha (s)= \mathcal{F}(v^{s},s)$. Hence, by monotonicity (4.17), we arrive at

$$ \alpha (c_{5})+ c_{4} \bigl\Vert \sqrt{\mu } \mathbb{D}v^{s} \bigr\Vert _{0}^{2}/4\leqslant \mathcal{F}\bigl(v^{s},s/2\bigr) \leqslant \alpha (s/2) \leqslant \alpha ( c _{4}/2), $$

which yields

$$ \bigl\Vert \sqrt{\mu } \mathbb{D}v^{s} \bigr\Vert _{0}^{2}/2 \leqslant 2\bigl(\alpha (c_{4}/2)- \alpha (c_{5})\bigr)/c_{4}=:\xi \quad \text{for any }s\in [c_{4},c_{5}]. $$

Therefore, for any $s_{1}, s_{2}\in [c_{4},c_{5}]$,

$$ \begin{aligned} \alpha (s_{1})-\alpha (s_{2}) \leqslant & \mathcal{F}\bigl(v^{s_{1}},s_{1}\bigr)- \mathcal{F} \bigl(v^{s_{1}},s_{2}\bigr)\leqslant \xi \vert {s_{2}}-s_{1} \vert \end{aligned} $$

and

$$ \alpha (s_{2})-\alpha (s_{1})\leqslant \xi \vert {s_{2}}-s_{1} \vert , $$

which immediately imply $\vert \alpha (s_{1})-\alpha (s_{2}) \vert \leqslant \xi \vert {s_{2}}-s_{1} \vert$. Hence (4.18) holds.

Finally, we can infer (4.19) from the definition of α by using Korn’s inequality and (4.3), while (4.20) can obviously get from the definition of α and (3.8). □

(4) Construction of an interval for fixed point

To begin with, let $\mathfrak{I}:=\sup \{\text{all the real constant }s, \text{which satisfy that }\alpha ( \tau )>0\text{ for any }\tau \in (0,s)\}$. By (4.19) and (4.20), it is clear that $\mathfrak{I}\in \mathbb{R}^{+}$. In addition, $\alpha (s)>0$ for any $s\in (0,\mathfrak{I})$, and, by the continuity of $\alpha (s)$, we have

$$ \alpha ( \mathfrak{I})=0. $$

(4.21)

Applying the upper boundedness and the monotonicity of $\alpha (s)$, we see that

$$ \lim_{s\rightarrow 0}\alpha (s)=\varsigma \quad \text{for some positive constant }\varsigma . $$

(4.22)

Then, using (4.21), (4.22), and the continuity of ${\alpha }(s)$ on $(0,\mathfrak{I})$, we immediately find by a fixed-point argument on $(0,\mathfrak{I})$ that there exists a unique $\varLambda \in (0,\mathfrak{I})$ satisfying

$$ \varLambda =\sqrt{ \alpha (\varLambda )}= \sqrt{ \sup _{\varpi \in \mathcal{A}}\mathcal{F}(\varpi , \varLambda )}\in (0, \mathfrak{I}). $$

(4.23)

Hence, we get a classical solution $(w,\beta ) \in H^{\infty }$ to boundary problem (3.5) with Λ constructed by (4.23). Furthermore, it is easy to see that

$$ \varLambda = \sqrt{ \mathcal{F}(w, \varLambda )}>0, $$

(4.24)

and (3.7) directly follows (4.24) and the fact $w\in H_{\sigma }^{1}$.

Next, we shall prove that Λ constructed in the above is the largest growth rate of RT instability in the linearized stratified VRT problem, and thus complete the proof of Theorem 3.1.

Proposition 4.4

Under the assumptions of Theorem 3.1, $\varLambda >0$ constructed by (4.23) is the largest growth rate of RT instability in the linearized stratified VRT problem.

Proof

According to the definition of largest growth rate, it suffices to prove that Λ enjoys the first condition in Definition 3.1.

First, let u be a strong solution to the linearized stratified VRT problem. Then we can get that, for a.e. $t\in I_{T}$ and all $w\in H_{\sigma }^{1}$,

$$\begin{aligned} \int \rho u_{t}\cdot w\,\mathrm{d}y={} & \int (\mu \Delta u-\nabla q) \cdot w\,\mathrm{d}y+ \int \kappa \rho \operatorname{div}\mathbb{D}\eta \cdot w \,\mathrm{d}y \\ ={} & \int _{\varSigma }\bigl(g[\!\![\rho ]\!\!]\eta _{3}w_{3}+\vartheta \Delta _{\mathrm{h}}\eta _{3} w_{3} \bigr)\,\mathrm{d}y_{\mathrm{h}}- \int \mathbb{D}(\mu u+\kappa \rho \eta ): \nabla w\,\mathrm{d}y. \end{aligned}$$

(4.25)

Thus,

$$\begin{aligned} \frac{\mathrm{d}}{\mathrm{d}t} \int \rho u_{t}\cdot w\,\mathrm{d}y = \int _{\varSigma }\bigl(g[\!\![\rho ]\!\!]u_{3}w_{3}+\vartheta \Delta _{\mathrm{h}}u_{3}w_{3} \bigr)\,\mathrm{d}y_{\mathrm{h}}- \int \mathbb{D}(\mu u_{t}+\kappa \rho u): \nabla w\, \mathrm{d}y. \end{aligned}$$

(4.26)

Exploiting regularity of $(\eta ,u)$, we can see that the right-hand side of (4.26) is bounded above by $A(t)( \Vert w \Vert _{1}+ \vert w \vert _{1})$ for some positive function $A(t)\in L^{2}(I _{T})$. Then there exists $f\in L^{2}(I_{T}, H^{-1}_{\sigma })$ such that, for a.e. $t\in I_{T}$,

$$ \langle f, w\rangle _{H^{-1}_{\sigma }\times H^{1}_{\sigma }}:= \int _{\varSigma }\bigl(g [\!\![\rho ]\!\!]u_{3}w_{3}+\vartheta \Delta _{\mathrm{h}}u _{3} w_{3} \bigr)\,\mathrm{d}y_{\mathrm{h}}- \int \mathbb{D}(\mu u_{t}+\kappa \rho u): \nabla w\, \mathrm{d}y. $$

(4.27)

Therefore, it follows from Lemma 2.6 that

$$ (\rho u_{t})_{t}=f\in L^{2}\bigl(I_{T},H^{-1}_{\sigma } \bigr). $$

Moreover, using the classical regularization method (referring to Theorem 3 in Chap. 5.9 in [7] and Lemma 6.5 in [37]), we get

$$\begin{aligned} &\frac{1}{2}\frac{\mathrm{d}}{\mathrm{d}t} \int \rho \vert u_{t} \vert ^{2} \,\mathrm{d}y= \bigl\langle \partial _{t}(\rho u_{t}), u_{t} \bigr\rangle _{H^{-1}_{\sigma } \times H^{1}_{\sigma }}, \\ & \int _{\varSigma }\Delta _{\mathrm{h}}u_{3} \partial _{t} u_{3} \,\mathrm{d}y_{\mathrm{h}}=-\frac{1}{2} \frac{\mathrm{d}}{\mathrm{d}t} \int _{\varSigma } \vert \nabla _{\mathrm{h}}u_{3} \vert ^{2}\,\mathrm{d}y_{\mathrm{h}}. \end{aligned}$$

Thus, we can derive from the above two identities and (4.27) that

$$\begin{aligned} \frac{\mathrm{d}}{\mathrm{d}t} \bigl( \Vert \sqrt{\rho } u_{t} \Vert ^{2} _{0}+\mathcal{E}(u ) \bigr)+ \Vert \sqrt{\mu } \mathbb{D} u _{t} \Vert _{0} ^{2} =0. \end{aligned}$$

Next, integrating the above identity in time from 0 to t yields that

$$\begin{aligned} \Vert \sqrt{\rho } u_{t} \Vert ^{2}_{0}+ \mathcal{E}(u )+ \int _{0}^{t} \Vert \sqrt{ \mu } \mathbb{D} u _{s} \Vert _{0}^{2}\,\mathrm{d}s = I^{0}:= \mathcal{E}(u| _{t=0})+ \Vert \sqrt{\rho }u_{t}| _{t=0} \Vert ^{2}_{0}. \end{aligned}$$

(4.28)

Applying Newton–Leibniz’s formula and Young’s inequality, we can find that

$$\begin{aligned} \varLambda \bigl\Vert \sqrt{\mu }\mathbb{D} u(t) \bigr\Vert _{0}^{2} &=\varLambda \bigl\Vert \sqrt{ \mu } \mathbb{D} u^{0} \bigr\Vert _{0}^{2} + 2\varLambda \int _{0}^{t} \int \mu \mathbb{D} u(s):\mathbb{D} u_{s} \,\mathrm{d} y\, \mathrm{d}s \\ & \leqslant \varLambda \bigl\Vert \sqrt{\mu } \mathbb{D} u^{0} \bigr\Vert _{0}^{2} + \int _{0}^{t} \Vert \sqrt{\mu } \mathbb{D} u_{s} \Vert _{0}^{2} \,\mathrm{d}s + \varLambda ^{2} \int _{0}^{t} \bigl\Vert \sqrt{\mu } \mathbb{D} u (s) \bigr\Vert _{0}^{2} \,\mathrm{d}s. \end{aligned}$$

(4.29)

In addition, by (3.6), we arrive at

$$ -\mathcal{E}(u) \leqslant {\varLambda ^{2}} \Vert \sqrt{\rho } {u} \Vert ^{2} _{0} +\frac{\varLambda }{2} \Vert \sqrt{\mu } \mathbb{D} u \Vert ^{2}_{0}. $$

(4.30)

Therefore, we deduce from (4.28)–(4.30) that

$$\begin{aligned} &\frac{1}{\varLambda } \Vert \sqrt{ {\rho }} u_{t} \Vert ^{2}_{0}+ \frac{1}{2} \bigl\Vert \sqrt{\mu } \mathbb{D} u (t) \bigr\Vert _{0}^{2} \\ &\quad \leqslant {\varLambda } \bigl\Vert \sqrt{ {\rho }} u (t) \bigr\Vert ^{2}_{0}+ {\varLambda } \int _{0}^{t} \bigl\Vert \sqrt{\mu } \mathbb{D} u(s) \bigr\Vert _{0}^{2} \,\mathrm{d}s +\frac{I ^{0}+ \varLambda \Vert \sqrt{\mu }\mathbb{D} u^{0} \Vert _{0}^{2} }{\varLambda }. \end{aligned}$$

(4.31)

According to

$$ \begin{aligned} \varLambda \frac{\mathrm{d}}{\mathrm{d}t} \Vert \sqrt{ {\rho }} u \Vert ^{2} _{0} = 2\varLambda \int \rho u(t)\cdot u_{t}\,\mathrm{d} y \leqslant \Vert \sqrt{ \rho } u_{t} \Vert ^{2}_{0} +\varLambda ^{2} \bigl\Vert \sqrt{ {\rho }} u(t) \bigr\Vert ^{2} _{0}, \end{aligned} $$

we further derive the differential inequality from (4.31) as follows:

$$ \begin{aligned} & \frac{\mathrm{d}}{\mathrm{d}t} \Vert \sqrt{ {\rho }} u \Vert ^{2}_{0}+ \frac{1}{2} \bigl\Vert \sqrt{\mu } \mathbb{D} u(t) \bigr\Vert _{0}^{2} \\ & \quad \leqslant 2\varLambda \biggl( \bigl\Vert \sqrt{\rho } u(t) \bigr\Vert ^{2}_{0} +\frac{1}{2} \int _{0} ^{t} \bigl\Vert \sqrt{\mu } \mathbb{D} u(s) \bigr\Vert _{0}^{2} \,\mathrm{d}s \biggr) + \frac{I ^{0} + \varLambda \Vert \sqrt{\mu } \mathbb{D} u^{0} \Vert _{0}^{2} }{\varLambda }. \end{aligned} $$

Applying Gronwall’s inequality [37, Lemma 1.2] to the above inequality, we obtain

$$\begin{aligned} \bigl\Vert \sqrt{ {\rho }} u (t) \bigr\Vert ^{2}_{0}+ \frac{1}{2} \int _{0}^{t} \bigl\Vert \sqrt{ \mu } \mathbb{D} u(s) \bigr\Vert ^{2}_{0} \,\mathrm{d}s \leqslant \biggl( \bigl\Vert \sqrt{ {\rho }}u^{0} \bigr\Vert _{0}^{2}+ \frac{ I^{0}+ \varLambda \Vert \sqrt{\mu } \mathbb{D} u^{0} \Vert _{0}^{2} }{2\varLambda ^{2}} \biggr) e^{2\varLambda t}, \end{aligned}$$

(4.32)

which, together with (4.31), yields

$$\begin{aligned} \frac{1}{\varLambda } \bigl\Vert \sqrt{ {\rho }} u_{t} (t) \bigr\Vert ^{2}_{0} + \frac{1}{2} \bigl\Vert \sqrt{\mu } \mathbb{D} u (t) \bigr\Vert _{0}^{2} \leqslant{} & 2 \biggl(\varLambda \bigl\Vert \sqrt{ {\rho }}u^{0} \bigr\Vert _{0}^{2}+\frac{ I^{0}+ \varLambda \Vert \sqrt{\mu } \mathbb{D} u^{0} \Vert _{0}^{2} }{2\varLambda } \biggr) e^{2\varLambda t} \\ &{} +\frac{I^{0}+ \varLambda \Vert \sqrt{\mu } \mathbb{D} u^{0} \Vert _{0}^{2} }{ \varLambda }. \end{aligned}$$

(4.33)

And then, multiplying (1.13)₂ by $u_{t}$ in $L^{2}$ and using integration by parts, we have

$$\begin{aligned} \int \rho \vert u_{t} \vert ^{2}\,\mathrm{d}y= \int _{\varSigma } [\!\![q ]\!\!]\partial _{t}u_{3} \,\mathrm{d}y_{\mathrm{h}} + \int \mu \Delta u\cdot u _{t}\,\mathrm{d}y+ \int \kappa \rho \operatorname{div}\mathbb{D}\eta \cdot u _{t} \,\mathrm{d}y. \end{aligned}$$

(4.34)

Using (2.14), we can estimate that

$$\begin{aligned} \int _{\varSigma } [\!\![q ]\!\!]\partial _{t}u_{3} \,\mathrm{d}y _{\mathrm{h}} \lesssim & \bigl\vert [\!\![q ]\!\!]\bigr\vert _{1/2} \vert \partial _{t} u_{3} \vert _{-1/2}\lesssim \bigl\vert [\!\![q ]\!\!]\bigr\vert _{1/2} \Vert u_{t} \Vert _{0}. \end{aligned}$$

Moreover, exploiting (1.13)₅ and trace estimate (2.6), we obtain

$$ \bigl\vert [\!\![q ]\!\!]\bigr\vert _{1/2}\lesssim \Vert \eta \Vert _{3}+ \Vert u \Vert _{2}. $$

Applying the above two estimates, we can infer from (4.34) that

$$\begin{aligned} \Vert u_{t} \Vert ^{2}_{0}\lesssim \Vert \eta \Vert _{3}^{2}+ \Vert u \Vert _{2}^{2} , \end{aligned}$$

which implies that

$$ \Vert \sqrt{\rho }u_{t}| _{t=0} \Vert ^{2}_{0} \lesssim \bigl\Vert \bigl(\eta ^{0},u ^{0}\bigr) \bigr\Vert _{3}^{2}. $$

By Korn’s inequality and the above estimate, we deduce from (4.32) and (4.33) that

$$\begin{aligned} \Vert u \Vert _{1}^{2}+ \Vert u_{t} \Vert ^{2}_{0 }+ \int _{0}^{t} \bigl\Vert u(s) \bigr\Vert ^{2}_{1} \,\mathrm{d}s\lesssim e^{2\varLambda t}\bigl( \bigl\Vert \eta ^{0} \bigr\Vert _{3}^{2}+ \bigl\Vert u^{0} \bigr\Vert _{2} ^{2}\bigr) . \end{aligned}$$

Finally, from (1.13)₁ we arrive at

$$\begin{aligned} \begin{aligned} \bigl\Vert \eta (t) \bigr\Vert _{1} \lesssim{} & \bigl\Vert \eta ^{0} \bigr\Vert _{1} + \int _{0}^{t} \Vert \eta _{s} \Vert _{1} \,\mathrm{d}s \lesssim \bigl\Vert \eta ^{0} \bigr\Vert _{1} + \int _{0}^{t} \bigl\Vert u(s) \bigr\Vert _{1}\,\mathrm{d}s \\ \lesssim{} & e^{\varLambda t}\bigl( \bigl\Vert \eta ^{0} \bigr\Vert _{3}+ \bigl\Vert u^{0} \bigr\Vert _{2} \bigr). \end{aligned} \end{aligned}$$

From the two estimates above, we immediately see that Λ satisfies the first condition in Definition 3.1. The proof is completed. □

4.2 The effect of surface tension

In this subsection, to prove Theorem 3.2, we shall further derive relations (3.1) and (3.14) of surface tension coefficient and the largest growth rate. And in order to emphasize the dependence of Λ and $\mathcal{M}$ upon ϑ, we will denote them by $\varLambda _{\vartheta }$ and $\mathcal{M}_{\vartheta }$, respectively. To this end, we need the following auxiliary conclusions (i.e., the properties of $\alpha (s, \vartheta )$ with respect to ϑ).

Proposition 4.5

Let $g>0$, $\rho >0$, and $\mu >0$ be given.

• (1) Strict monotonicity: if $\vartheta _{1}$ and $\vartheta _{2}$ are constants satisfying $0\leqslant \vartheta _{1}<\vartheta _{2}$, then

  $$ \alpha (s,\vartheta _{2})< \alpha (s,\vartheta _{1}) $$

  (4.35)

  for any given $s>0$. Moreover, if $\vartheta _{2}$ further satisfies $\vartheta _{2}<\vartheta _{\mathrm{c}}$,

  $$ \mathcal{M}_{\vartheta _{\mathrm{1}}}>\mathcal{M}_{\vartheta _{ \mathrm{2}}}, $$

  (4.36)

  where

  $$ \mathcal{M}_{\vartheta _{i}}:=\sup \bigl\{ s\in \mathbb{R}\mid \alpha (\tau , \vartheta _{i})>0\textit{ for any } \tau \in (0,s)\bigr\} \quad \textit{and}\quad \alpha (\mathcal{M}_{\vartheta _{i}}, \vartheta _{i})=0. $$

  (4.37)

• (2) Continuity: for given $s>0$, $\alpha (s,\vartheta )\in C ^{0,1}_{\mathrm{loc}}(\mathbb{R}^{+})$ with respect to the variable ϑ.

Proof

(1) To begin with, let $s>0$ be fixed, and $0\leqslant \vartheta _{1}< \vartheta _{2}$. Then there exist functions ${w}^{\vartheta _{i}} \in H^{\infty }\cap \mathcal{A}_{\vartheta _{i}}$, $i=1,2$, such that

$$ \begin{aligned} \alpha (s,\vartheta _{i})= {E} \bigl( {w} ^{\vartheta _{i}}\bigr)-\vartheta _{i} \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{\vartheta _{i}} \bigr\vert _{0}^{2}, \end{aligned} $$

where ${E}( {w}^{\vartheta _{i}}):=g[\!\![\rho ]\!\!]\vert {w}^{\vartheta _{i}}_{3} \vert _{0}^{2} -{s} \Vert \sqrt{\mu } \mathbb{D} {w} ^{\vartheta _{i}} \Vert _{0}^{2}/2- \Vert \sqrt{\kappa \rho } \mathbb{D} {w} ^{\vartheta _{i}} \Vert _{0}^{2}/2$. Since ${w}^{\vartheta _{i}}\in \mathcal{A}_{\vartheta _{i}}$, due to (3.7), we get

$$ \begin{aligned} 0< \bigl\vert {w}^{\vartheta _{2}} \bigr\vert _{0}\lesssim \bigl\vert \nabla _{\mathrm{h}} {w}^{ \vartheta _{2}} \bigr\vert _{0}, \end{aligned} $$

and thus we have

$$ \begin{aligned} \alpha (s,\vartheta _{2})\leqslant \alpha (s,\vartheta _{1})+ {( \vartheta _{1}-\vartheta _{2}) } \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{\vartheta _{2}} \bigr\vert _{0}^{2}< \alpha (s,\vartheta _{1}). \end{aligned} $$

Therefore, we immediately get the desired conclusion (4.35).

Next, we begin to prove (4.36) by contradiction. If $\mathcal{M}_{\vartheta _{\mathrm{1}}}<\mathcal{M}_{\vartheta _{ \mathrm{2}}}$, then we can get from the strict monotonicity of $\alpha (s,\cdot )$ with respect to s and (4.35) that

$$ \begin{aligned} 0=\alpha (\mathcal{M}_{\vartheta _{\mathrm{2}}},\vartheta _{2}) < \alpha(\mathcal{M}_{\vartheta _{\mathrm{2}}},\vartheta _{1}) < \alpha (\mathcal{M}_{\vartheta _{\mathrm{1}}},\vartheta _{1})=0, \end{aligned} $$

which is a paradox. If $\mathcal{M}_{\vartheta _{\mathrm{1}}}= \mathcal{M}_{\vartheta _{\mathrm{2}}}$, using (4.35), we arrive at

$$ \begin{aligned} 0=\alpha (\mathcal{M}_{\vartheta _{\mathrm{2}}},\vartheta _{2})< \alpha (\mathcal{M}_{\vartheta _{\mathrm{2}}},\vartheta _{1})= \alpha ( \mathcal{M}_{\vartheta _{\mathrm{1}}},\vartheta _{1})=0, \end{aligned} $$

which is also a paradox. This yields the desired conclusion.

(2) Let $s>0$ be fixed. We first choose a bounded interval $[b_{1},b _{2}]\subset \mathbb{R}^{+}$. Then, for any given $\theta \in [ b_{1},b _{2}]$, there exists a function ${w}^{\theta }\in \mathcal{A}_{ \vartheta }$ satisfying $\alpha (s,\theta )={E}( {w}^{\theta }) - \theta \vert \nabla _{\mathrm{h}} {w}_{3}^{\theta } \vert _{0}^{2}$. Thus, considering the monotonicity of $\alpha (\cdot ,\theta )$, we easily know that

$$\begin{aligned} \alpha (s,b_{2})+b_{1} \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{\theta } \bigr\vert _{0}^{2}/2 \leqslant{} & \alpha (s,\theta )+\theta \bigl\vert \nabla _{\mathrm{h}} {w}_{3} ^{\theta } \bigr\vert _{0}^{2}/2 \\ \leqslant{} & \alpha (s,\theta /2 ) \leqslant \alpha (s,b_{1}/2), \end{aligned}$$

(4.38)

which yields

$$ \begin{aligned} \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{\vartheta } \bigr\vert _{0}^{2} \leqslant 2 \bigl(\alpha (s,b_{1}/2)-\alpha (s,b_{2}) \bigr)/b_{1} :=K(s) \quad \text{for any }\vartheta \in [b_{1},b_{2}]. \end{aligned} $$

Therefore, for any $\vartheta _{1}$, $\vartheta _{2}\in [b_{1},b_{2}]$,

$$ \begin{aligned} \alpha (s,\vartheta _{1})-\alpha (s, \vartheta _{2})\leqslant{} & {E}\bigl( {w}^{\vartheta _{1}}\bigr)- \vartheta _{1} \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{ \vartheta _{1} } \bigr\vert _{0}^{2} - \bigl({E}\bigl( {w}^{\vartheta _{1}}\bigr)-\vartheta _{2} \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{\vartheta _{1} } \bigr\vert _{0}^{2}\bigr) ) \\ \leqslant {}& K(s) \vert \vartheta _{2}-\vartheta _{1} \vert . \end{aligned} $$

Reversing the role of indices 1 and 2 in the derivation of the above inequality, we can obtain the same boundedness with the indices switched. Thus, we derive that

$$ \begin{aligned} \bigl\vert \alpha (s,\vartheta _{1})- \alpha (s,\vartheta _{2}) \bigr\vert \leqslant K(s) \vert \vartheta _{1}-\vartheta _{2} \vert , \end{aligned} $$

which yields $\alpha (s,\vartheta )\in C^{0,1}_{\mathrm{loc}}( \mathbb{R}^{+})$. The proof is completed. □

By Proposition 4.5, we have finished the proof of properties of $\alpha (s,\vartheta )$ with respect to ϑ. Next, we will complete the proof of Theorem 3.2 in three steps.

(1) The monotonicity of $\varLambda _{\vartheta }$ with respect to the variable $\vartheta \in [0, \vartheta _{\mathrm{c}})$.

First, for given two constants $\vartheta _{1}$ and $\vartheta _{2}$ satisfying $0\leqslant \vartheta _{1}<\vartheta _{2}< \vartheta _{ \mathrm{c}}$, there exist two associated curve functions $\alpha (s, \vartheta _{1})$ and $\alpha (s,\vartheta _{2})$ defined in $(0, \vartheta _{\mathrm{c}})$. Then, from the first assertion in Proposition 4.5, we know that

$$ \alpha (s,\vartheta _{1})>\alpha (s,\vartheta _{2}). $$

Through the analysis, we can see that the fixed point $\varLambda _{\vartheta _{i}}$ satisfying $\varLambda _{\vartheta _{i}}=\sqrt{ \alpha (\varLambda _{\vartheta _{i}})}$ can be obtained from the intersection point of the two curves $y=\sqrt{\alpha (s,\vartheta _{i})}$ and $y=s$ on $(0,\mathcal{M}_{\vartheta _{i}})$ for $i=1$ and 2. Therefore, we can immediately obtain the monotonicity

$$ \begin{aligned} \varLambda _{\vartheta _{1}}>\varLambda _{\vartheta _{2}}\quad \text{for }0\leqslant \vartheta _{1}< \vartheta _{2}< \vartheta _{\mathrm{c}}. \end{aligned} $$

(4.39)

(2) The continuity of $\varLambda _{\vartheta }$.

In order to prove the continuity of $\varLambda _{\vartheta }$, we first need to choose a constant $\vartheta _{0}>0$ and an associated function $\alpha (s,\vartheta _{0})$. Noting that $\alpha (\varLambda _{\vartheta _{0}},\vartheta _{0})=\varLambda ^{2}_{\vartheta _{0}}>0$ and $\alpha ( \cdot ,\vartheta )\in C^{0,1}_{\mathrm{loc}}[0,\vartheta _{\mathrm{c}})$ are continuous and strictly decreasing with respect to ϑ, thus, for any given $\varepsilon >0$, there is a constant $\delta >0$ such that

$$ \begin{aligned} (\vartheta _{0}-\delta ,\vartheta _{0}+\delta ) \subset (0, \vartheta _{\mathrm{c}} ), \alpha ( \varLambda _{\vartheta _{0}},\vartheta _{0}+\delta )>0, \end{aligned} $$

and

$$ \begin{aligned} 0< \sqrt{\alpha (\varLambda _{\vartheta _{0}},\vartheta _{0})}-\sqrt{ \alpha (\varLambda _{\vartheta _{0}},\vartheta _{0}+\delta )}< \varepsilon , 0< \sqrt{\alpha (\varLambda _{\vartheta _{0}},\vartheta _{0}-\delta )}-\sqrt{ \alpha (\varLambda _{\vartheta _{0}},\vartheta _{0})}< \varepsilon . \end{aligned} $$

Therefore, from the above two inequalities, we have

$$ \begin{aligned} \varLambda _{\vartheta _{0}}-\varepsilon < \sqrt{\alpha ( \varLambda _{\vartheta _{0}},\vartheta _{0}+\delta )}\quad \text{and}\quad \sqrt{ \alpha (\varLambda _{\vartheta _{0}},\vartheta _{0}-\delta )}< \varLambda _{\vartheta _{0}}+\varepsilon . \end{aligned} $$

According to the monotonicity of $\varLambda _{\vartheta }$ with respect to ϑ, we arrive at

$$ \begin{aligned} \varLambda _{\vartheta _{0}-\delta }>\varLambda _{\vartheta _{0}}> \varLambda _{\vartheta _{0}+\delta }. \end{aligned} $$

Thus, by the monotonicity of $\alpha (s,\cdot )$ with respect to s, we get

$$ \begin{aligned} \sqrt{\alpha (\varLambda _{\vartheta _{0}},\vartheta _{0}+\delta )}< \sqrt{ \alpha (\varLambda _{\vartheta _{0}+\delta },\vartheta _{0}+\delta )}= \varLambda _{\vartheta _{0}+\delta } \end{aligned} $$

and

$$ \begin{aligned} \sqrt{\alpha (\varLambda _{\vartheta _{0}},\vartheta _{0}-\delta )}>\sqrt{ \alpha (\varLambda _{\vartheta _{0}-\delta },\vartheta _{0}-\delta )}= \varLambda _{\vartheta _{0}-\delta }. \end{aligned} $$

Chaining the five inequalities above, we immediately obtain

$$ \begin{aligned} \varLambda _{\vartheta _{0}}-\varepsilon < \varLambda _{\vartheta _{0}+\delta }< \varLambda _{\vartheta _{0}-\delta }< \varLambda _{\vartheta _{0}}+ \varepsilon . \end{aligned} $$

Then, for any $\vartheta \in (\vartheta _{0}-\delta ,\vartheta _{0}+ \delta )$, we have $\varLambda _{\vartheta _{0}}-\varepsilon < \varLambda _{\vartheta }<\varLambda _{\vartheta _{0}}+\varepsilon$. Therefore,

$$ \begin{aligned} \varLambda _{\vartheta }\text{ is a continuous function of } \vartheta \in (0,\vartheta _{\mathrm{c}}). \end{aligned} $$

(4.40)

Now, we consider the limit of $\varLambda _{\vartheta }$ as $\vartheta \to 0$. For any $\varepsilon >0$, there is ${w}\in \mathcal{A}_{0}$ such that

$$ \begin{gathered} w_{3}\neq 0 \quad \text{on }\varSigma \quad \text{and} \\ \varLambda _{0} -\varepsilon < \sqrt{ g[\!\![\rho ]\!\!]\vert {w}_{3} \vert _{0}^{2} - \varLambda _{0} \Vert \sqrt{\mu } \mathbb{D} {w} \Vert _{0}^{2}/2- \Vert \sqrt{\kappa \rho } \mathbb{D} {w} \Vert _{0}^{2}/2 }=\varLambda _{0}. \end{gathered} $$

(4.41)

Moreover,

$$ \varLambda _{\vartheta }< \varLambda _{0}. $$

(4.42)

Therefore, using (3.6), (4.41), and (4.42), there is a sufficiently small constant $\vartheta _{1}\in (0,\vartheta _{\mathrm{c}})$ such that, for any $\vartheta \in (0,\vartheta _{1})$,

$$ \begin{aligned}[b] \varLambda _{0} -\varepsilon &< \sqrt { g[\!\![\rho ]\!\!]\vert {w}_{3} \vert _{0}^{2} - \varLambda _{\vartheta } \Vert \sqrt{\mu } \mathbb{D} {w} \Vert _{0}^{2}/2- \Vert \sqrt{\kappa \rho } \mathbb{D} {w} \Vert _{0}^{2}/2- \vartheta \vert \nabla _{\mathrm{h}}w_{3} \vert ^{2}_{0}}\\& \leqslant \varLambda _{\vartheta }< \varLambda _{0}.\end{aligned} $$

(4.43)

Hence, we have

$$ \lim_{\vartheta \rightarrow 0}\varLambda _{\vartheta }= \varLambda _{0}, $$

which, together with (4.40), yields that

$$ \begin{aligned} \varLambda _{\vartheta }\text{ is a continuous function of } \vartheta \in [0,\vartheta _{\mathrm{c}}). \end{aligned} $$

(4.44)

(3) The upper bound ( 3.1 ) of $\varLambda _{\vartheta }$.

According to the definition of $\vartheta _{\mathrm{c}}$, we can deduce from (3.10) that

$$ \begin{aligned} g[\!\![\rho ]\!\!]\vert {w}_{3} \vert _{0}^{2}- \Vert \sqrt{\kappa \rho } \mathbb{D} {w} \Vert _{0}^{2}/2\leqslant \vartheta _{\mathrm{c}} \vert \nabla _{\mathrm{h}} {w}_{3} \vert _{0}^{2} \quad \text{for any } {w}\in H ^{1}_{\sigma ,3}. \end{aligned} $$

Thus, by (3.6), for any given $\vartheta \in [0, \vartheta _{\mathrm{c}})$, there is ${w}^{\vartheta }\in \mathcal{A} _{\vartheta }$ such that

$$\begin{aligned} 0 \leqslant {}&\varLambda _{\vartheta }^{2}= \mathcal{F} \bigl(w^{\vartheta }, \varLambda _{\vartheta }\bigr) \\ \leqslant{} &\frac{(\vartheta _{\mathrm{c}}-\vartheta )}{ \vartheta _{\mathrm{c}}}\bigl(g[\!\![\rho ]\!\!]\bigl\vert {w}^{\vartheta }_{3} \bigr\vert _{0}^{2}- \bigl\Vert \sqrt{\kappa \rho }\mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2}/2\bigr)-\frac{\varLambda _{\vartheta }}{2} \bigl\Vert \sqrt{\mu } \mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2}, \end{aligned}$$

which yields that

$$\begin{aligned} \varLambda _{\vartheta }^{2}+ \frac{\varLambda _{\vartheta }}{2} \bigl\Vert \sqrt{ \mu }\mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2} \leqslant \frac{( \vartheta _{\mathrm{c}}-\vartheta )}{\vartheta _{\mathrm{c}}}\bigl(g[\!\![\rho ]\!\!]\bigl\vert {w}^{\vartheta }_{3} \bigr\vert _{0}^{2}- \bigl\Vert \sqrt{\kappa \rho }\mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2}/2\bigr). \end{aligned}$$

(4.45)

Making use of (2.3) and trace estimate (2.6), we can easily estimate that

$$ \begin{aligned} \bigl\vert {w}^{\vartheta }_{3} \bigr\vert _{0}^{2} \leqslant \frac{h_{+}}{8 \mu _{+}} \bigl\Vert \sqrt{ \mu } \mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2} . \end{aligned} $$

Similarly, we also have

$$ \begin{aligned} \bigl\vert {w}^{\vartheta }_{3} \bigr\vert _{0}^{2} \leqslant \frac{ h_{-}}{8 \mu _{-}} \bigl\Vert \sqrt{ \mu } \mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2} . \end{aligned} $$

By the above two estimates, we deduce from (4.45) that

$$ \begin{aligned} \varLambda _{\vartheta } \bigl\Vert \sqrt{\mu } \mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2} \leqslant \frac{ g[\!\![\rho ]\!\!]( \vartheta _{\mathrm{c}}-\vartheta )}{ 4\vartheta _{\mathrm{c}} }\min \biggl\{ \frac{h_{+}}{\mu _{+}},\frac{h_{-}}{\mu _{-}} \biggr\} \bigl\Vert \sqrt{ \mu } \mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2}-\frac{(\vartheta _{ \mathrm{c}}-\vartheta )}{\vartheta _{\mathrm{c}}} \bigl\Vert \sqrt{\kappa \rho } \mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2}, \end{aligned} $$

which yields that

$$ \begin{aligned} \varLambda _{\vartheta }\leqslant \frac{ (\vartheta _{\mathrm{c}}-\vartheta )}{ 4\vartheta _{\mathrm{c}} }\biggl(g[\!\![\rho ]\!\!]\min \biggl\{ \frac{h_{+}}{\mu _{+}},\frac{h_{-}}{\mu _{-}} \biggr\} -4\max \biggl\{ \frac{\kappa _{+}\rho _{+}}{\mu _{+}}, \frac{\kappa _{-}\rho _{-}}{\mu _{-}} \biggr\} \biggr). \end{aligned} $$

(4.46)

Noting that $\Vert \sqrt{\rho }{w}^{\vartheta } \Vert _{0}=1$, then, by (2.13),

$$ \begin{aligned} \bigl\vert {w}^{\vartheta }_{3} \bigr\vert _{0}^{2} \leqslant \frac{2}{\sqrt{\rho _{-}}} \bigl\Vert \sqrt{\rho _{-}} {w}^{\vartheta }_{3} \bigr\Vert _{L^{2}(\varOmega _{-})} \bigl\Vert \partial _{3}{w}^{\vartheta }_{3} \bigr\Vert _{L^{2}(\varOmega _{-})}\leqslant \frac{ \Vert \sqrt{\mu }\mathbb{D}{w}^{\vartheta } \Vert _{0}}{\sqrt{2\rho _{-}\mu _{-}}}. \end{aligned} $$

Putting the above estimate into (4.45), and then applying Young’s inequality, we obtain

$$\begin{aligned} \varLambda _{\vartheta }^{2} \leqslant \frac{(g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}-\vartheta ))^{2} }{4\vartheta _{\mathrm{c}} ^{2} \rho _{-}\mu _{-} \varLambda _{\vartheta }}- \frac{ (\vartheta _{ \mathrm{c}}-\vartheta )}{ 2\vartheta _{\mathrm{c}} } \bigl\Vert \sqrt{\kappa \rho }\mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2}, \end{aligned}$$

which yields that

$$\begin{aligned} \varLambda _{\vartheta }^{3} \leqslant \frac{(g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}-\vartheta ))^{2} }{4\vartheta _{\mathrm{c}} ^{2} \rho _{-}\mu _{-} }. \end{aligned}$$

(4.47)

Similarly, we also have

$$\begin{aligned} \varLambda _{\vartheta }^{3} \leqslant \frac{(g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}-\vartheta ))^{2} }{4\vartheta _{\mathrm{c}} ^{2} \rho _{+}\mu _{+} }. \end{aligned}$$

(4.48)

Summing up the above two estimates, we can immediately get that

$$\begin{aligned} \varLambda _{\vartheta }\leqslant \biggl(\frac{(g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}-\vartheta ))^{2} }{4\vartheta _{\mathrm{c}} ^{2} \max \{\rho _{+}\mu _{+} , \rho _{-}\mu _{-}\}} \biggr)^{\frac{1}{3}}, \end{aligned}$$

(4.49)

which, together with (4.46), yields that

$$\begin{aligned} \varLambda _{\vartheta }\leqslant m. \end{aligned}$$

(4.50)

Hence, we complete the proof of Theorem 3.2 from (4.39), (4.44), and (4.50).

5 Conclusion

In this paper, we investigate the effect of surface tension in the Rayleigh–Taylor (RT) problem of stratified incompressible viscoelastic fluids. We prove that there exists an unstable solution to the linearized stratified RT problem with a largest growth rate Λ under the instability condition (i.e., the surface tension coefficient ϑ is less than a threshold $\vartheta _{c}$). Moreover, for this instability condition, the largest growth rate $\varLambda _{\vartheta }$ decreases from a positive constant to 0, when ϑ increases from 0 to $\vartheta _{c}$, which mathematically verifies that the internal surface tension can constrain the growth of the RT instability during the linear stage.