Boundary Value Problems

Impact Factor 0.642

Open Access

The existence of positive solutions to an elliptic system with nonlinear boundary conditions

Boundary Value Problems20132013:159

DOI: 10.1186/1687-2770-2013-159

Accepted: 16 June 2013

Published: 1 July 2013

Abstract

In this paper, we consider the following system:

$\left\{\begin{array}{c}\mathrm{\Delta }u=u,\phantom{\rule{2em}{0ex}}\mathrm{\Delta }v=v,\phantom{\rule{1em}{0ex}}x\in \mathrm{\Omega },\hfill \\ \frac{\partial u}{\partial \nu }=f\left(x,v\right),\phantom{\rule{2em}{0ex}}\frac{\partial v}{\partial \nu }=g\left(x,u\right),\phantom{\rule{1em}{0ex}}x\in \partial \mathrm{\Omega },\hfill \end{array}$

where Ω is a bounded domain in ${\mathbb{R}}^{N}$ ($N\ge 3$) with smooth boundary, $\frac{\partial }{\partial \nu }$ is the outer normal derivative and $f,g:\partial \mathrm{\Omega }×\mathbb{R}\to {\mathbb{R}}^{+}$ are positive and continuous functions. Under certain assumptions on $f\left(x,t\right)$ and $g\left(x,t\right)$, but without the usual (AR) condition, we prove that the problem has at least one positive strong pair solution $\left(u,v\right)$ (see Definition 1.4 below) by applying a linking theorem for strong indefinite functional.

MSC:35A01, 35J20, 35J25.

Keywords

fractional Sobolev spaces linking theorem nonlinear boundary conditions strong solution

1 Introduction and main result

In this paper, we mainly study the following system:
$\left\{\begin{array}{c}\mathrm{\Delta }u=u,\phantom{\rule{2em}{0ex}}\mathrm{\Delta }v=v,\phantom{\rule{1em}{0ex}}x\in \mathrm{\Omega },\hfill \\ \frac{\partial u}{\partial \nu }=f\left(x,v\right),\phantom{\rule{2em}{0ex}}\frac{\partial v}{\partial \nu }=g\left(x,u\right),\phantom{\rule{1em}{0ex}}x\in \partial \mathrm{\Omega },\hfill \end{array}$
(1.1)

where Ω is a bounded domain in ${\mathbb{R}}^{N}$ ($N\ge 3$) with smooth boundary, $\frac{\partial }{\partial \nu }$ is the outer normal derivative and $f,g:\partial \mathrm{\Omega }×\mathbb{R}\to {\mathbb{R}}^{+}$ are positive and continuous functions.

Existence results for nonlinear elliptic systems have received a lot of interest in recent years (see [112]), particularly when the nonlinear term appears as a source in the equation, complemented with Dirichlet boundary conditions. To our knowledge, about the system with nonlinear boundary conditions, there are not many results. Here we refer to [9, 13, 14].

We are mainly motivated by [12] and [14].

In [12], Li and one of the authors considered
$\left\{\begin{array}{c}-\mathrm{\Delta }u+u=f\left(x,v\right),\phantom{\rule{1em}{0ex}}x\in {\mathbb{R}}^{N},\hfill \\ -\mathrm{\Delta }v+v=g\left(x,u\right),\phantom{\rule{1em}{0ex}}x\in {\mathbb{R}}^{N}.\hfill \end{array}$
(1.2)

Under some given conditions, we proved that (1.2) had at least one positive solution pair $\left(u,v\right)\in {H}^{1}\left({\mathbb{R}}^{N}\right)×{H}^{1}\left({\mathbb{R}}^{N}\right)$.

In [14], Bonder, Pinasco, Rossi studied
$\left\{\begin{array}{c}\mathrm{\Delta }u=u,\phantom{\rule{2em}{0ex}}\mathrm{\Delta }v=v,\phantom{\rule{1em}{0ex}}x\in \mathrm{\Omega },\hfill \\ \frac{\partial u}{\partial \nu }={H}_{v}\left(x,u,v\right),\phantom{\rule{2em}{0ex}}\frac{\partial v}{\partial \nu }={H}_{u}\left(x,u,v\right),\phantom{\rule{1em}{0ex}}x\in \partial \mathrm{\Omega }.\hfill \end{array}$
(1.3)

They assumed that H satisfied the following conditions:

(${\stackrel{˜}{\mathrm{H}}}_{1}$) $|H\left(x,u,v\right)|\le C\left({|u|}^{p+1}+{|v|}^{q+1}+1\right)$.

(${\stackrel{˜}{\mathrm{H}}}_{2}$) The Ambrosetti-Rabinowitz type condition: For R large, if $|\left(u,v\right)|\ge R$,
$\frac{1}{\alpha }\frac{\partial H}{\partial u}\left(x,u,v\right)u+\frac{1}{\beta }\frac{\partial H}{\partial v}\left(x,u,v\right)u\ge H\left(x,u,v\right)>0,$
(1.4)
where $p+1\ge \alpha >p>0$ and $q+1\ge \beta >q>0$ with
$1>\frac{1}{\alpha }+\frac{1}{\beta },$
(1.5)
$max\left\{\frac{p}{\alpha }+\frac{q}{\beta };\frac{q}{q+1}\frac{p+1}{\alpha }+\frac{p}{p+1}\frac{q+1}{\beta }\right\}<1+\frac{1}{N-1},$
(1.6)
$\frac{q}{q+1}\frac{p+1}{\alpha }<1,\phantom{\rule{1em}{0ex}}\text{and}\phantom{\rule{1em}{0ex}}\frac{p}{p+1}\frac{q+1}{\beta }<1.$
(1.7)
When $N\ge 4$, they also assumed
$max\left\{\frac{p}{\alpha }+\frac{q}{\beta };\frac{q}{q+1}\frac{p+1}{\alpha }+\frac{p}{p+1}\frac{q+1}{\beta }\right\}<\frac{N+1}{2\left(N-1\right)}.$
(1.8)

(${\stackrel{˜}{\mathrm{H}}}_{3}$) $|\frac{\partial H}{\partial u}\left(x,u,v\right)|\le \left({|u|}^{p}+{|v|}^{\frac{p\left(q+1\right)}{p+1}}+1\right)$, $|\frac{\partial H}{\partial v}\left(x,u,v\right)|\le \left({|u|}^{\frac{q\left(p+1\right)}{q+1}}+{|v|}^{q}+1\right)$.

(${\stackrel{˜}{\mathrm{H}}}_{4}$) $H\left(x,u,v\right)=H\left(x,-u,-v\right)$.

They obtained infinitely many nontrivial solutions of (1.3) under the assumptions (${\stackrel{˜}{\mathrm{H}}}_{1}$) to (${\stackrel{˜}{\mathrm{H}}}_{4}$) by using variational arguments and a fountain theorem. Note that (1.4) implies
$|H\left(x,u,v\right)|\ge c\left({|u|}^{\alpha }+{|v|}^{\beta }\right)-C$
(1.9)

(see Lemma 1.1 in [6]). Therefore it is not difficult to verify that any (PS) sequence (or (C) c sequence) of the corresponding functional is bounded in some suitable space.

The crucial part in the nonlinear boundary conditions case is to find the proper functional setting for (1.1) that allows us to treat our problem variationally. We accomplish this by defining a self-adjoint operator that takes into account the boundary conditions together with the equations and considering its fractional powers that satisfy a suitable ‘integration by parts’ formula. In order to obtain nontrivial solutions, we use a linking theorem (see [11]).

The assumptions we impose on $f\left(x,t\right)$ and $g\left(x,t\right)$ are as follows:

(H1) $f,g\in {C}^{0}\left(\partial \mathrm{\Omega }×{\mathbb{R}}^{1},{\mathbb{R}}^{1}\right)$ with $f\left(x,t\right)=g\left(x,t\right)=0$ for any $\left(x,t\right)\in \partial \mathrm{\Omega }×\left(-\mathrm{\infty },0\right]$,

(H2) ${lim}_{t\to 0}\frac{f\left(x,t\right)}{t}={lim}_{t\to 0}\frac{g\left(x,t\right)}{t}=0$ uniformly in $x\in \partial \mathrm{\Omega }$.

(H3) There is a positive constant $C>0$ such that
$|f\left(x,t\right)|\le C\left(1+{|t|}^{p-1}\right),\phantom{\rule{2em}{0ex}}|g\left(x,t\right)|\le C\left(1+{|t|}^{q-1}\right),\phantom{\rule{1em}{0ex}}\mathrm{\forall }\left(x,t\right)\in \partial \mathrm{\Omega }×{\mathbb{R}}^{1},$
(1.10)
where $p,q>2$ and satisfy
$\frac{1}{p}+\frac{1}{q}>1-\frac{1}{N-1}.$
(1.11)

(H4) ${lim}_{|t|\to +\mathrm{\infty }}\frac{F\left(x,t\right)}{{t}^{2}}={lim}_{|t|\to +\mathrm{\infty }}\frac{G\left(x,t\right)}{{t}^{2}}=+\mathrm{\infty }$ uniformly in $x\in \partial \mathrm{\Omega }$, where $F\left(x,t\right):={\int }_{0}^{t}f\left(x,s\right)\phantom{\rule{0.2em}{0ex}}ds$, $G\left(x,t\right):={\int }_{0}^{t}g\left(x,s\right)\phantom{\rule{0.2em}{0ex}}ds$.

(H5) For all $0, $x\in \partial \mathrm{\Omega }$ or $s, $x\in \partial \mathrm{\Omega }$, there are two positive constants ${C}_{1,\ast }$, ${C}_{2,\ast }$ such that
${H}_{1}\left(x,t\right)\le {H}_{1}\left(x,s\right)+{C}_{1,\ast },\phantom{\rule{2em}{0ex}}{H}_{2}\left(x,t\right)\le {H}_{2}\left(x,s\right)+{C}_{2,\ast },$
(1.12)

where ${H}_{1}\left(x,t\right)=tf\left(x,t\right)-2F\left(x,t\right)$, ${H}_{2}\left(x,t\right)=tg\left(x,t\right)-2G\left(x,t\right)$ with ${H}_{1}\left(x,t\right),{H}_{2}\left(x,t\right)>0$ for any $t>0$, $x\in \partial \mathrm{\Omega }$.

Remark 1.1 By (1.11), there exist l and m with $l+m=1$, $l,m>\frac{1}{4}$ such that
$\frac{1}{p}>\frac{1}{2}-\frac{2l-\frac{1}{2}}{N-1},\phantom{\rule{2em}{0ex}}\frac{1}{q}>\frac{1}{2}-\frac{2m-\frac{1}{2}}{N-1}.$
(1.13)

Remark 1.2 (H5) was first introduced by Miyagaki and Souto in [15]. A typical pair of functions $f\left(x,t\right)={t}^{p-1}$, $g\left(x,t\right)={t}^{q-1}$, $t>0$, $p,q>2$; $f\left(x,t\right)=g\left(x,t\right)=0$, $t\le 0$ satisfy (H1) to (H5). However, the pair of functions $f\left(x,t\right)=t\left(2lnt+1\right)$, $g\left(x,t\right)=t\left(2lnt+1\right)$, $t>0$; $f\left(x,t\right)=g\left(x,t\right)=0$, $t\le 0$ satisfy (H5) but do not satisfy the usual (AR) condition and (${\stackrel{˜}{\mathrm{H}}}_{2}$) in this paper.

Remark 1.3 The assumptions we impose on f and g are different from the assumptions in [14]. To our best knowledge, it is the first time the group assumptions have been used to deal with a system with nonlinear boundary conditions.

In order to state our main result, first we give a definition.

Definition 1.4 We say that $\left(u,v\right)$ is a strong solution of (1.1) if
$u\in {W}^{2,\frac{p}{p-1}}\left(\mathrm{\Omega }\right),\phantom{\rule{2em}{0ex}}v\in {W}^{2,\frac{q}{q-1}}\left(\mathrm{\Omega }\right)$

and $\left(u,v\right)$ satisfies (1.1) a.e. in Ω.

Our main results is as follows.

Theorem 1.5 Let (H1)-(H5) hold. Then system (1.1) possesses at least one positive strong solution pair $z=\left(u,v\right)$.

The main difficulties to deal with system (1.1) consist in at least three aspects. Firstly, due to the type of growth of the functions f and g, we cannot work with the usual ${H}^{1}\left(\mathrm{\Omega }\right)$, and then we need fractional Sobolev spaces. Secondly, although we have a variational problem, the functional associated to it always has a strong indefinite quadratic part. So, the functional possesses no mountain-pass structure but the linking geometric structure, which is more complicated to handle. Thirdly, as we do not assume that the functions f and g satisfy the (AR) conditions, it is much more difficult to show that any (C) c sequence is uniformly bounded in E (see Section 2).

To prove Theorem 1.5, we try to find a critical point of the functional Φ (see (2.5)) in E. We prove that Φ has a linking geometric structure and use a linking theorem under (C) c condition (see Theorem 2.1 in [11]) to get a (C) c -sequence $\left\{{z}_{n}\right\}\subset E$ of Φ. The main difficulty now will be to prove that $\left\{{z}_{n}\right\}$ is uniformly bounded in E without the (AR) condition. Then we prove that any (C) c -sequence $\left\{{z}_{n}\right\}\subset E$ of Φ is bounded. To overcome this difficulty, we use some techniques used in [12, 16] for which the assumptions (H4), (H5) play important roles. As $\left\{{z}_{n}\right\}$ is bounded, then we can prove that $\left\{{z}_{n}\right\}$ has a subsequence which converges to a nontrivial critical point of Φ. Hence, by the strong maximum principle, we can prove that the pair solution $\left(u,v\right)$ is positive.

The paper is organized as follows. In Section 2, we give some preliminaries. We prove our main result in Section 3.

2 Some preliminaries

In this section we mainly give some preliminaries which will be used in Section 3. We follow the structure in [13].

Throughout this paper, we consider the space ${L}^{2}\left(\mathrm{\Omega }\right)×{L}^{2}\left(\partial \mathrm{\Omega }\right)$ which is a Hilbert space with the inner product, which we denote by $〈\cdot ,\cdot 〉$, given by
Now we let $A:D\left(A\right)\subset {L}^{2}\left(\mathrm{\Omega }\right)×{L}^{2}\left(\partial \mathrm{\Omega }\right)\to {L}^{2}\left(\mathrm{\Omega }\right)×{L}^{2}\left(\partial \mathrm{\Omega }\right)$ be the operator defined by
$A\left(u,u{|}_{\partial \mathrm{\Omega }}\right)=\left(-\mathrm{\Delta }u+u,\frac{\partial u}{\partial \nu }\right),$
where $D\left(A\right)=\left\{\left(u,u{|}_{\partial \mathrm{\Omega }}\right):u\in {H}^{2}\left(\mathrm{\Omega }\right)\right\}$. It is not difficult to verify that $D\left(A\right)$ is dense in ${L}^{2}\left(\mathrm{\Omega }\right)×{L}^{2}\left(\partial \mathrm{\Omega }\right)$. Note that A is invertible with its inverse given by
${A}^{-1}\left({h}_{1},{h}_{2}\right)=\left(u,u{|}_{\partial \mathrm{\Omega }}\right),$
where u is the solution of
(2.1)
By standard regularity (see [17]), it follows that ${A}^{-1}$ is bounded and compact. Hence, $R\left(A\right)={L}^{2}\left(\mathrm{\Omega }\right)×{L}^{2}\left(\partial \mathrm{\Omega }\right)$. Therefore, in order to see that A (hence ${A}^{-1}$) is self-adjoint, it suffices to prove that A is symmetric ([18], p.512). In fact, for $u,v\in D\left(A\right)$, applying Green’s formula, we obtain
$〈Au,v〉={\int }_{\mathrm{\Omega }}\left(-\mathrm{\Delta }u+u\right)v\phantom{\rule{0.2em}{0ex}}dx+{\int }_{\partial \mathrm{\Omega }}\frac{\partial u}{\partial \nu }v\phantom{\rule{0.2em}{0ex}}d\sigma ={\int }_{\mathrm{\Omega }}\left(-\mathrm{\Delta }v+v\right)u\phantom{\rule{0.2em}{0ex}}dx+{\int }_{\partial \mathrm{\Omega }}\frac{\partial v}{\partial \nu }u\phantom{\rule{0.2em}{0ex}}d\sigma =〈v,Au〉.$
Hence A is symmetric. Also we can check that A (and so ${A}^{-1}$) is positive. For any $u\in D\left(A\right)$ and by Green’s formula again, we have
$〈Au,u〉={\int }_{\mathrm{\Omega }}\left(-\mathrm{\Delta }u+u\right)u\phantom{\rule{0.2em}{0ex}}dx+{\int }_{\partial \mathrm{\Omega }}\frac{\partial u}{\partial \nu }u\phantom{\rule{0.2em}{0ex}}d\sigma ={\int }_{\mathrm{\Omega }}\left({|\mathrm{\nabla }u|}^{2}+{u}^{2}\right)\phantom{\rule{0.2em}{0ex}}dx\ge 0.$
Hence there is a sequence of eigenvalues $\left({\lambda }_{n}\right)\in \mathbb{R}$ with eigenfunctions $\left({\phi }_{n},{\psi }_{n}\right)\in {L}^{2}\left(\mathrm{\Omega }\right)×{L}^{2}\left(\partial \mathrm{\Omega }\right)$ satisfying $0<{\lambda }_{1}\le {\lambda }_{2}\le \cdots \le {\lambda }_{n}\le \cdots ↗$ and ${\phi }_{n}\in {H}^{2}\left(\mathrm{\Omega }\right)$, ${\phi }_{n}{|}_{\partial \mathrm{\Omega }}={\psi }_{n}$,
(2.2)
Now we consider the following fractional powers of A, i.e., for $0,
where $u={\sum }_{n=1}^{\mathrm{\infty }}{a}_{n}\left({\phi }_{n},{\psi }_{n}\right)$. Let ${E}^{l}=D\left({A}^{l}\right)$, which is a Hilbert space under the inner product
${\left(u,\phi \right)}_{{E}^{l}}=〈{A}^{l}u,{A}^{l}\phi 〉.$
Note that ${E}^{l}\subset {H}^{2l}\left(\mathrm{\Omega }\right)$. Indeed, if we define ${A}_{1}:{H}^{2}\left(\mathrm{\Omega }\right)\subset {L}^{2}\left(\mathrm{\Omega }\right)\to {L}^{2}\left(\mathrm{\Omega }\right)$ by
${A}_{1}u=-\mathrm{\Delta }u+u,$
and ${A}_{2}:{H}^{2}\left(\mathrm{\Omega }\right)\subset D\left({A}_{2}\right)\subset {L}^{2}\left(\partial \mathrm{\Omega }\right)\to {L}^{2}\left(\partial \mathrm{\Omega }\right)$ by
${A}_{2}u=\frac{\partial u}{\partial \nu },$
then $\overline{A}=\left({A}_{1},{A}_{2}\right)$ satisfies
$A=\overline{A}{|}_{\left(u,u\right)},\phantom{\rule{1em}{0ex}}u\in D\left({A}_{1}\right)\cap D\left({A}_{2}\right),$
and hence
${A}^{l}={\overline{A}}^{l}{|}_{\left(u,u\right)},\phantom{\rule{1em}{0ex}}u\in D\left({A}_{1}^{l}\right)\cap D\left({A}_{2}^{l}\right).$
Since $D\left({A}_{1}\right)={H}^{2}\left(\mathrm{\Omega }\right)\subset D\left({A}_{2}\right)$, we have $D\left({A}_{1}^{l}\right)\subset D\left({A}_{2}^{l}\right)$. Therefore,
${E}^{l}=D\left({A}^{l}\right)=D\left({A}_{1}^{l}\right).$
Noting that Ω is smooth, it follows from the results of p.187 in [19] (see also [18, 20]) that
${E}^{l}=D\left({A}_{1}^{l}\right)\subset {H}^{2l}\left(\mathrm{\Omega }\right).$

The following compact result will be useful later.

Proposition 2.1 (Theorem 2.1, [14])

Given $l>\frac{1}{4}$ and $r\ge 1$ so that $\frac{1}{r}\ge \frac{1}{2}-\frac{2l-\frac{1}{2}}{N-1}$, the inclusion map $i:{E}^{l}\to {L}^{r}\left(\partial \mathrm{\Omega }\right)$ is well defined and bounded. Moreover, if $\frac{1}{r}>\frac{1}{2}-\frac{2l-\frac{1}{2}}{N-1}$, then the inclusion is compact.

Denote $E={E}^{l}×{E}^{m}$, where $l+m=1$, l, m are the same as in Remark 1.1 and define $B:E×E\to \mathbb{R}$ by
$B\left(\left(u,v\right),\left(\phi ,\psi \right)\right)=〈{A}^{l}u,{A}^{m}\psi 〉+〈{A}^{l}\phi ,{A}^{m}v〉.$
Associated to B, we have the quadratic form
$Q\left(z\right):=\frac{1}{2}B\left(z,z\right)=〈{A}^{l}u,{A}^{m}v〉.$
It is easy to see (one can refer to [14]) that the bounded self-adjoint operator $L:E\to E$ defined by $\left(Lz,\eta \right):=B\left(z,\eta \right)$ has exactly two eigenvalues +1 and −1, and that the corresponding eigenvalues ${E}^{+}$ and ${E}^{-}$ are given by
${E}^{+}=\left\{\left(u,{A}^{-m}{A}^{l}u\right):u\in {E}^{l}\right\}\phantom{\rule{1em}{0ex}}\text{and}\phantom{\rule{1em}{0ex}}{E}^{-}=\left\{\left(u,-{A}^{-m}{A}^{l}u\right):u\in {E}^{l}\right\},$
where we use the notation ${A}^{-m}={\left({A}^{m}\right)}^{-1}$. Then $E={E}^{+}\oplus {E}^{-}$. The spaces ${E}^{+}$ and ${E}^{-}$ are orthogonal with respect to the bilinear B, that is,
$B\left({z}^{+},{z}^{-}\right)=0,\phantom{\rule{1em}{0ex}}\mathrm{\forall }{z}^{+}\in {E}^{+},{z}^{-}\in {E}^{-}.$
Moreover, we have
$\frac{1}{2}{\parallel {z}^{±}\parallel }_{E}^{2}=Q\left({z}^{±}\right)={\int }_{\mathrm{\Omega }}{|{A}^{l}u|}^{2}\phantom{\rule{0.2em}{0ex}}dx,$
if ${z}^{±}=\left(u,±\left({A}^{-m}{A}^{l}u\right)\right)$. We see also that for $z=\left(u,v\right)$, $z={z}^{+}+{z}^{-}$ with ${z}^{+}=\left(\frac{\left(u+{A}^{-l}{A}^{m}v\right)}{2},\frac{\left({A}^{-m}{A}^{l}u+v\right)}{2}\right)$, ${z}^{-}=\left(\frac{\left(u-{A}^{-l}{A}^{m}uv\right)}{2},\frac{\left(-{A}^{-m}{A}^{l}u+v\right)}{2}\right)$ and
$〈{A}^{l}u,{A}^{m}v〉=\frac{1}{2}B\left(z,z\right)=\frac{1}{2}〈Lz,z〉=\frac{1}{2}\left({\parallel {z}^{+}\parallel }_{E}^{2}-{\parallel {z}^{-}\parallel }_{E}^{2}\right).$
(2.3)
From (1.10), Remark 1.1 and Proposition 2.1, we can define the functional $\mathcal{H}:E\to \mathbb{R}$ as
$\mathcal{H}\left(u,v\right)={\int }_{\partial \mathrm{\Omega }}F\left(x,v\right)\phantom{\rule{0.2em}{0ex}}d\sigma +{\int }_{\partial \mathrm{\Omega }}G\left(x,u\right)\phantom{\rule{0.2em}{0ex}}d\sigma .$
(2.4)
Lemma 2.2 The functional defined by (2.4) is of class ${C}^{1}$ and its derivative is given by
${\mathcal{H}}^{\prime }\left(u,v\right)\left(\phi ,\psi \right)={\int }_{\partial \mathrm{\Omega }}f\left(x,v\right)\psi \phantom{\rule{0.2em}{0ex}}d\sigma +{\int }_{\partial \mathrm{\Omega }}g\left(x,u\right)\phi \phantom{\rule{0.2em}{0ex}}d\sigma .$

Moreover, ${\mathcal{H}}^{\prime }$ is compact.

Proof From (1.10), Hölder’s inequality and Proposition 2.1, we have
$\begin{array}{rcl}|{\int }_{\partial \mathrm{\Omega }}f\left(x,v\right)\psi \phantom{\rule{0.2em}{0ex}}d\sigma |& \le & {\int }_{\partial \mathrm{\Omega }}|f\left(x,v\right)\psi |\phantom{\rule{0.2em}{0ex}}d\sigma \\ \le & {\int }_{\partial \mathrm{\Omega }}\left(1+{|v|}^{p-1}\right)|\psi |\phantom{\rule{0.2em}{0ex}}d\sigma \\ \le & {\parallel v\parallel }_{{L}^{p}\left(\partial \mathrm{\Omega }\right)}^{p-1}{\parallel \psi \parallel }_{{L}^{p}\left(\partial \mathrm{\Omega }\right)}+{\parallel \psi \parallel }_{{L}^{p}\left(\partial \mathrm{\Omega }\right)}{|\partial \mathrm{\Omega }|}^{\frac{p-1}{p}}\\ \le & C\left({\parallel v\parallel }_{{E}^{m}}^{p-1}+1\right){\parallel \psi \parallel }_{{E}^{m}}.\end{array}$
Similarly, we have
$|{\int }_{\partial \mathrm{\Omega }}g\left(x,u\right)\phi \phantom{\rule{0.2em}{0ex}}d\sigma |\le C\left({\parallel u\parallel }_{{E}^{l}}^{q-1}+1\right){\parallel \phi \parallel }_{{E}^{l}}.$

Hence ${\mathcal{H}}^{\prime }$ is well defined and bounded in E. A standard argument yields that is Fréchet differentiable with ${\mathcal{H}}^{\prime }$ continuous. By Proposition 2.1 we know that ${\mathcal{H}}^{\prime }$ is compact (see [21] for the details). □

Now we define the functional $\mathrm{\Phi }:E\to \mathbb{R}$ for (1.1) given by
$\mathrm{\Phi }\left(z\right)=Q\left(z\right)-\mathcal{H}\left(z\right).$
(2.5)

Moreover, Φ is class ${C}^{1}\left(E,\mathbb{R}\right)$.

Definition 2.3 We say that $z=\left(u,v\right)\in E={E}^{l}×{E}^{m}$ is an $\left(l,m\right)$-weak solution of (1.1) if z is a critical point of Φ. In other words, for every $\left(\phi ,\psi \right)\in E$, we have
$〈{A}^{l}u,{A}^{m}\psi 〉+〈{A}^{l}\phi ,{A}^{m}v〉-{\int }_{\partial \mathrm{\Omega }}f\left(x,v\right)\psi \phantom{\rule{0.2em}{0ex}}d\sigma -{\int }_{\partial \mathrm{\Omega }}g\left(x,u\right)\phi \phantom{\rule{0.2em}{0ex}}d\sigma =0.$
(2.6)

Now we give a regularity result of an $\left(l,m\right)$ weak solution.

Proposition 2.4 If $\left(u,v\right)\in E$ is an $\left(l,m\right)$-weak solution of (1.1), then $u\in {W}^{2,\frac{p}{p-1}}\left(\mathrm{\Omega }\right)$, $v\in {W}^{2,\frac{q}{q-1}}\left(\mathrm{\Omega }\right)$ and
$\mathrm{\Delta }u=u\phantom{\rule{1em}{0ex}}\mathit{\text{in}}\phantom{\rule{0.1em}{0ex}}\mathrm{\Omega },\phantom{\rule{2em}{0ex}}\frac{\partial u}{\partial \nu }=f\left(x,v\right)\phantom{\rule{1em}{0ex}}\mathit{\text{on}}\phantom{\rule{0.1em}{0ex}}\partial \mathrm{\Omega },$
(2.7)
$\mathrm{\Delta }v=v\phantom{\rule{1em}{0ex}}\mathit{\text{in}}\phantom{\rule{0.1em}{0ex}}\mathrm{\Omega },\phantom{\rule{2em}{0ex}}\frac{\partial v}{\partial \nu }=g\left(x,u\right)\phantom{\rule{1em}{0ex}}\mathit{\text{on}}\phantom{\rule{0.1em}{0ex}}\partial \mathrm{\Omega }.$
(2.8)

In other words, $\left(u,v\right)$ is a strong solution of (1.1).

Proof Although the proof is only needed to make some minor modifications as that of Theorem 2.2 in [13], for the readers’ convenience, we give its detailed proof.

Let us consider $\phi =0$ in (2.6), then
$〈{A}^{l}u,{A}^{m}\psi 〉-{\int }_{\partial \mathrm{\Omega }}f\left(x,v\right)\psi =0$
(2.9)

for all $\psi \in {E}^{m}$.

If we take $\psi \in {H}^{2}\left(\mathrm{\Omega }\right)$, then we have
$〈{A}^{l}u,{A}^{m}\psi 〉=〈u,A\psi 〉={\int }_{\mathrm{\Omega }}\left(-\mathrm{\Delta }\psi +\psi \right)u\phantom{\rule{0.2em}{0ex}}dx+{\int }_{\partial \mathrm{\Omega }}\frac{\partial \psi }{\partial \nu }u\phantom{\rule{0.2em}{0ex}}d\sigma .$
(2.10)
On the other hand, by (1.10) and Proposition 2.1, we have
$\begin{array}{rl}{\int }_{\partial \mathrm{\Omega }}{|f\left(x,v\right)|}^{\frac{p}{p-1}}\phantom{\rule{0.2em}{0ex}}d\sigma & \le {\int }_{\partial \mathrm{\Omega }}{\left(1+{|v|}^{p-1}\right)}^{\frac{p}{p-1}}\phantom{\rule{0.2em}{0ex}}d\sigma \le C{\int }_{\partial \mathrm{\Omega }}\left(1+{|v|}^{p}\right)\phantom{\rule{0.2em}{0ex}}d\sigma \\ \le C+C{\parallel v\parallel }_{{E}^{m}}^{p}<+\mathrm{\infty },\end{array}$
(2.11)
i.e., $f\left(x,v\right)\in {L}^{\frac{p}{p-1}}\left(\partial \mathrm{\Omega }\right)$. Then from basic elliptic theory (Theorem 9.9, p.9, [17]) there exists one function $w\in {W}^{2,\frac{p}{p-1}}\left(\mathrm{\Omega }\right)$ such that
Then we get
$0={\int }_{\mathrm{\Omega }}\left(-\mathrm{\Delta }w+w\right)\psi \phantom{\rule{0.2em}{0ex}}dx={\int }_{\mathrm{\Omega }}\left(-\mathrm{\Delta }\psi +\psi \right)w\phantom{\rule{0.2em}{0ex}}dx+{\int }_{\partial \mathrm{\Omega }}w\frac{\partial \psi }{\partial \nu }\phantom{\rule{0.2em}{0ex}}d\sigma -{\int }_{\partial \mathrm{\Omega }}f\left(x,v\right)\psi \phantom{\rule{0.2em}{0ex}}d\sigma .$
(2.12)
From (2.10), (2.11) and (2.12), we have
$〈u-w,A\psi 〉={\int }_{\mathrm{\Omega }}\left(u-w\right)\left(-\mathrm{\Delta }\psi +\psi \right)\phantom{\rule{0.2em}{0ex}}dx+{\int }_{\partial \mathrm{\Omega }}\left(u-w\right)\frac{\partial \psi }{\partial \nu }=0,$

which implies that $u=w$. We have gotten that $u\in {W}^{2,\frac{p}{p-1}}\left(\mathrm{\Omega }\right)$. Finally, since $u=w$, we conclude that u satisfies (2.7). We can make the same argument for v. □

3 The proof of our main result

In this section, we mainly want to prove Theorem 1.5. First we present a linking theorem from [11]. Then we prove that it can be applied to our functional setting stated in Section 2.

Suppose that $f\left(x,t\right)$, $g\left(x,t\right)$ satisfy the assumptions (H1)-(H3), then it is easy to see that for any $ϵ>0$ there is a ${C}_{ϵ}>0$ such that for $\left(x,t\right)\in \partial \mathrm{\Omega }×{\mathbb{R}}^{1}$ we have
$|f\left(x,t\right)|\le ϵ|t|+{C}_{ϵ}{|t|}^{p-1},\phantom{\rule{2em}{0ex}}|g\left(x,t\right)|\le ϵ|t|+{C}_{ϵ}{|t|}^{q-1}$
(3.1)
and
$|F\left(x,t\right)|\le ϵ{|t|}^{2}+{C}_{ϵ}{|t|}^{p},\phantom{\rule{2em}{0ex}}|G\left(x,t\right)|\le ϵ{|t|}^{2}+{C}_{ϵ}{|t|}^{q}.$
(3.2)

Since $f\left(x,t\right)=g\left(x,t\right)=0$ when $t\le 0$, $\left(u,v\right)=\left(0,0\right)\in E$ is a solution of (1.1). So we are interested in nontrivial and nonnegative solutions of (1.1).

Recall that $\left\{{z}_{n}\right\}\subset E$ is called a Palais-Smale sequence of a ${C}^{1}$ functional I on E at level c ((PS) c -sequence for short) if $I\left({z}_{n}\right)\to c$ and ${I}^{\prime }\left({z}_{n}\right)\to 0$ in ${E}^{\ast }$ as $n\to \mathrm{\infty }$. If $I\left({z}_{n}\right)\to c$ and $\left(1+{\parallel {z}_{n}\parallel }_{E}\right){I}^{\prime }\left({z}_{n}\right)\to 0$ in ${E}^{\ast }$ as $n\to \mathrm{\infty }$, then $\left\{{z}_{n}\right\}$ will be called a Cerami sequence at level c ((C) c -sequence for short). A standard way to prove the existence of a positive solution to (1.1) is to get a (PS) c or (C) c sequence for Φ and then to prove that the sequence converges to a solution to (1.1). In this paper, we want to get a (C) c sequence by a linking theorem (Theorem 2.1, in [11]). So, we need to recall some terminology (see, e.g., [11, 22]).

Let ${H}^{-}$ be a closed separable subspace of a Hilbert space H with the norm ${\parallel \cdot \parallel }_{H}$ and let ${H}^{+}:={\left({H}^{-}\right)}^{\mathrm{\perp }}$. For $u\in H$, we shall write $u={u}^{+}+{u}^{-}$, where ${u}^{±}\in {H}^{±}$. On H we define a new norm
${\parallel u\parallel }_{\tau }:=max\left\{{\parallel {u}^{+}\parallel }_{H},\sum _{k=1}^{\mathrm{\infty }}\frac{1}{{2}^{k}}|〈{u}^{-},{e}^{k}〉|\right\},$
where $\left\{{e}_{k}\right\}$ is a total orthonormal sequence in ${H}^{-}$. The topology generated by ${\parallel \cdot \parallel }_{\tau }$ will be called the τ-topology. Recall from [22] that a homotopy $h=I-g:A×\left[0,1\right]\to H$, where $A\subset H$, is called admissible if:
1. (i)

h is τ-continuous, i.e., $h\left({u}_{n},{s}_{n}\right)\to h\left(u,s\right)$ in τ-topology as $n\to \mathrm{\infty }$ whenever ${u}_{n}\to u$ in τ-topology and ${s}_{n}\to s$ as $n\to \mathrm{\infty }$.

2. (ii)

g is τ-locally finite-dimensional, i.e., for each $\left(u,s\right)\in A×\left[0,1\right]$, there is a neighborhood U of $\left(u,s\right)$ in the product topology of $\left(E,\tau \right)$ and $\left[0,1\right]$ such that $g\left(U\cap \left(A×\left[0,1\right]\right)\right)$ is contained in a finite-dimensional subspace of H.

Admissible maps are defined similarly. Recall also that admissible maps and homotopies are necessarily continuous, and on bounded subsets of H the τ-topology coincides with the product topology of ${H}_{\mathrm{weak}}^{-}$ and ${H}_{\mathrm{strong}}^{+}$.

Let $\mathrm{\Phi }\in {C}^{1}\left(H,{\mathbb{R}}^{1}\right)$, $R>r>0$ and ${z}_{0}\in {H}^{+}\setminus \left\{0\right\}$ and define
$M=\left\{z={z}^{-}+t{z}_{0}:\parallel z\parallel \le R,t\ge 0\right\},\phantom{\rule{2em}{0ex}}N=\left\{z\in {H}^{+}:\parallel z\parallel =r\right\}$
and

Proposition 3.1 (Theorem 2.1, [11])

Let $H={H}^{+}\oplus {H}^{-}$ be a separable Hilbert space with ${H}^{-}$ orthogonal to ${H}^{+}$. Suppose that
1. (i)

$\mathrm{\Phi }\left(z\right)=\frac{1}{2}\left(\parallel {z}^{+}\parallel -{\parallel {z}^{-}\parallel }^{2}\right)-\mathrm{\Psi }\left(z\right)$, where $\mathrm{\Psi }\in {C}^{1}\left(H,{\mathbb{R}}^{1}\right)$ is bounded below, weakly sequentially lower semi-continuous and ${\mathrm{\Psi }}^{\prime }$ is weakly sequentially continuous.

2. (ii)

There exist ${z}_{0}\in {H}^{+}\mathrm{\setminus }\left\{0\right\}$, $\alpha >0$, and $R>r>0$ such that $\mathrm{\Phi }{|}_{N}\ge \alpha$ and $\mathrm{\Phi }{|}_{\partial M}\le 0$.

Then there exists a (C) c -sequence for Φ, where
$c:=\underset{h\in \mathrm{\Gamma }}{inf}\underset{u\in M}{sup}\mathrm{\Phi }\left(h\left(u,1\right)\right).$

Moreover, $c\ge \alpha$.

For fixed ${z}_{0}\in {E}^{+}\mathrm{\setminus }\left\{0\right\}$ and $R>r>0$, let
${M}_{R}=\left\{z={z}^{-}+\rho {z}_{0}:{z}^{-}\in {E}^{-},{\parallel z\parallel }_{E}\le R,\rho \ge 0\right\},\phantom{\rule{2em}{0ex}}{N}_{r}=\left\{z\in {E}^{+}:{\parallel z\parallel }_{E}=r\right\}.$

Lemma 3.2 There exist $r>0$ and $\alpha >0$ such that $\mathrm{\Phi }{|}_{{N}_{r}}\ge \alpha$.

Proof For any $z\in {N}_{r}$, $z=\left(u,v\right)\in {E}^{+}$, we know that $v={A}^{-m}{A}^{l}u$ or, equivalently, $u={A}^{-l}{A}^{m}v$. By (3.2) and Proposition 2.1, we have
$\begin{array}{r}|{\int }_{\partial \mathrm{\Omega }}F\left(x,v\right)\phantom{\rule{0.2em}{0ex}}d\sigma +{\int }_{\partial \mathrm{\Omega }}G\left(x,u\right)\phantom{\rule{0.2em}{0ex}}d\sigma |\\ \phantom{\rule{1em}{0ex}}\le {\int }_{\partial \mathrm{\Omega }}\left(|F\left(x,v\right)|+|G\left(x,u\right)|\right)\phantom{\rule{0.2em}{0ex}}d\sigma \\ \phantom{\rule{1em}{0ex}}\le {\int }_{\partial \mathrm{\Omega }}\left[\left(ϵ{|v|}^{2}+{C}_{ϵ}{|v|}^{p}\right)+\left(ϵ{|u|}^{2}+{C}_{ϵ}|{u}^{q}\right)\right]\phantom{\rule{0.2em}{0ex}}d\sigma \\ \phantom{\rule{1em}{0ex}}\le ϵ{\parallel u\parallel }_{{E}^{l}}^{2}+{C}_{ϵ}{\parallel u\parallel }_{{E}^{l}}^{p}+ϵ{\parallel v\parallel }_{{E}^{m}}^{2}+{C}_{ϵ}{\parallel v\parallel }_{{E}^{m}}^{q}\\ \phantom{\rule{1em}{0ex}}\le ϵ{\parallel z\parallel }_{E}^{2}+{C}_{ϵ}{\parallel z\parallel }_{E}^{p}+ϵ{\parallel z\parallel }_{E}^{2}+{C}_{ϵ}{\parallel z\parallel }_{E}^{q}.\end{array}$
Since $p,q>2$, we have if ${\parallel z\parallel }_{E}=r$ is small enough,
$\begin{array}{rcl}\mathrm{\Phi }\left(z\right)& =& {\int }_{\mathrm{\Omega }}{A}^{l}u{A}^{m}v\phantom{\rule{0.2em}{0ex}}dx-{\int }_{\partial \mathrm{\Omega }}F\left(x,v\right)\phantom{\rule{0.2em}{0ex}}d\sigma -{\int }_{\partial \mathrm{\Omega }}G\left(x,u\right)\phantom{\rule{0.2em}{0ex}}d\sigma \\ \ge & \frac{1}{2}{\parallel z\parallel }_{E}^{2}-ϵ{\parallel z\parallel }_{E}^{2}-{C}_{ϵ}{\parallel z\parallel }_{E}^{p}-{C}_{ϵ}{\parallel z\parallel }_{E}^{q}\ge \alpha >0\end{array}$

for some $\alpha >0$. □

Lemma 3.3 For the r given by Lemma 3.2 and any ${z}_{0}=\left({u}_{0},{v}_{0}\right)\in {E}^{+}\mathrm{\setminus }\left\{0\right\}$ with ${\parallel {z}_{0}\parallel }_{E}=1$, there exists $R>r$ such that $\mathrm{\Phi }{|}_{\partial {M}_{R}}\le 0$, where ${M}_{R}=\left\{z={z}^{-}+\rho {z}_{0}:{\parallel z\parallel }_{E}\le R,\rho \ge 0\right\}$.

Proof If $z\in \partial {M}_{R}$, then $z={z}^{-}+\rho {z}_{0}$ with either $:{\parallel z\parallel }_{E}=R$, $\rho \ge 0$ or $:{\parallel z\parallel }_{E}, $\rho =0$.
1. (i)
If $\rho =0$, then we have $z\in {E}^{-}$, $z=\left(u,-{A}^{-m}{A}^{l}u\right)$ and
$\mathrm{\Phi }\left(z\right)=-\frac{1}{2}{\parallel {z}^{-}\parallel }_{E}^{2}-{\int }_{\partial \mathrm{\Omega }}F\left(x,v\right)-{\int }_{\partial \mathrm{\Omega }}G\left(x,u\right)\le 0,$

since $F\left(x,t\right),G\left(x,t\right)\ge 0$ for any $\left(x,t\right)\in \partial \mathrm{\Omega }×{\mathbb{R}}^{1}$.
1. (ii)
Assume that $\rho >0$. We argue by contradiction. Suppose that there exists a sequence $\left\{{z}_{n}\right\}\in \partial {M}_{n}$, ${z}_{n}={\rho }_{n}{z}_{0}+{z}_{n}^{-}$, ${\rho }_{n}>0$, ${\parallel {z}_{0}\parallel }_{E}=1$, ${\parallel {z}_{n}\parallel }_{E}=n$ such that $\mathrm{\Phi }\left({z}_{n}\right)>0$. If ${z}_{n}=\left({u}_{n},{v}_{n}\right):=\left({\rho }_{n}{u}_{0}+{\phi }_{n},{\rho }_{n}{v}_{0}+{\psi }_{n}\right)$, then by the definitions of ${E}^{+}$ and ${E}^{-}$, we have
$\begin{array}{rcl}〈{A}^{l}{u}_{n},{A}^{m}{v}_{n}〉& =& {\int }_{\mathrm{\Omega }}{A}^{l}{u}_{n}{A}^{m}{v}_{n}\phantom{\rule{0.2em}{0ex}}dx\\ =& {\int }_{\mathrm{\Omega }}{A}^{l}\left({\rho }_{n}{u}_{0}+{\phi }_{n}\right){A}^{m}\left({\rho }_{n}{v}_{0}+{\psi }_{n}\right)\phantom{\rule{0.2em}{0ex}}dx\\ =& {\int }_{\mathrm{\Omega }}\left({\rho }_{n}{A}^{l}{u}_{0}+{A}^{l}{\phi }_{n}\right)\left({\rho }_{n}{A}^{m}{v}_{0}+{A}^{m}{\psi }_{n}\right)\phantom{\rule{0.2em}{0ex}}dx\\ =& {\int }_{\mathrm{\Omega }}\left({\rho }_{n}^{2}{|{A}^{l}{u}_{0}|}^{2}-{\rho }_{n}{A}^{l}{u}_{0}{A}^{l}{\phi }_{n}+{\rho }_{n}{A}^{l}{u}_{0}{A}^{l}{\phi }_{n}-{|{A}^{m}{\psi }_{n}|}^{2}\right)\phantom{\rule{0.2em}{0ex}}dx\\ =& \frac{{\rho }_{n}^{2}}{2}{\parallel {z}_{0}\parallel }_{E}^{2}-\frac{1}{2}{\parallel {z}_{n}^{-}\parallel }_{E}^{2}.\end{array}$

Hence,
$\mathrm{\Phi }\left({z}_{n}\right)=\frac{1}{2}\left({\rho }_{n}^{2}{\parallel {z}_{0}\parallel }_{E}^{2}-{\parallel {z}_{n}^{-}\parallel }_{E}^{2}\right)-{\int }_{\partial \mathrm{\Omega }}F\left(x,{v}_{n}\right)\phantom{\rule{0.2em}{0ex}}d\sigma -{\int }_{\partial \mathrm{\Omega }}G\left(x,{u}_{n}\right)\phantom{\rule{0.2em}{0ex}}d\sigma >0.$
Therefore,
$\frac{\mathrm{\Phi }\left({z}_{n}\right)}{{\parallel {z}_{n}\parallel }_{E}^{2}}=\frac{1}{2}\left(\frac{{\rho }_{n}^{2}}{{\parallel {z}_{n}\parallel }_{E}^{2}}{\parallel {z}_{0}\parallel }_{E}^{2}-\frac{{\parallel {z}_{n}^{-}\parallel }_{E}^{2}}{{\parallel {z}_{n}\parallel }_{E}^{2}}\right)-{\int }_{\partial \mathrm{\Omega }}\frac{F\left(x,{v}_{n}\right)+G\left(x,{u}_{n}\right)}{{\parallel {z}_{n}\parallel }_{E}^{2}}\phantom{\rule{0.2em}{0ex}}d\sigma >0.$
Denote ${\delta }_{n}:=\frac{{\rho }_{n}}{{\parallel {z}_{n}\parallel }_{E}}$, ${w}_{n}^{-}:=\frac{{z}_{n}^{-}}{{\parallel {z}_{n}\parallel }_{E}}=\left({\stackrel{˜}{\phi }}_{n},{\stackrel{˜}{\psi }}_{n}\right)$. Then
$\frac{\mathrm{\Phi }\left({z}_{n}\right)}{{\parallel {z}_{n}\parallel }_{E}^{2}}=\frac{1}{2}\left({\delta }_{n}^{2}-{\parallel {w}_{n}^{-}\parallel }_{E}^{2}\right)-{\int }_{\partial \mathrm{\Omega }}\frac{F\left(x,{v}_{n}\right)+G\left(x,{u}_{n}\right)}{{\parallel {z}_{n}\parallel }_{E}^{2}}\phantom{\rule{0.2em}{0ex}}d\sigma >0.$
(3.3)

Since $F\left(x,t\right),G\left(x,t\right)\ge 0$ for any $\left(x,t\right)\in \partial \mathrm{\Omega }×{\mathbb{R}}^{1}$, by (3.3) we know that ${\delta }_{n}\ge {\parallel {w}_{n}^{-}\parallel }_{E}$.

On the other hand, ${\delta }_{n}^{2}+{\parallel {w}_{n}^{-}\parallel }_{E}^{2}=1$, which implies that ${\delta }_{n}^{2}\to {\delta }^{2}\ge 0$ for some $\delta \ge 0$ and ${w}_{n}^{-}⇀w=\left(\stackrel{˜}{\phi },\stackrel{˜}{\psi }\right)\in E$ as $n\to \mathrm{\infty }$, where denotes the weak convergence in E.

If $\delta =0$, then from (3.3) we get
${\parallel {w}_{n}^{-}\parallel }^{2}\to 0,\phantom{\rule{2em}{0ex}}{\int }_{\partial \mathrm{\Omega }}\frac{F\left(x,{v}_{n}\right)}{{\parallel {z}_{n}\parallel }_{E}^{2}}\phantom{\rule{0.2em}{0ex}}d\sigma \to 0,\phantom{\rule{2em}{0ex}}{\int }_{\partial \mathrm{\Omega }}\frac{G\left(x,{u}_{n}\right)}{{\parallel {z}_{n}\parallel }_{E}^{2}}\phantom{\rule{0.2em}{0ex}}d\sigma \to 0.$
Therefore,
$1={\delta }_{n}^{2}+{\parallel {w}_{n}^{-}\parallel }_{E}^{2}\to 0,$

which is impossible.

If $\delta >0$, since ${\delta }_{n}^{2}\to {\delta }^{2}>0$ and ${\parallel {z}_{n}\parallel }_{E}\to +\mathrm{\infty }$ as $n\to \mathrm{\infty }$, it follows that ${\rho }_{n}\to +\mathrm{\infty }$. If $x\in \partial \mathrm{\Omega }$ is such that $\delta {u}_{0}+\stackrel{˜}{\phi }\ne 0$, we have
$\underset{n\to +\mathrm{\infty }}{lim}\frac{{\rho }_{n}{u}_{0}+{\phi }_{n}}{{\parallel {z}_{n}\parallel }_{E}}=\delta {u}_{0}+\stackrel{˜}{\phi }\left(x\right)\ne 0,$
thus,
${u}_{n}={\rho }_{n}{u}_{0}+{\phi }_{n}\to \mathrm{\infty },$
(3.4)

as $n\to \mathrm{\infty }$.

Similarly, if $\delta {u}_{0}\left(x\right)+\stackrel{˜}{\psi }\left(x\right)\ne 0$, we have
${v}_{n}={\rho }_{n}{v}_{0}+{\psi }_{n}\to \mathrm{\infty },$
(3.5)

as $n\to \mathrm{\infty }$.

Since $\frac{\mathrm{\Phi }\left({z}_{n}\right)}{{\parallel {z}_{n}\parallel }_{E}^{2}}>0$ and $F\left(x,t\right),G\left(x,t\right)\ge 0$, we get
$\begin{array}{rcl}0& <& \frac{1}{2}\left({\delta }_{n}^{2}-{\parallel {w}_{n}^{-}\parallel }_{E}^{2}\right)-{\int }_{\partial \mathrm{\Omega }}\left[\frac{F\left(x,{v}_{n}\right)}{{v}_{n}^{2}}{\left(\frac{{v}_{n}}{{\parallel {z}_{n}\parallel }_{E}}\right)}^{2}+\frac{G\left(x,{u}_{n}\right)}{{u}_{n}^{2}}{\left(\frac{{u}_{n}}{{\parallel {z}_{n}\parallel }_{E}}\right)}^{2}\right]\phantom{\rule{0.2em}{0ex}}d\sigma \\ \le & \frac{1}{2}\left({\delta }_{n}^{2}-{\parallel {w}_{n}^{-}\parallel }_{E}^{2}\right)-{\int }_{\left\{\delta {u}_{0}+\stackrel{˜}{\psi }\ne 0\right\}}\frac{F\left(x,{v}_{n}\right)}{{v}_{n}^{2}}{\left(\frac{{v}_{n}}{{\parallel {z}_{n}\parallel }_{E}}\right)}^{2}\phantom{\rule{0.2em}{0ex}}d\sigma \\ -{\int }_{\left\{\delta {u}_{0}+\stackrel{˜}{\phi }\ne 0\right\}}\frac{G\left(x,{u}_{n}\right)}{{u}_{n}^{2}}{\left(\frac{{u}_{n}}{{\parallel {z}_{n}\parallel }_{E}}\right)}^{2}\phantom{\rule{0.2em}{0ex}}d\sigma .\end{array}$
Note that
and
as $n\to +\mathrm{\infty }$. Hence, by Proposition 2.1 we may assume, passing to a subsequence, that
as $n\to +\mathrm{\infty }$. By (3.8), (3.9) and (H4), taking limit in (3.10), using Fatou’s lemma and the fact that ${lim inf}_{n\to \mathrm{\infty }}{\parallel {w}_{n}^{-}\parallel }_{E}^{2}\ge {\parallel {w}^{-}\parallel }_{E}^{2}$, we obtain
$\begin{array}{rcl}0& \le & \frac{1}{2}\left({\delta }^{2}-{\parallel {w}^{-}\parallel }_{E}^{2}\right)-{\int }_{\left\{\delta {v}_{0}+\stackrel{˜}{\psi }\ne 0\right\}}\left(+\mathrm{\infty }\right){\left(\delta {v}_{0}+\stackrel{˜}{\psi }\right)}^{2}\phantom{\rule{0.2em}{0ex}}d\sigma -{\int }_{\left\{\delta {u}_{0}+\stackrel{˜}{\phi }\ne 0\right\}}\left(+\mathrm{\infty }\right){\left(\delta {u}_{0}+\stackrel{˜}{\phi }\right)}^{2}\phantom{\rule{0.2em}{0ex}}d\sigma \\ \to & -\mathrm{\infty },\end{array}$

which is impossible, thus the lemma is proved. □

Lemma 3.4 If $\left\{{z}_{n}\right\}$ is a (C) c -sequence of Φ, then $\left\{{z}_{n}\right\}$ is bounded in E.

Proof Suppose that $\left\{{z}_{n}\right\}\subset E$ is a (C) c sequence for Φ, that is,
$\mathrm{\Phi }\left({z}_{n}\right)\to c,\phantom{\rule{1em}{0ex}}{\parallel {\mathrm{\Phi }}^{\prime }\left({z}_{n}\right)\parallel }_{{E}^{\ast }}\left(1+{\parallel {z}_{n}\parallel }_{E}\right)\to 0,$
which shows that
$c+o\left(1\right)=\mathrm{\Phi }\left({z}_{n}\right),\phantom{\rule{2em}{0ex}}〈{\mathrm{\Phi }}^{\prime }\left({z}_{n}\right),{z}_{n}〉=o\left(1\right),$
(3.6)

where $o\left(1\right)\to 0$ as $n\to +\mathrm{\infty }$.

${\parallel {z}_{n}\parallel }_{E}\to +\mathrm{\infty },$
(3.7)
and let ${w}_{n}={\frac{{z}_{n}}{\parallel {z}_{n}\parallel }}_{E}:=\left({w}_{n}^{1},{w}_{n}^{2}\right)$. Then ${w}_{n}\in E$ with
${\parallel {w}_{n}\parallel }_{E}=1.$
By Proposition 2.1, $\left\{{w}_{n}\right\}$ contains a subsequence, denoted again by $\left\{{w}_{n}\right\}=\left\{\left({w}_{n}^{1},{w}_{n}^{2}\right)\right\}$ such that we may assume that
(3.8)
Let $\partial {\mathrm{\Omega }}_{\ne }=\left\{x\in \partial \mathrm{\Omega },w\left(x\right)\ne \left(0,0\right)\right\}$. Then we have
and (3.7) implies that
We may assume, without loss of generality, that
(3.9)
By (H4), we see that
This means that
(3.10)
By (H4), there is an ${N}_{0}>0$ such that
$\frac{G\left(x,s\right)}{{|s|}^{2}}>1$
(3.11)
for any $x\in \partial \mathrm{\Omega }$ and $s\in {\mathbb{R}}^{1}$ with $|s|\ge {N}_{0}$. Since $G\left(x,s\right)$ is continuous on $\partial \mathrm{\Omega }×\left[-{N}_{0},{N}_{0}\right]$, there is an $M>0$ such that
$|G\left(x,s\right)|\le M,$
(3.12)
for $\left(x,t\right)\in \overline{\mathrm{\Omega }}×\left[-{N}_{0},{N}_{0}\right]$. From (3.11) and (3.12), we see that there is a constant C, such that for any $\left(x,s\right)\in \partial \mathrm{\Omega }×{\mathbb{R}}^{1}$, we have
$G\left(x,s\right)\ge C,$
which shows that
$\frac{G\left(x,{u}_{n}\left(x\right)\right)-C}{{\parallel {z}_{n}\parallel }_{E}^{2}}\ge 0.$
This means that
$\frac{G\left(x,{u}_{n}\left(x\right)\right)}{{|{u}_{n}\left(x\right)|}^{2}}{|{w}_{n}^{1}\left(x\right)|}^{2}-\frac{C}{{\parallel {z}_{n}\parallel }_{E}^{2}}\ge 0.$
(3.13)
Since by (3.6) we have that
$c+o\left(1\right)=\mathrm{\Phi }\left({z}_{n}\right)=〈{A}^{l}{u}_{n},{A}^{m}{v}_{n}〉-{\int }_{\partial \mathrm{\Omega }}\left[F\left(x,{v}_{n}\right)+G\left(x,{u}_{n}\right)\right]\phantom{\rule{0.2em}{0ex}}d\sigma ,$
which shows that
$0←\frac{c+o\left(1\right)}{{\parallel {z}_{n}\parallel }_{E}^{2}}=〈{A}^{l}{w}_{n}^{1},{A}^{m}{w}_{n}^{2}〉-{\int }_{\partial \mathrm{\Omega }}\frac{\left[F\left(x,{v}_{n}\right)+G\left(x,{u}_{n}\right)\right]}{{\parallel {z}_{n}\parallel }_{E}^{2}}\phantom{\rule{0.2em}{0ex}}d\sigma .$
(3.14)
Since $F\left(x,t\right)\ge 0$, we have
$\begin{array}{rl}\frac{1}{2}+o\left(1\right)& =o\left(1\right)+\frac{1}{2}{\int }_{\mathrm{\Omega }}\left({|{A}^{l}{w}_{n}^{1}|}^{2}+{|{A}^{m}{w}_{n}^{2}|}^{2}\right)\phantom{\rule{0.2em}{0ex}}dx\ge o\left(1\right)+〈{A}^{l}{w}_{n}^{1},{A}^{m}{w}_{n}^{2}〉\\ ={\int }_{\partial \mathrm{\Omega }}\frac{\left[F\left(x,{v}_{n}\right)+G\left(x,{u}_{n}\right)\right]}{{\parallel {z}_{n}\parallel }_{E}^{2}}\phantom{\rule{0.2em}{0ex}}d\sigma \\ \ge {\int }_{\partial \mathrm{\Omega }}\frac{G\left(x,{u}_{n}\right)}{{\parallel {z}_{n}\parallel }_{E}^{2}}\phantom{\rule{0.2em}{0ex}}d\sigma .\end{array}$
(3.15)

We claim that $|\partial {\mathrm{\Omega }}_{\ne }|=0$.

If $|\partial {\mathrm{\Omega }}_{\ne }|\ne 0$, then by Fatou’s lemma, (H4) and Hölder’s inequality, we get
$\begin{array}{rcl}+\mathrm{\infty }& =& +\mathrm{\infty }|\partial {\mathrm{\Omega }}_{\ne }|\\ =& \left[{\int }_{\partial {\mathrm{\Omega }}_{\ne }}\underset{n\to +\mathrm{\infty }}{lim inf}\frac{G\left(x,{u}_{n}\left(x\right)\right)}{{|{u}_{n}\left(x\right)|}^{2}}{|{w}_{n}^{1}\left(x\right)|}^{2}\phantom{\rule{0.2em}{0ex}}d\sigma -{\int }_{\partial {\mathrm{\Omega }}_{\ne }}\underset{n\to +\mathrm{\infty }}{lim sup}\frac{C}{{\parallel {z}_{n}\parallel }_{E}^{2}}\phantom{\rule{0.2em}{0ex}}d\sigma \right]\\ =& {\int }_{\partial {\mathrm{\Omega }}_{\ne }}\underset{n\to +\mathrm{\infty }}{lim inf}\left(\frac{G\left(x,{u}_{n}\left(x\right)\right)}{{|{u}_{n}\left(x\right)|}^{2}}{|{w}_{n}^{1}\left(x\right)|}^{2}-\frac{C}{{\parallel {z}_{n}\parallel }_{E}^{2}}\right)\phantom{\rule{0.2em}{0ex}}d\sigma \\ \le & \underset{n\to +\mathrm{\infty }}{lim inf}{\int }_{\partial {\mathrm{\Omega }}_{\ne }}\left(\frac{G\left(x,{u}_{n}\left(x\right)\right)}{{|{u}_{n}\left(x\right)|}^{2}}{|{w}_{n}^{1}\left(x\right)|}^{2}-\frac{C}{{\parallel {z}_{n}\parallel }_{E}^{2}}\right)\phantom{\rule{0.2em}{0ex}}d\sigma \\ \le & \underset{n\to +\mathrm{\infty }}{lim inf}{\int }_{\partial \mathrm{\Omega }}\left(\frac{G\left(x,{u}_{n}\left(x\right)\right)}{{|{u}_{n}\left(x\right)|}^{2}}{|{w}_{n}^{2}\left(x\right)|}^{2}-\frac{C}{{\parallel {z}_{n}\parallel }_{E}^{2}}\right)\phantom{\rule{0.2em}{0ex}}d\sigma \\ =& \underset{n\to +\mathrm{\infty }}{lim inf}{\int }_{\partial \mathrm{\Omega }}\frac{G\left(x,{u}_{n}\left(x\right)\right)}{{\parallel {u}_{n}\parallel }^{2}}\phantom{\rule{0.2em}{0ex}}d\sigma -\underset{n\to +\mathrm{\infty }}{lim}{\int }_{\partial \mathrm{\Omega }}\frac{C}{{\parallel {z}_{n}\parallel }_{E}^{2}}\phantom{\rule{0.2em}{0ex}}d\sigma \\ =& \underset{n\to +\mathrm{\infty }}{lim inf}{\int }_{\partial \mathrm{\Omega }}\frac{G\left(x,{u}_{n}\left(x\right)\right)}{{\parallel {z}_{n}\parallel }_{E}^{2}}\phantom{\rule{0.2em}{0ex}}d\sigma \\ \le & \frac{1}{2}+o\left(1\right),\end{array}$

which is impossible.

This shows that
$|\partial {\mathrm{\Omega }}_{\ne }|=0.$

Hence $w\left(x\right)=\left(0,0\right)$ a.e. in Ω.

Since $\mathrm{\Phi }\left(t{z}_{n}\right)$ is continuous in $t\in \left[0,1\right]$, there exists ${t}_{n}\in \left[0,1\right]$ ($n=1,2,\dots$), such that
$\mathrm{\Phi }\left({t}_{n}{z}_{n}\right)=\underset{0\le t\le 1}{max}\mathrm{\Phi }\left(t{z}_{n}\right).$
As $〈{\mathrm{\Phi }}^{\prime }\left({z}_{n}\right),{z}_{n}〉=o\left(1\right)$, we see that
$〈{\mathrm{\Phi }}^{\prime }\left({t}_{n}{z}_{n}\right),{t}_{n}{z}_{n}〉=o\left(1\right).$
By (H5), then we get for $t\in \left[0,1\right]$ that
$\begin{array}{rl}2\mathrm{\Phi }\left(t{z}_{n}\right)\le & 2\mathrm{\Phi }\left({t}_{n}{u}_{n}\right)\\ =& 2\mathrm{\Phi }\left({t}_{n}{z}_{n}\right)-〈{\mathrm{\Phi }}^{\prime }\left({t}_{n}{z}_{n}\right),{t}_{n}{z}_{n}〉+o\left(1\right)\\ =& {\int }_{\partial \mathrm{\Omega }}\left[\left({t}_{n}{v}_{n}f\left(x,{t}_{n}{v}_{n}\right)-2F\left(x,{t}_{n}{v}_{n}\right)\right)+\left({t}_{n}{u}_{n}g\left(x,{t}_{n}{u}_{n}\right)-2G\left(x,{t}_{n}{u}_{n}\right)\right)\right]\phantom{\rule{0.2em}{0ex}}d\sigma +o\left(1\right)\\ \le & {\int }_{\partial \mathrm{\Omega }}\left[\left({v}_{n}f\left(x,{v}_{n}\right)-2F\left(x,{v}_{n}\right)+{C}_{1,\ast }\right)\\ +\left({u}_{n}g\left(x,{u}_{n}\right)-2G\left(x,{u}_{n}\right)+{C}_{2,\ast }\right)\right]\phantom{\rule{0.2em}{0ex}}d\sigma +o\left(1\right)\\ =& 2c+\left({C}_{1,\ast }+{C}_{2,\ast }\right)|\partial \mathrm{\Omega }|+o\left(1\right).\end{array}$
(3.16)
On the other hand, taking $t=\frac{R}{{\parallel {z}_{n}\parallel }_{E}}\in \left[0,1\right]$ and ${z}_{n}=\left({u}_{n},{u}_{n}\right)$, by (3.8) then ${w}_{n}=\frac{\left({u}_{n},{u}_{n}\right)}{{\parallel {z}_{n}\parallel }_{E}}:=\left({\theta }_{n},{\theta }_{n}\right)\in {E}^{1}×{E}^{1}$ and ${\theta }_{n}\to 0$ in ${L}^{r}\left(\partial \mathrm{\Omega }\right)$ ($1\le r<\frac{2\left(N-1\right)}{N-4}$). From (3.2) and ${\theta }_{n}\to 0$ in ${L}^{r}\left(\partial \mathrm{\Omega }\right)$ ($1\le r<\frac{2\left(N-1\right)}{N-4}$) as $n\to \mathrm{\infty }$, we obtain
$2\mathrm{\Phi }\left(R{w}_{n}\right)=2{R}^{2}-2{\int }_{\partial \mathrm{\Omega }}\left[F\left(x,R{\theta }_{n}\right)+G\left(x,R{\theta }_{n}\right)\right]\phantom{\rule{0.2em}{0ex}}d\sigma =2{R}^{2}+o\left(1\right).$
So we have
$2{R}^{2}+o\left(1\right)=2\mathrm{\Phi }\left(R{w}_{n}\right)\le 2c+\left({C}_{1,\ast }+{C}_{2,\ast }\right)|\partial \mathrm{\Omega }|+o\left(1\right).$
Letting $n\to \mathrm{\infty }$, we get
$2{R}^{2}\le \left({C}_{1,\ast }+{C}_{2,\ast }\right)|\partial \mathrm{\Omega }|+2c.$

Letting $R\to \mathrm{\infty }$, we get a contradiction. This proves that ${\parallel {z}_{n}\parallel }_{E}\le C<+\mathrm{\infty }$ for some constant C. □

Proof of Theorem 1.5 Under the assumptions (H1)-(H5), we know that the functional Φ given by (2.5) is in ${C}^{1}\left(E,{\mathbb{R}}^{1}\right)$. By Lemma 3.2, there exist $r>0$ and $\alpha >0$ such that $\mathrm{\Phi }{|}_{{N}_{r}}\ge \alpha$, where ${N}_{r}=\left\{z\in {E}^{+}:{\parallel z\parallel }_{E}=r\right\}$. By Lemma 3.3, for such an r, there exist $R>r$ and suitable ${z}_{0}\in {E}^{+}\mathrm{\setminus }\left\{0\right\}$ such that $\mathrm{\Phi }{|}_{\partial {M}_{R}}\le 0$, where ${M}_{R}$ was given before Lemma 3.3. Note that $E={E}^{+}\oplus {E}^{-}$ and for $z=\left(u,v\right)\in E$, we have
$\mathrm{\Phi }\left(z\right)=\frac{1}{2}{\parallel {z}^{+}\parallel }_{E}^{2}-\frac{1}{2}{\parallel {z}^{-}\parallel }_{E}^{2}-{\int }_{\partial \mathrm{\Omega }}\left[F\left(x,v\right)+G\left(x,u\right)\right]\phantom{\rule{0.2em}{0ex}}d\sigma .$
Since from Proposition 2.1 and Remark 1.1 we know that $E\subset \subset {L}^{q}\left(\partial \mathrm{\Omega }\right)×{L}^{p}\left(\partial \mathrm{\Omega }\right)$, from (3.2) and Fatou’s lemma, we know that
$\mathcal{H}\left(z\right)={\int }_{\partial \mathrm{\Omega }}\left[F\left(x,v\right)+G\left(x,u\right)\right]\phantom{\rule{0.2em}{0ex}}d\sigma$

is ${C}^{1}$ and $\mathcal{H}\ge 0$ is weakly sequentially lower semicontinuous and ${\mathcal{H}}^{\prime }$ is weakly sequentially continuous in ${E}^{\ast }$. Hence by Proposition 3.1 there exists a (C) c -sequence $\left\{{z}_{n}\right\}$ for Φ, where $c\ge \alpha >0$. By Lemma 3.4, $\left\{{z}_{n}\right\}$ is bounded in E. So, up to a subsequence, we may assume that ${z}_{n}⇀z\ne \left(0,0\right)$ in E, as $n\to \mathrm{\infty }$. From Lemma 2.2, we know that ${\mathcal{H}}^{\prime }$ is compact. So it is easy to check that ${\mathrm{\Phi }}^{\prime }\left(z\right)\to 0$ in ${E}^{\ast }$. Hence z is a nontrivial solution pair of (1.1). Obviously, $z=\left(u,v\right)\in E$ is a nonnegative solution pair of (1.1). Applying the strong maximum principle, we obtain that $u>0$ and $v>0$. This completes the proof. □

Declarations

Acknowledgements

The authors were partially supported by NSFC (No. 11071092; No. 11071095; No. 11101171), the PhD specialized grant of the Ministry of Education of China (20110144110001).

Authors’ Affiliations

(1)
School of Mathematics and Statistics, Huazhong Normal University

References

1. Bartsch T, de Figueiredo DG: Infinitely many solutions of nonlinear elliptic systems. Progr. Nonlinear Differential Equations Appl. 35. In Topics in Nonlinear Analysis. Birkhäuser, Basel; 1999:51–67.
2. Clément P, de Figueiredo DG, Mitidieri E: Positive solutions of semilinear elliptic systems. Commun. Partial Differ. Equ. 1992, 17: 923–940. 10.1080/03605309208820869
3. de Figueiredo DG: Semilinear elliptic systems: a survey of superlinear problems. Resenhas 1996, 2: 373–391.
4. de Figueiredo DG, Felmer PL: On superquadratic elliptic systems. Trans. Am. Math. Soc. 1994, 343: 99–116.
5. de Figueiredo DG, Magalhaẽs CA: On nonquadratic Hamiltonian elliptic systems. Adv. Differ. Equ. 1996, 1: 881–898.
6. Felmer PL: Periodic solutions of “superquadratic” Hamiltonian systems. J. Differ. Equ. 1993, 102: 188–207. 10.1006/jdeq.1993.1027
7. Felmer PM, Manásevich RF, de Thélin F: Existence and uniqueness of positive solutions for certain quasilinear elliptic systems. Commun. Partial Differ. Equ. 2013, 17: 1992–2029.
8. Felmer PL, Wang ZQ: Multiplicity for symmetric indefinite functionals: application to Hamiltonian and elliptic systems. Topol. Methods Nonlinear Anal. 1998, 12: 207–226.
9. Fernández Bonder J, Rossi JD: Existence for an elliptic system with nonlinear boundary conditions via fixed point methods. Adv. Differ. Equ. 2001, 6: 1–20.
10. Hulshof J, van der Vorst R: Differential systems with strongly indefinite variational structure. J. Funct. Anal. 1993, 114: 32–58. 10.1006/jfan.1993.1062
11. Li GB, Szulkin A: An asymptotically periodic Schrödinger equation with indefinite linear part. Commun. Contemp. Math. 2002, 4: 763–776. 10.1142/S0219199702000853
12. Li GB, Wang CH:The existence of nontrivial solutions to a semilinear elliptic system on ${\mathbb{R}}^{N}$ without the Ambrosetti-Rabinowitz condition. Acta Math. Sci. 2010, 30(6):1917–1936.
13. Fernández Bonder J, Pinasco JP, Rossi JD: Existence results for a Hamiltonian elliptic system with nonlinear boundary conditions. Electron. J. Differ. Equ. 1999, 1999(40):1–15. (electronic)
14. Fernández Bonder J, Pinasco JP, Rossi JD: Infinitely many solutions for an elliptic system with nonlinear boundary conditions. Electron. J. Differ. Equ. Conf. 2000, 6: 141–154.
15. Miyagaki OH, Souto MAS: Superlinear problems without Ambrosetti and Rabinowitz growth condition. J. Differ. Equ. 2008, 245: 3628–3638. 10.1016/j.jde.2008.02.035
16. Li GB, Wang CH: The existence of a nontrivial solution to a nonlinear elliptic problem of linking type without the Ambrosetti-Rabinowitz condition. Ann. Acad. Sci. Fenn., Ser. A 1 Math. 2011, 36: 461–480.
17. Gilbarg D, Trudinger NS: Elliptic Partial Differential Equations of Second Order. Springer, Berlin; 2001.
18. Taylor ME Texts in Applied Mathematics 23. In Partial Differential Equations: Basic Theory. Springer, New York; 1996.
19. Thayer J Projecto Euclides. In Operadores auto-adjointos e equacões differenciais parciais. IMPA, Rio de Janeiro; 1987.Google Scholar
20. Lions JL, Magenes E 3. In Pronlémes aux limites non homogénes et applications. Dunod, Paris; 1970.Google Scholar
21. Rabinowitz P: Minimax methods in critical point theory with applications to differential equations. 65. In CBMS Regional Conference Series in Mathematics. Am. Math. Soc., Providence; 1986.Google Scholar
22. Kryszewski W, Szulkin A: Generalized linking theorem with an application to semilinear Schrödinger equation. Adv. Differ. Equ. 1998, 3: 441–472.