rt.representation theory - a question about irreducibility of representations and Kirillov conjecture

I think it's best to look at the relatively recent paper of Moshe Baruch, Annals of Math., "A Proof of Kirillov's Conjecture" -- in the introduction of his paper, he discusses the basic techniques of proof, and a bit of the history (Bernstein proved this conjecture in the p-adic case, for example).

Baruch and others (e.g. Kirillov, in the original conjecture, I think) consider unitary representations. This is necessary for the methods which they use. From the beginning, they use the "converse of Schur's lemma", i.e., if $Hom_P(V,V)$ is one-dimensional then $V$ is irreducible. This converse of Schur's lemma requires one to work in the unitary (or unitarizable) setting.

Now, to address Kevin Buzzard's point, consider $G = GL_2(F)$ for a $p$-adic field, and a unitary principal series representation $V = Ind_B^G chi delta^{1/2}$, where $chi = chi_1 boxtimes chi_2$ is a unitary character of the standard maximal torus, and $delta$ is the modular character for the Borel $B$.

Restricting $V$ back down to $B$, one gets a short exact sequence of $B$-modules:
$$0 rightarrow V(BwB) rightarrow V rightarrow V(B) rightarrow 0,$$
where $V(X)$ denotes a space of functions (compactly supported modulo $B$ on the left) on the ($B$-stable) locus $X$. On can check that these spaces are nonzero using the structure of the Bruhat cells, and hence the restriction of $V$ to $B$ is reducible as Kevin suggests.

But, if one considers the Hilbert space completion $hat V$ of $V$, with respect to a natural Hermitian inner product, one finds that $hat V$ is an irreducible unitary representation of $G$ which remains irreducible upon restriction to $B$ (and to the even smaller "mirabolic" subgroup of Kirillov's conjecture). Here it is important to note that "irreducibility" for unitary representations on Hilbert spaces refers to closed subspaces. The $B$-stable subspace $V(BwB)$ of $V$ is not closed, and its closure is all of $hat V$ I think.

So - I think that Kirillov's conjecture is false, in the setting of smooth representations of $p$-adic groups (and most probably for smooth representations of moderate growth of real groups).

However, the techniques still apply in the smooth setting to give weaker (but still useful) results. After all, it is still useful to know that $Hom_P(V,V)$ is one-dimensional! This can be used to prove multiplicity one for certain representations, for example.

The general technique to prove $Hom_P(V,V)$ is one-dimensional involves various forms of Frobenius reciprocity and characterization of distributions. Without explaining too much (you should look at old papers of Bernstein, perhaps), and being sloppy about dualities sometimes,
$$Hom_P(V,V) cong Hom_P(End(V), C) cong Hom_G(End(V), Ind_P^G C).$$
Some sequence of Frobenius reciprocity and linear algebra (I don't think I have it quite right above) identifies $Hom_P(V,V)$ with a space of functions or distributions: $f: G rightarrow End(V)$, which are $(P,V)$-bi-invariant. In other words,
$$f(p_1 g p_2) = pi(p_1^{-1}) circ f(g) circ pi(p_2),$$
or something close.

So in the end, one is led to classify a family of $P$-bi-quasi-invariant $End(V)$-valued distributions on $G$. This leads to two problems: one geometric, involving the $P$-double cosets in $G$. This is particularly easy for the "mirabolic" subgroup $P$. The second problem is often more difficult, analyzing distributions on each double coset, and proving most of them are zero or else have very simple properties.

Hope this clarifies a little bit... you might read more on the Gelfand-Kazhdan method (Gross has an exposition in the Bulletin) to understand this better.