ag.algebraic geometry - When is a homogeneous space a variety?

I'll try to answer both questions, though I will change the first question somewhat. Let's work in the setting of a real reductive algebraic group $G$ and a closed subgroup $H subset G$.

Your first question asks when $G/H$ is an open subset of some (presumably complex) variety. I think that this question should be modified in a few ways.

You can't really say that $G/H$ "is a subset" of a variety, since $G/H$ is not a priori endowed with a complex structure. So you need a bit more data to go with the question -- a complex structure on the homogeneous space $G/H$. Such a complex structure can be given by an embedding of the circle group $U(1)$ as a subgroup of the center of $H$. Let $phi: U(1) rightarrow G$ be such an embedding, and let $iota = phi(i)$ be the image of $e^{pi i} in U(1)$ under this map. Such an embedding yields an integrable complex structure on the real manifold $G/H$, I believe (though I haven't seen this stated in this degree of generality).

So now one can ask if $G/H$, endowed with such a complex structure, is an open subset of a complex algebraic variety. But again, I have some objection to this question -- it's not really the right one to ask. Indeed, it's very interesting when one finds that some quotients $Gamma backslash G /H$ are (quasiprojective) varieties -- but such quotients are not obtained as quotients in a category of varieties, from $G/H$ to $Gamma backslash G / H$. They are complex analytic quotients, but not quotient varieties in any sense that I know.

So what's the point of knowing whether $G/H$ is an open subset of a variety? Really, one needs to know properties of $G/H$ as a Riemannian manifold and complex analytic space (e.g. curvature, whether it's a Stein space). That's the most important thing!

As Kevin Buzzard and his commentators note, under the assumption that $G$ comes from a reductive group over $Q$, and under the assumption that $H$ is a maximal compact subgroup of $G$, and under the assumption that there is a "Shimura datum" giving the quotient $G/H$ a complex structure, the quotient $G/H$ is a period domain for Hodge structures, and the quotients $Gamma backslash G / H$ are quasiprojective varieties when $Gamma$ is an arithmetic subgroup of $G$.

But these are quite strong conditions, on $G$ and on $H$! I have also wondered about other situations when $X = Gamma backslash G / H$ might have a natural structure of a quasiprojective variety. A general technique to prove such a thing is to use a differential-geometric argument. A great theorem along this line is due to Mok-Zhong (Compactifying complete Kähler-Einstein manifolds of finite topological type and bounded curvature, Ann. of Math 1989). The theorem, as quoted from MathSciNet, reads:

"Let $X$ be a complex manifold of finite topological type. Let $g$ be a complete Kähler metric on $X$ of finite volume and negative Ricci curvature. Suppose furthermore that the sectional curvatures are bounded. Then $X$ is biholomorphic to a Zariski-open subset $X'$ of a projective algebraic variety $M$."

Such results can be applied to prove quasiprojectivity of Shimura varieties of Hodge type. I believe I first learned this by reading J. Milne's notes on Shimura varieties.

I tried once to apply this to an arithmetic quotient of $G/H$, where $H$ was a bit smaller than a maximal compact (when $G/H$ was the twistor covering of a quaternionic symmetric space) -- I couldn't prove Mok-Zhong's conditions for quasiprojectivity, and I still don't know whether such quotients are quasiprojective.