How is representation theory used in modular/automorphic forms?

Since you mentioned Galois representations, I can briefly discuss the simplest version of the connection there and point you to Diamond and Shurman's excellent book which discusses modular forms with an aim towards this perspective.

The connection here is to representations of the absolute Galois group $G = text{Gal}(bar{mathbb{Q}}/mathbb{Q})$. By the Kronecker-Weber theorem, one-dimensional (continuous, complex) representations of $G$ are classified by Dirichlet characters, so it is natural to ask about the next hardest case, the two-dimensional representations. A large class of them can be constructed as follows. Given an elliptic curve $E$ defined over $mathbb{Q}$, the elements of order $n$ (hereby designated by $E[n]$) form a group isomorphic to $(mathbb{Z}/nmathbb{Z})^2$, and since their coordinates are algebraic numbers, $G$ acts on them. This gives a representation

$$G to text{GL}_2(mathbb{Z}/nmathbb{Z}).$$

As is, this representation causes problems because $mathbb{Z}/nmathbb{Z}$ isn't an integral domain. So what we do is we take $n$ to be all the powers of $ell$ for a fixed prime $ell$ and take the inverse limit over all the corresponding $E[ell^n]$. The result is a gadget called a Tate module, which is a $G$-module isomorphic (as an abstract group) to $mathbb{Z}_{ell}^2$, and which therefore defines a representation

$$G to text{GL}_2(mathbb{Z}_{ell}).$$

So how does one identify the representation corresponding to $E$? The standard answer is to look at certain ("conjugacy classes" of) elements of $G$ called Frobenius elements, which come from lifts of Frobenius morphisms. Although Frobenius elements aren't always well-defined, it turns out that the trace $a_{p,E}$ of the Frobenius element corresponding to $p$ in a representation is, and so we can identify a representation by giving the numbers $a_p$ for all $p$. (I am not really familiar with the details here, but I believe this works because Frobenius elements are dense in $G$.) It turns out that if $p$ is a prime of good reduction, $a_{p,E} = p + 1 - |E(mathbb{F}_p)|$, so these numbers can actually be obtained in a fairly concrete manner (where $E(mathbb{F}_p)$ is the set of points of $E$ over $mathbb{F}_p$). (Again, I am not really familiar with the details here, including what happens when $p$ doesn't have good reduction.)

Now: one statement of the modularity theorem, formerly the Taniyama-Shimura conjecture, is that there exists a cusp eigenform $f$ of weight $2$ for $Gamma_0(N)$ for some $N$ (called the conductor of $E$) such that, whenever $p$ is a prime of good reduction,

$$a_{p, f} = a_{p, E}$$

where $a_{p, f}$ is the $p^{th}$ Fourier coefficient of $f$. In other words, cusp eigenforms of weight $2$ "are the same thing as" a large class of two-dimensional representations of $G$. The Langlands program is at least in part about generalizations of this statement to higher-dimensional representations of $G$, but there are many qualified number theorists here who can tell you what this is all about.

• Next ag.algebraic geometry - Is there any rational curve on an Abelian variety?
• Previous ag.algebraic geometry - how good an approximation to the equivariant derived category is given by the Grassmannian filtration of the classifying space?