Finite dimensionality of certain $C^{star}$-algebras

In the discussion about the question Finite-dimensional subalgebras of $C^{star}$-algebras the following separate question came up:

Let $H$ be a Hilbert space and $a_1, dots, a_n in B(H)$ be self-adjoint operators. Consider the operators $x_1a_1+x_2a_2+dots + x_n a_n$ , where the $x_i$'s are complex variables and assume that there is a polynomial $p(z,x_1,dots,x_n) in mathbb C[z,x_1,dots,x_n]$ such that $z$ is in the spectrum of $x_1a_1+x_2a_2+dots + x_n a_n$ if and only if $p(z,x_1,dots,x_n)=0$.

Question: Is the subalgebra of $B(H)$ which is generated by the operators $a_1 , dots, a_n$ finite dimensional?