How to calculate magnitude of a star in a triple star system?

Basically you need to convert between luminosities (which you can add) and magnitudes using
$$M-M_odot=-2.5log_{10}(L/L_odot)$$
Let's call the total luminosity $L_0$ and magnitude $M_0$ and the individual luminosities and magnitudes $L_1$, $L_2$ and $L_3$ and $M_1$, $M_2$ and $M_3$.
Then, you have the total luminosity of the system, directly
$$L_0/L_odot=10^{-0.4(M_0-M_odot)}$$
and as the sum of the components
$$L_0/L_odot=(L_1+L_2+L_3)/L_odot=10^{-0.4(M_1-M_odot)}+10^{-0.4(M_2-M_odot)}+10^{-0.4(M_3-M_odot)}$$
Solving these equations for $M_3$ gives
$$M_3-M_odot=-2.5log_{10}left(10^{-0.4(M_0-M_odot)}-10^{-0.4(M_1-M_odot)}-10^{-0.4(M_2-M_odot)}right)$$
I'm assuming you have absolute magnitudes, but you can rewrite the formulae in terms of apparent magnitudes using
$$M=m+5(1-log_{10}d)$$
but I think the result then also depends on the distance.