How could Lt. Hicox fluently speak German and then give himself away by using the wrong gesture?

I'd say this gesture was something so subtle and unconcious that it just falls below any cultural assimilation. He has "counted to three" in the "British way" all over his life and even excessive exposure to German native speakers and culture probably won't change such a highly non-descript and intuitive gesture. You can speak fluently in a language and be acquainted with many of its cultural subtleties and might still not be aware of the most subtle an non-descript differences. It's not that if you do it the "wrong" way among a bunch of natives everybody will shout "hey, you're counting wrong". In most situations nobody will notice and thus nobody would have ever told him how to do it "properly". That's because in all but this particular covert operation nobody would have cared about how he counts to three.

Add to this that Hicox actually wasn't a professional spy trained for such situations. He was really only a British soldier who happened to have good knowledge about German movies and language (maybe even mainly from movies). It is true that they would have to face a situation amongst many German officers sooner or later (yet on a film fest, a métier Hicox would be much more familiar with). But in this particular situation they weren't prepared for any German soldiers at all, let alone such a cunning officer like Major Hellstrom, who continuously needled them and tried to bring them into revealing situations. Lt. Hicox just wasn't prepared well enough for this battle of wits and espionage where even the slightest misbehaviour counts (The situation may not even have gotten that far hadn't he spoken with such an Irish accent, which actually aroused Hellstrom's attention in the first place).

(As a counter example, I for myself think to speak English pretty fluently, even if maybe with a slight accent (though not stronger than Michael Fassbender's either), and have quite a bit of knowledge about Hollywood movies. This might even be sufficient to play an American film expert on a quest to bomb the Academy Awards. Yet I never ever heard about this difference in counting to three until this movie and would indeed intuitively order my drinks in the "German way" even when sitting in a bar full of U.S. officers.)