meaning - What does 'per say' mean in this context?

First, a dictionary definition:

Per se adverb
By or in itself or themselves; intrinsically: it is not these facts per se that are important
- ODO

Let's work with a more complete variant of that example.

Defendant: The facts are that only 50 cents was taken, and the money was returned.
Council: It is not these facts per se that are important, it is that they are repeated. This is the tenth time that we've had to do this.

This means that what is important isn't the facts intrinsically, it's something else (in this case, the repetition).

Let's now take this back to your example:

Not just for you [per se].

Assuming the context is one of giving 'you' a present, the sentence fragment has the sense that it's not given to (for) 'you' just because 'you' are 'you'. It might be because of who 'you' represent or for 'your' help, etc, but not for 'you'. Note that without further context, this can come across as somewhat awkward both socially and in the English, possibly even rude.