"I told him i wanted to tell him things, he freaks out before I even start and tells me that i need to let go, that he is just not the right person to give me what i needed to grow. He got stressed out so we stopped the conversation. "
This is fear operating and nothing much else. The fear of not being good enough.

"We meet up with some friends . He acts super affectionate and loving towards me all night."
This is him detecting their guilt which stirs his pride and removes his fears and enables him to close up to you

"I wonder if he actually DOES want me to be there, but feels too guilty about holding me back. Or maybe a bit out of both?"
To prevent you sympathetically looking at his fear he doesn't want you to be there. At the same time it is useful to him to generate anger in you as it means you are the one facing up to his fears not him. If he can confirm you are always angry, he can use you to face up to them any time he wishes. To get you into this mode he will say anything to get you suspicious. Once he generates suspicion in you he has you in the persistent state of anger he needs. He isn't going to move out. If you rumble him and confirm your suspicions he knows you are always going to be angry and face his fears for him even if he moves out and lives elsewhere. Perhaps

He feared his friends mightn't have accepted him as he was and felt he deserved to be accepted by them so he changed his attitude towards you. When he closed up to you he removed their guilt and the likelihood that they would be risk criticising him again. The only thing which removes blackness is anger. He had to detect they had an underlying anger towards him throughout the evening for exposing their guilt. Or he detected you had an underlying anger that he became intimate at their house and not at your own.

My opinion is don't let him detect your anger, find out what his fears are and tell him you don't need to use your anger because they don't scare you.

Something like this. Female SOs only