Like I know that the char's exp won't go up, but will they still technically "take" some of the exp from the monster kills?
yes, you will still share exp with your party but it just wont be added to your character holding the stone, it will disappear