Because ':3' seems a fairly generic string, the emoticon is entered as ':3 ' (note the trailing space) in the forum software. Alternatively, using '[:3]' will work, without the trailing space. Why the parser incorrectly parsed the first of the two ':3' in your second example I don't know, and I can't seem to replicate it.
Odd, I have seen it occur in the shoutbox and after posting a message, but I haven't seen the issue while composing a thread/comment. Also, the 3 :3lines in the first post were all created by clicking on the icon, so I assume that somewhere along the line, the trailing space is getting removed. :3:3 test line :3123 :3:3 :3 As you can tell from me messing around above, in some cases that trailing space is getting removed. Currently it seems that if the icon is the last string of text in a message, Xenforo removes the space before parsing icons. Edit: More specifically, if the icon is the last string of text on the line, rather than the message entirely. Editing a message (without modifying any previous text) and saving it will sometimes remove the space, as well.
Turns out the JavaScript responsible for inserting the emoticons after you click on them passes the strings through .trim(), explaining the first observation. The second observation stems from XenForo trimming most messages to some extent. White space at the beginning or end of lines, in particular.