It depends on which popup menu state you are in (see
:help popupmenu-completion). I understand from your question that you're in state 2 (since you've pressed arrow keys to find a completion). However, the default behavior for Enter in state 2 is to insert the completion without newline; what you describe is normally the behavior of state 1 (which is when you use Ctrl+N/Ctrl+P.)
A way that works consistently in all states is to use Ctrl+Y. I like to remember the Y as standing for "yes, accept that word." It's also possible to just start typing the text that should come after the completed word, unless you've remapped things as in geedoubleya's answer.
In the same context, you can press Ctrl+E to cancel the menu and leave your text as it was before you invoked it. If you're used to the pairings of Ctrl+E and Ctrl+Y in other contexts (e.g. to scroll up or down in normal mode, or to insert the character below or above the cursor in insert mode), that's one way to remember it here. I guess you could also think of it as "exiting" the menu or similar.
:help popupmenu-keys for more.