Terminals transmit characters¹, not keys. When you press a key or key combination like Ctrl+Alt+Enter, the terminal has to translate it to a character or character sequence. There aren't nearly enough characters to represent all keys, so most such combinations are transmitted as escape sequences: a sequence of characters starting with the escape character.
There's no universal standard for the escape sequences, though there are some common conventions. For example, Alt+key often sends the escape character followed by the character or escape sequence for key; but there's no standard for what Ctrl+Enter sends. Some terminals are configurable, some have hard-coded escape sequences.
See also Problems with keybindings when using terminal for a more detailed explanation as well as possible solutions with Xterm.
If you can't get Ctrl+Alt+Enter to be transmitted through your favorite terminal, you can choose a different key binding. Figure out what command Ctrl+Alt+Enter runs by pressing
C-h c C-M-return when you're using Matlab in a GUI Emacs. Then type
C-h w matlab-foo RET where
matlab-foo is the command name; this will tell you what key(s)
matlab-foo is bound to. If there isn't a binding that you can type conveniently, add one to your init file, with something like
(defun my-eval-after-load-matlab ()
(define-key matlab-mode-map [f9] 'matlab-foo))
(eval-after-load "matlab" '(my-eval-after-load-matlab))
"matlab", substitude the name of the Lisp file that defines the
matlab-foo command. To find out what that is, type
C-h k C-M-return; this tells you something like “C-M-return runs the command matlab-foo, which is an interactive compiled Lisp function in `matlab.el`”. The name of the Lisp file is that last bit on the line, minus the
.el part. For
matlab-mode-map, substitute the name of the keymap; this is generally the name of the major mode (
C-h v major-mode RET, “Its value is matlab-mode”) plus
-map at the end.
Alternatively, do you really need to use Screen? If you're only using it so that Emacs survives the disconnection, this isn't actually necessary. Start Emacs as a daemon (
emacs --daemon), and connect to it remotely over SSH by running
emacsclient -c to open a new GUI frame. You can open and close GUI frames on multiple displays, and Emacs will keep running even if the displays come and go.
¹ Actually bytes, but that bit is irrelevant here.