It is impossible for an unprivileged process to create a file that it can read but other processes running as that user cannot.
Proof: suppose process A (unprivileged) has the file open, and process B (running as the same user) wants to open the file as well. Process B can call
ptrace to take control of process A. The
ptrace system call lets process B execute arbitrary code inside process A, including establishing a unix socket to process A and passing the open file descriptor over it (file descriptor passing is a feature of unix datagram sockets that, as the name indicates, allows the sender to send a file descriptor to the receiver, after which the receiver has the same file open in the same mode).
It's possible that what your professor intended is to open a file then delete it. A deleted file — technically, an unlinked file — cannot be opened by conventional means: No process (even a privileged one) can open it with the
open system call, since there is no name that could be passed as an argument. But the process (or processes) that has the file open can continue working with it; the file will only actually be deleted when the last process that has it open dies. Other processes can still open a handle to the file by using
ptrace as described above.
In addition to the
ptrace method, on some unix variants including Linux, there may be other ways to open a file that's opened in another process, through