There isn't a universal way, but you can make an educated guess by looking for things only done by one compiler.
GCC is the easiest; it writes a
.comment section that contains the GCC version string (the same string you get if you run
gcc --version). I don't know if there's a way to display it with
readelf, but with
objdump -s --section .comment /path/binary
I just realized I ignored the rest of your question. Flags aren't generally saved anywhere; they would be in a comment section most likely, but I've never seen that done. There's a spot in the COFF header for a timestamp, but there's no equivalent in ELF, so I don't think the compile time is available either