There isn't a universal way, but you can make an educated guess by looking for things only done by one compiler.
GCC is the easiest; it writes a .comment section that contains the GCC version string (the same string you get if you run gcc --version). I don't know if there's a way to display it with readelf, but with objdump it's:
objdump -s --section .comment /path/binary
I just realized I ignored the rest of your question. Flags aren't generally saved anywhere; they would be in a comment section most likely, but I've never seen that done. There's a spot in the COFF header for a timestamp, but there's no equivalent in ELF, so I don't think the compile time is available either