UART, which I believe you are referring to is a bi-directional 2 wire communication protocol that is typically used by micro-controllers to talk to each other - commonly used to transfer text between two devices. It is what most people mean when they say
serial, but there are a lot of different serial protocols out there.
/dev/ttyAMA0 is a UART serial device on the raspberry pi. This device file controls the rx (10) and tx (8) pins on the raspberry pi header. Any program can open the serial port and read/write data to it, but with nothing attached to it all the data sent is just lost and there is nothing sending data back.
Most raspberry pi Linux distributions have a getty listening on
/dev/ttyACM0, this is a program that gives you a login prompt and shell. When you launch
/dev/ttyAMA0 you and the getty are sending data on the tx pin, and receiving data on the rx pin - this will confuse anything that is attached to them if you had anything attached you them.
To make the serial device do anything useful, it needs something to communicate with. You can get it to communicate with your pc by getting one of these USB to UART adaptors and attaching the rx to the tx pin, the tx to the rx pin and the gnd pin to gnd then plugging it into your pc. Once plugged in you will get a serial device on your pc (
/dev/ttyUSB0 on most Linux distros) and any data you send, will appear on
/dev/ttyAMA0 on the pi, and any data the pi sends on
/dev/ttyAMA0 will appear on your pc.
Given you the pi already has a getty listening, all you need to do is start
minicom or similar program on your pc and you should get a login prompt (might need to hit enter few times). You can of course stop the getty and run another program that can talk to the serial line.