10

This is more like a conceptual question. I need some clarifications.


Today I was learning some socket programming stuff and wrote a simple chat server and chat client based on Beej's Guide to Network Programming. (chat server receives clients message and send messages to all the other clients)

I copied the chat server and I wrote my own chat client.

The chat client is just a program to send stdin input to server and print socket data from server.

Later I noticed that the guide says I can just use telnet to connect to the server. I tried and it worked.


I was unfamiliar with telnet and for a long time I don't know what exactly it is.

So now my experience confuses me:

Isn't telnet just a simple TCP send/echo program? What makes it so special to be a protocol thing? My dumb chat client program doesn't create a [application] protocol.

From Wikipedia Communication_protocol :

In telecommunication, a communication protocol is a system of rules that allow two or more entities of a communications system to transmit information via any kind of variation of a physical quantity.

What rules does Telnet create? telnet host port, open a TCP stream socket for raw input/output? That's not a rule.

  • 3
    Even if telnet was a trivial case with raw input/output (answers say it's not, but if it was), then it still would be a protocol. In the same manner the zero vector is a vector. The zero vector has no direction and the trivial protocol has no overhead/header/footer/transformations. This distinguishes it from any other vector/protocol. – Kamil Maciorowski Nov 15 at 5:27
  • 1
    @KamilMaciorowski I like the idea of "trivial protocol with no overhead". In my answer I show this was the idea from the start in 1969. – rastafile Nov 15 at 6:26
  • 2
    It might add to the confusion that many Telnet clients change their behavior depending on what port you're using – they don't actively start option negotiation if you connect to a nonstandard port (i.e. not 23). – grawity Nov 15 at 16:09
  • 1
    I describe TELNET as being basically TCP, but with some added details like a "BREAK" signal which is part of the TELNET protocol but not part of the TCP protocol. Although, such details about TELNET are seldom used, so common usage of TELNET might be identical to TCP, but that doesn't mean that the entire protocol specifications are identical. – TOOGAM Nov 15 at 18:33
  • 1
    This may partly be a simple semantic confusion: "telnet" has two distinct meanings. It is (1) the communication protocol defined in the RFC; the programs or machines using it are not part of the RFC. It is (2) a command line user program on -nix machines using the telnet protocol -- hence the name -- to login to other machines. As you discovered, this program is one of the possible partners on the end of a RFC telnet session. – Peter - Reinstate Monica Nov 15 at 21:53
15

Telnet is defined in RFC 854. What makes it (and anything else) a protocol is a set of rules/constraints. One such rule is that Telnet is done over TCP, and assigned port 23 - this stuff might seem trivial, but it needs to be specified somewhere.

You can't just send whatever you want, there are limitations and special meaning to some things. For example, it defines a "Network Virtual Terminal" - this is because when telnet was established, there could be many different terminals: A printer, a black/white monitor, a color monitor that supported ANSI codes, etc.

Also, there's stuff like this:

In summary, WILL XXX is sent, by either party, to indicate that party's desire (offer) to begin performing option XXX, DO XXX and DON'T XXX being its positive and negative acknowledgments; similarly, DO XXX is sent to indicate a desire (request) that the other party (i.e., the recipient of the DO) begin performing option XXX, WILL XXX and WON'T XXX being the positive and negative acknowledgments. Since the NVT is what is left when no options are enabled, the DON'T and WON'T responses are guaranteed to leave the connection in a state which both ends can handle. Thus, all hosts may implement their TELNET processes to be totally unaware of options that are not supported, simply returning a rejection to (i.e., refusing) any option request that cannot be understood.

In modern times, most of the stuff isn't really that important anymore (then again, telnet as a protocol isn't being used much anymore, not just because it lacks security) so in practice it boils down to send/echo unless you have to actually interface with terminals.

  • 1
    Ok I see. In short, telnet has its own protocol which we barely touch nowadays. And the send/echo TCP raw input/output service is just part of its functionality. – Rick Nov 15 at 9:47
  • 3
    One surprising place Telnet is still used—FTP. FTP runs “on top” of Telnet, and in theory, FTP clients and servers must be able to respond to the WILL / DO / DON'T commands. – Dietrich Epp Nov 15 at 15:20
  • And what makes Telnet rather exceptional is that it uses TCP "urgent data". Which I've never heard that anything else would use. – ilkkachu Nov 15 at 15:28
  • @ilkkachu ftp uses it too. – mosvy Nov 15 at 19:35
  • @Rick Be careful. Telnet does not send raw data bytes. Some bytes have defined meaning in the telnet protocol and you MUST handle/encode them properly if you want to send them via telnet – slebetman Nov 16 at 9:34
7

The fact that telnet “just works” when connecting two terminals is in itself a protocol decision: the designers of telnet decided to use a model with “network virtual terminals” at either end of a TCP connection, and decided to model those virtual terminals on basic terminal features (teletypes really).

Beyond that, telnet is more than a simple TCP-based echo service; it supports control codes, sent in-band, for various purposes. It is codified in RFC 854.

  • 1
    Ok I see. In short, telnet has its own protocol which we barely touch nowadays. And the send/echo TCP raw input/output service is just part of its functionality. – Rick Nov 15 at 9:47
  • 1
    @Rick Carefully read this answer. Telnet handles some control codes (bytes) in special ways. By definition this means it does not send raw input/output. Telnet is only "safe" for printable characters – slebetman Nov 16 at 9:36
3

A sub-system called "Telnet" is proposed which is a shell program around the network system primitives, allowing a teletype or similar terminal at a remote host to function as a teletype at the serving host.

This is from RFC 15, which is mentioned in wikipedia "telnet". This article also starts by calling telnet a "application protocol".

The RFC 15 is from September 1969 - just some days ago I heard an interesting interview with the guy who made the first "internet" connection, exactly 50 years ago. His story went like that:

"Unlike N. Armstrong, we hadn't thought of any special message. But after typing "lo" (for "login"), the system crashed. So "lo" (as in "lo and behold") was the first two characters transmitted over the internet. In retrospect, we couldn't have thought of something better."

(Here is some of that interview: wiki arpanet Kleinrock)


Telnet predates TCP/IP. RFC 15 speaks of "network primitives". Because of TCP/IP, telnet was only left to take care of the "rest", the application layer (the "negotiated options" in RFC854).

Yes, telnet is quite simple, even without TCP. RFC 15 ends with:

TELNET subsystem constitutes a "level 0" network program which will quickly be surpassed. It is, however, simple enough to be working fairly soon.


"Protocol" is used with telnet because inherently two (different) systems need to talk in one "language".

RFC 854 also explains the concept:

The TELNET Protocol is built upon three main ideas: first, the concept of a "Network Virtual Terminal"...


1

The existing answers already explain that telnet can do more than just stream input data over TCP unchanged; I'd like to instead focus on the "My dumb chat client program doesn't create a [application] protocol." -part of the question. (As an aside, for actual "raw data over TCP", you can use netcat instead of telnet)

Just because a protocol is simple or bad - or not even well-specified anywhere - doesn't mean it's not a protocol. Two systems can't communicate without having an agreed-upon protocol, even if that protocol is just "raw data over TCP". If you try speaking FTP on one end and HTTP on the other, you're going to have problems, one way or another - if you're lucky, it won't work; if you're not, you'll get wrong behaviour. (It at least used to be the case that if you ran an FTP daemon on a non-standard port on the same host as an HTTP server, on some browsers you could use that to perform XSS and bypass CSRF protections by pointing the victim's browser to POST to the FTP server, which would echo the POSTed content in error messages, and the browser would interpret it as a HTTP/0.9 response - and happily run any javascript.)

It is a perfectly valid question to ask "What protocol does that thing you call my dumb chat client program use? I want to talk to it from my program." and one possible answer would be "raw data over TCP". It's not an exhaustive answer, however.

  • If you have some default port, that should probably be part of the answer.
  • If you wanted to be extremely clear, you could specify that you mean "normal TCP data, not TCP urgent/out-of-band data" (which is a bit of a stretch, since it wouldn't make any sense, and judging by the amount of people I've seen that think the third fd set of select() is for errors, the majority of people don't even seem to know out-of-band data exists - but even if it's bit of a bad example, I hope you see what I'm getting at)
  • Even things you might not even have thought about might implicitly be part of the protocol: How do you close the connection? Do you just call close() or do you use shutdown() to make sure all pending data is first delivered?

You said "open a TCP stream socket for raw input/output? That's not a rule.", but is that really so? "You must use TCP", "you must send the data as-is" and "you should use port XXX" sound like rules to me.

(P.S. I got a bit swept up in writing this; The rest of this answer is a bit tangential, and tries to answer the natural follow-up question of "Okay, so everything is a protocol - does that have any implications?")

So, keeping in mind that every time you make two systems talk to each other, you're either using an existing protocol or creating a new protocol: Document your protocols before it's too late! (Ie. at the latest, when the protocol grows beyond simple, or starts having more users than just the developers) Eg. many early P2P-protocols said "the source of the official client is the documentation", which is extremely annoying in a complex project, and just leads to clients speaking subtly different protocols and being incompatible here and there.

There's the old adage of "be strict in what you produce and liberal in what you accept" - the core sentiment is nice, but frankly, at least the way it tends to be applied, I think that it's bullcrap, and only leads to more problems in maintainability and compatibility. I say: Specify in detail (try to think of every corner case), and be extremely strict in what you produce and accept, and you'll likely prevent/solve real bugs.

An example, about strictness solving bugs: Years ago, a certain bittorrent tracker began having a problem: Some of their torrents didn't work on some clients; the clients didn't complain, they just got a different infohash, and thus couldn't find the torrent. They couldn't figure out what was going on, and declared that the affected clients must be buggy - stop using them. Bittorrent uses bencoding, which is very strictly and well specified - and for a good reason: Same input always results in the exact same encoded output string, so that the [info]hash is always same for the same input data. When I first bumped into one of those torrents, I tried feeding it into my own - very strict - parser, and straight away it said: Error: Bencoded dictionary keys not sorted (last='sha1', key='private').

So, what happened? The tracker started automatically adding the key "private" to each torrent that was uploaded, but just added it to the end, instead of keeping the dictionary keys sorted, like bencoding specifies - it wasn't strict in what it produced. Some clients decoded the the torrent only partially, and calculated the hash for the undecoded, unaltered string. Some clients decoded the whole torrent, and re-encoded the part they needed to hash, sorting the keys correctly in the process (creating a valid bencoding), and got a different hash. To my knowledge/recollection, no client complained that the data was invalid - the clients weren't strict in what they accepted. So, which hashes were correct? Neither! Both ways of calculating the hash are correct, but the torrents were corrupted! If even one of the clients would've complained about it, the bug would've likely been fixed the same day, instead of it taking months. (IIRC I also reported an identical bug in another completely unrelated tracker some years later; the software of the first one was made in-house, so it wasn't the same codebase either)

Another example, this time about corner cases: The eDonkey2000 P2P-protocol used a custom hash called ed2k; The specification didn't specify the behaviour in the corner-case of input data length being exactly divisible by the block size, so two incompatible variants were born, which produce a different hash for some files, and are incompatible. (I said "specification", but since this was an early P2P-protocol, I'm pretty sure there wasn't a specification; the special-case was probably missed when reverse-engineering the protocol)

About the need for detailed specifications: Many times, when trying to write a client for a protocol or a parser for a file format, I've encountered a situation where even the original programmers have no idea how some data is actually encoded, because they've used some library, and have never checked in full detail what it actually does. Two examples:

A certain API used a custom UDP protocol, with encrypted [variable-length] packets - the documentation specified AES128, but didn't mention the used padding. When the developers were queried about it, the response was along the lines of:

Uh, we actually have no idea, we just used this function in this Java library, and the documentation doesn't seem to mention which padding it uses

The website of a certain device has this to say about interfacing with their savefiles:

It is not possible to create or modify a saved *.logicdata files. That is because the files are generated with boost binary serialization and contain a very complex object structure. In truth, we don't even understand how it works. Boost serialization is extremely complex.

(Note: In reality, though, it doesn't take that long to reverse-engineer enough of the file format to be able to export the data)

Some of the examples were about encodings, file formats or custom algorithms (though still used inside protocols, except for the last one), but I'd say all of this applies to all of those. No matter if you're talking about a protocol, a file format, an encoding or a custom algorithm:

  • Write a good specification / document it well
    • Try to think about all the corner-cases
    • Try to include all the needed details
  • Follow the specification to the letter / be strict
1

The telnet protocol is a very simple protocol but it exists. By "exists" I mean that a telnet client does not send all bytes it receives to the program reading form it (usually a shell).

The protocol is

The byte 0xff (255) means that the next byte is a Telnet command. If you want to send 0xff then you must double it (0xff, 0xff) to tell telnet that you don't intend to send it a command.

So yes, the protocol is extremely simple, almost trivial. But is is not a simple tcp pass-through.


Telnet Commands

If you're interested what commands you can send to telnet (instead of the software/server on the other side) you can read the RFC: https://tools.ietf.org/html/rfc854. Basically telnet defines:

  NAME               CODE              MEANING

  SE                  240 (0xf0)    End of subnegotiation parameters.
  NOP                 241 (0xf1)    No operation.
  Data Mark           242 (0xf2)    The data stream portion of a Synch.
                                    This should always be accompanied
                                    by a TCP Urgent notification.
  Break               243 (0xf3)    NVT character BRK.
  Interrupt Process   244 (0xf4)    The function IP.
  Abort output        245 (0xf5)    The function AO.
  Are You There       246 (0xf6)    The function AYT.
  Erase character     247 (0xf7)    The function EC.
  Erase Line          248 (0xf8)    The function EL.
  Go ahead            249 (0xf9)    The GA signal.
  SB                  250 (0xfa)    Indicates that what follows is
                                    subnegotiation of the indicated
                                    option.
  WILL (option code)  251 (0xfb)    Indicates the desire to begin
                                    performing, or confirmation that
                                    you are now performing, the
                                    indicated option.
  WON'T (option code) 252 (0xfc)    Indicates the refusal to perform,
                                    or continue performing, the
                                    indicated option.
  DO (option code)    253 (0xfd)    Indicates the request that the
                                    other party perform, or
                                    confirmation that you are expecting
                                    the other party to perform, the
                                    indicated option.
  DON'T (option code) 254 (0xfe)    Indicates the demand that the
                                    other party stop performing,
                                    or confirmation that you are no
                                    longer expecting the other party
                                    to perform, the indicated option.
  IAC                 255 (0xff)    Data Byte 255.

Note that the commands above are specifically Telnet commands. They are not part of any client or server side terminal protocols. For example, if you want to terminate a process in unix you would normally type ctrl-c. This is normally sent as 0x03 form the terminal to the shell which will then send SIGINT to the process the shell is running. But telnet itself defines 0xff, 0xf4 to send SIGINT to the process which is a telnet specific protocol to do it. Similarly, terminals normally send 0x1b, 0x4b to clear the current line but telnet defines 0xff, 0xf8.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.