I have to get an output from netcat, decode it and return it.

after typing:

nc cs2107.spro.ink 9000

the output i get is this:

Welcome to the Proof of work challenge!  
Rules: i will provide hex encoded byte strings to you.  
Your task is to decode it and calculate the md5 hash in hex encoded format and return it back to me. You will need to do this 500 times!

Ready? Go!



So I have to get take in that hex encoded string, decode it and output it again and that's suppose to happen 500 times.

I think I know what I have to do but I have no idea how to code it in unix. I think an .sh file is needed? But i'm not very sure.

  1. start with nc cs2107.spro.ink 9000

  2. search for the hex string from the nc output

  3. decode it and calculate the md5 hash

  4. finally, send it back


i know i'm able to save the output of nc by doing so

nc cs2107.spro.ink 9000 > somefile.txt  

how do i then search specifically for the hex string?
how do i decode the hex string?
and finally how to i return the result back into the terminal?

edit2: so i'm allowed to use python to do this assignment. some hints were to either use the subprocess module or the sockets module. tried doing it with subprocess as i read that


keeps the command running in the background

i'm currently stuck with this

import subprocess
p = subprocess.Popen('nc cs2107.spro.ink 9000', shell = True, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
a = subprocess.Popen('grep ^[0-9a-f] | xxd -r | md5sum | awk "{print $1}" ', stdin = p.stdout, stdout = subprocess.PIPE)

now i'm stuck trying to pipe a back into p

  • 3
    That looks like a pretty interesting assignment. There are dozens of ways that this could be done. Do, you have a specific question, or are you just hoping that a completed assignment will spring forth from this post? – Stephen Rauch Apr 1 '17 at 6:17
  • I agree with @StephenRauch on all points. But I will try to be slightly more helpful ;). You will need to separate your problem into smaller sub-problems, then find solutions for each of them and "glue" them together in the end. For a start, following keywords will probably help to find a starting point: "Shell pipeline", "Text filtering grep sed". From there you should be able to pickup further helpful search terms/methods, that will in the end lead to one of the aforementioned dozen ways. If you can then formulate a specific question not already answered here, post it. – Alex Stragies Apr 1 '17 at 8:44
  • What programming languages are you familiar with, or what tools are you supposed to use for the assignment? This is not so easy to do with basic shell scripting and netcat, not a beginner task. It would be a lot easier with a language that lets you interact directly with the network, without having to call netcat (could be bash and /dev/tcp). – Gilles 'SO- stop being evil' Apr 1 '17 at 14:12
  • i have basic knowledge of python, took a module in my school, but its almost the same as taking the basic python course on codeacademy, just a little more in depth. this assignment requires me to do it on ubuntu. – Helpisneeded Apr 2 '17 at 6:16
  • Does the assignment stipulate any other conditions/restrictions? For example, is using python acceptable, and/or can other packages/utilities be used? Or does it have to be solved using standard/built-in/base features? (Although I'm not sure whether there is even a definition for a "base" Ubuntu system). – ovirt Apr 3 '17 at 5:22

You are right that we are expected to do socket programming in order to communicate with the server. I tried Python before but ended up with using plain old Bash scripts instead.

For starters, you can take a look at how to initialize a socket with Bash by following the tutorial here: http://hacktux.com/bash/socket

Step 1: Initialize socket

You can make use of Bash exec and unix redirection <> to create a socket on the pseudo-path. The path syntax is /dev/<protocol>/<host>/<port>

exec 3<>/dev/tcp/cs2107.spro.ink/9000;

Step 2: Reading server output

We need to read the encoded bytes from the server output. This can be done with cat command but it will not work as you expected. This is because the server is waiting for the input from client and there's no EOF character. (I'm guessing it's related to streaming data) In order to prevent this you need to avoid reading beyond the server output, that is, stop at line 7.

head -7 <&3

Step 3: Grepping & transformation of data

egrep -o '^[0-9a-f]{20}' | xxd -r -p | md5sum

After we get read in the chunk of information, we can make use of egrep to retrieve the hex bytes. Since the hexdump length is fixed, we can make use of regular expression to extract the required data.

Piping the hexdump to xxd -r -p will convert hexdump into binary and output them as plain text. md5sum will then do the dirty work and calculate the answer for us.

Step 4: Writing back to Server

awk '{print $1}' >&3;

In my case md5sum output the answer and extra character at the end of string, so I make use of awk to print the first column and write back to server.

Step 5: Repeat another 499 times

Similar to the steps 2-4, except that we need to modify step 2 because the server will only output "correct" in subsequent response instead of long introductory text. Modify the head to read in 2 lines. We also need to do this for another 499 times:

for i in {1..499}
    head -2 <&3 | egrep -o '^[0-9a-f]{20}' | xxd -r -p | md5sum | awk '{print $1}' >&3;

Step 5: Capture the flag

Once all 500 response has been completed, the flag can be retrieved by catting the socket.

cat <&3

P/S I was taking the same module and doing the same assignment as OP so I guess I'll be sharing my answer here. Just happened to found out this post after the module ended.


Here's a partial solution, I got frustrated with making the netcat script input happen.

  1. How do I then search specifically for the hex string?

Here's one simple way:

$ nc -d cs2107.spro.ink 9000 > file

Welcome to the Proof of Work Challenge.
Rules: I will provide hex encoded byte strings to you.
Your task is to decode it and calculate the md5 hash.
Return the md5 hash in hex encoded format back to me.
You will need to do this 500 times.
Ready? Go!

Here I originally had a sed command that selected the specific line #7, but after you solve the first hex encoded byte string, the next output is given immediately. So I just changed this to a regular expression that matches hex characters:

$ cat file.txt | grep '^[a-f0-9]'

  1. How do I decode the hex string?

$ cat file.txt | grep '^[a-f0-9]' | xxd -r -p | md5


So I got verification that this works manually:

Welcome to the Proof of Work Challenge.
Rules: I will provide hex encoded byte strings to you.
Your task is to decode it and calculate the md5 hash.
Return the md5 hash in hex encoded format back to me.
You will need to do this 500 times.
Ready? Go!
MD5: 92b34e4055a92b9ec32b15f89cc22389
  1. And finally how to I return the result back into the terminal?

This last question is really the biggest issue. I wasn't able to get it to work. I wrote several scripts and none of them solved the problem. The issue as Gilles mentioned, is that you need to keep the connection open long enough to read the data, and feed it back to the still-open program. This interferes with the normal approach of using the piped commands on the shell like the ones I posted above.

This is the pseudocode that I keep failing to implement correctly in a shell script:

  1. for loop, iterate 500 times. WORKS.
  2. keep connection open to: cs2107.spro.ink 9000. IN PROGRESS.
  3. read bits sent by server. WORKS.
  4. decode hex string. WORKS.
  5. calculate md5 hash. WORKS.
  6. send input to open connection. IN PROGRESS.
  7. repeat steps 3-6.
  8. end.

Here are some resources I tried

  1. Using in/out named pipes for a TCP connection
  2. https://serverfault.com/questions/188936/writing-to-stdin-of-background-process/297095#297095
  3. https://stackoverflow.com/questions/21130757/send-commands-to-socket-using-netcat
  4. https://superuser.com/questions/261900/how-can-i-pipe-commands-to-a-netcat-that-will-stay-alive
  5. https://stackoverflow.com/questions/2559076/how-do-i-redirect-output-to-a-variable-in-this-shell-function
  6. http://www.linuxquestions.org/questions/programming-9/dynamically-supply-input-to-netcat-in-a-script-793526/

I would be curious to know the solution if you find it out. Maybe I'll try again later.


Doing this with a shell that invokes nc is difficult because you need to keep a connection open and get data out of nc, process that data and feed it back into that same instance of nc. It can be done, but it's difficult.

It would be a lot easier to do if you can interact with the network and control the data flow in the same language. If you use one of the shells bash, ksh or ksh, then you can use their network capabilities: these shells can be TCP clients. For example, here's a script that reads lines from the server and echoes them one by one:

  while IFS= read -r line <&3; do
    echo "$line" >&3
} 3<>/dev/tcp/cs2107.spro.ink/9000

To decode the hex string, look up xxd. I'll let you work out the data processing logic.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.