/etc/ssh/sshd for Computer B set:
From Computer A:
$ ssh -R 2222:localhost:22 ip.of.computer.b
From Computer B:
$ ssh localhost -p 2222
Note that 2222 is an arbitrary high-port number I picked. That port on Computer B will then be tunneled back through the SSH connection initialized on Computer A to port 22. If you have multiple machines you should use a different port for each machine.
For your use case you will probably want to run this from a script so that you can make it a daemon and periodically try to re-connect if the link is dropped. You will probably want a special account with a shell of just
/bin/true on Computer B to handle the incoming connections. You can then setup either a single key or multiple keys for each machine that are allowed to "call home".
On Computer A you might find the
-T options useful to disconnect it from local input (so it can run in the background), not try to run any remote command, just open the tunnel, and not create a tty.
Most normal methods of spawning a daemon don't work very well with setting up a network tunnel like this. A problem in network connectivity would make it try to beat the wall down to get through. A simple loop with a sleep to wait should do the trick. Ten minutes is a nice number because it doesn't flood the network and log files with attemps if there is a problem (like Computer B being offline) but it still gets you back in reasonably quick if the connection is droped.
while true; do
ssh -nNT -R 2222:localhost:22 ip.of.computer.b
A script like that can be run launched on boot
/etc/rc.local. Your first change to log into the machine will start about ten minutes after the Computer A boots.