I have a system that runs Linux. It's a NAS, and does not have any monitor, keyboard, or serial port. It does have a network port. I'm not happy with the software running on it, and am trying to get a different distro to run.
What I have
On the existing system, I can manage to get a new kernel and initrd booted, by using the web interface to upgrade the ROM, but that upgrade is a specially crafted image, that does nothing more than unpack a kernel, initrd,
kexec, and runs
kexec with the required arguments to boot the new kernel.
The initrd sets up a network connection, starts an SSH server (dropbear), and waits for it to end. It then runs a different script. Using this, I can do some testing: I can boot into that kernel/initrd, log in via SSH, customise the stage-2 script, kill dropbear, and hope for the best.
Using this approach, I have managed to install a functional OS on the hard disk. (For now, it's NixOS, if it matters, but I may change that in the future. My question isn't about any specific distro though.) I have intentionally not made it bootable. I wish to keep the flash memory as it is, so that aside from the data on the hard disk, the NAS remains "official". I have, however, obtained the distro's own kernel and initrd, that I would want to put in an upgrade image.
Using that kernel and initrd, the system does not start.
I have put together a shared initrd, containing the distro's setup as well as my own, which continues to boot into dropbear. In an SSH shell, I then attempt to run the distro's init scripts. This fails, however, because it relies on being run as PID 1.
I then attempted to make PID 1 take arbitrary commands: I made it run scripts from a pipe, and wrote to that pipe from my remote shell, intending to inspect manually whether the command had the desired effect. However, that doesn't work: the init-shell (PID 1) sees EOF after the very first command, and immediately exits. Hello kernel panic.
I also attempted to test what would happen if I made it not care whether it was being run as PID 1, by passing systemd's
--system option, and then run the distro's init script from my shell. In that form, I was unable to reproduce the problem: it just worked.
My question: What now?
At this point I'm looking back at the pipe approach for an alternative, something that actually works to run more than one command. Preferably something that also lets me see the commands' output.
Basically: I want to know how I can remotely see the output of the boot programs that run from PID 1, that cannot run in an SSH session, without actually attaching a monitor or serial cable.
Answers that take a completely different approach are also welcome, but please keep in mind the system restrictions I'm dealing with. I don't just not have a monitor or serial cable, I don't even have a VGA or serial port. I have one USB port that I could plug a keyboard into, if needed, but I am of course not able to see anything I type.