I don't want to change the picture (as would be the case with any vector based drawing program)
Which vector drawing program modifies the original bitmap? I just tried it with Inkscape, and immediately on File > Import, it asked if I wanted to link it or embed it.
If you tell it you want to link it, the resulting Inkscape
.svg file simply points at the bitmap file. No change made to the drawing affects the bitmap.
The only trick in going this way is making sure the SVG drawing continues to point at the separate bitmap file as you move the pair of files around.
Ideally, Inkscape would always use a relative path to refer to the linked-to bitmap. That way, as long as the pair of files stay in their same relative positions in the file system, the link won't break.
Unfortunately, the current behavior is buggy. You can end up with an absolute
file:/// URL instead.
If your system is affected by the bug, you can easily fix it with a text editor, since SVG is XML-based. The system I'm typing this on is affected by the bug, but Inkscape will still happily load a hand-hacked SVG file.
If instead you choose to embed the image, it copies the image data into the SVG file, encoding it but not actually changing the data. There are ways to extract the embedded image, but it's probably better to link instead.
Except for the relative path bugs, this behavior of Inkscape's is exactly what I expected going into this investigation. I suppose it is possible that there are other vector drawing programs that do actually modify the image, but that would be very odd. It is more likely that you find a program that doesn't know how to do relative path linking, but since Inkscape is available everywhere, I don't know that this really matters.
Another way to go is to use a bitmap based program that knows how to do layered compositions, like Gimp. Here, you would be doing something more like "embedding," but Gimp lets you lock a layer, so that it can't be changed. Then you can put your annotations on a layer above it.
The main problems with this option is that a Gimp image file is much larger than the original image, and few programs other than Gimp can open it, so if you want to send the original or annotated image to someone, you have to save a copy out of Gimp into a portable format like JPEG.
I like your idea of using a vector drawing program better.