You are mistaken.
/var/log/dpkg.log lists all installs, upgrades, removes and purges on the system.
e.g. the package most recently removed on my debian sid system was:
# grep remove /var/log/dpkg.log | tail -1
2015-10-08 17:21:56 remove libsmokebase3v5:amd64 4:4.14.3-1 <none>
It does log a lot of information, which can be "noise" if you're just interested in finding out which packages got installed or removed recently and when. As always, filtering out "noise" from log files is an art. Start with something like:
grep -E ' (install|upgrade|remove|purge) ' /var/log/dpkg.log
Note the leading and trailing spaces around
(...)...that prevents the regexp from matching those words if they happen to be embedded in a package name.