The problem here is with the format, the TAR (Tape ARchive) format is designed for sequential access, not random access. And gzip is a good complement to tar, since it is a stream based compression format, also not for random access.
So a high level tool that does not interact with the compressed blocks directly, will have to parse through the entire file every time it needs to read anything, first to get you the list of files, then perhaps the cache invalidates and it reads it again, and then for each file you copy off it might read through it again. You can make a tool that remembers the position of each file, and what blocks it needs to decompress to get it, but it seems that few have bothered with that.
If you want this to go faster, do a
tar tzf file.tar.gz > filelist, open that file list in vim, gedit or whatever, remove the lines of files you do not need, save, and then extract them with
tar xzf file.tar.gz -T filelist -C extracted/.
To get random access to a compressed file, you should use perhaps zip with posix extensions, rar, or as dru8274 suggested, squashfs, or even ZFS with compression turned on, or btrfs if btrfs has gotten compression to work at the time of reading.