Diagnosing where in the 20+ hops between you and your VoIP partner you're seeing the loss is really difficult because you can't talk to the intermediate devices.
You are testing it in the most direct way, by using your VoIP application, and you've proven that someone is dropping stuff somewhere.
In my experience (debugging carrier networks from the inside) it is most often the case that your ISP has over-subscribed their upstream connection, but they won't tell you if they are seeing loss there, nor can you do much to determine if this is true.
Try changing your VoIP codex to something that uses less bits (hence reducing quality). If your application can use a different IP port number, try that as telcos sometimes throttle VoIP traffic because it competes with their voice services.