The short answer is no.
The Unix security model is fundamentally designed to isolate one user's data from other users. It is not designed to confine applications. All the applications that you run have the same privileges. Your file manager can obviously read your files (that's the whole point) and it has network access too (e.g. to mount remote shared drives). Other applications perforce have the same privileges.
Most of the time, this porosity is desired. I want to save a file from my email client and open it in my word processor. I want to copy some data from my word processor to my web browser via the clipboard. I want my email client and web browser to open all kinds of external programs to view files that they don't understand natively. I want my desktop macro program to be able to capture all keystrokes and inject keystrokes in applications. I want my backup software to be able to read and write to all my files.
Modern Linux systems have additional security mechanisms that provide a limited form of security restrictions on applications. But they are mostly designed to confine system services that don't interact with many other local applications.
If you want to isolate your Bitcoin wallet from your normal activities, you need to sacrifice some convenience for security. The most basic step is to store your Bitcoin wallet under a different user account. That way, applications that you use only with your normal account would be unable to touch it.
Any application that you use with the Bitcoin account is part of your trusted base — the part of the system that you need to trust since it is able to breach your security. This includes the kernel, a number of system daemons and programs, whatever shell and file manager you run as the Bitcoin user, whatever method you use to log into the Bitcoin account, all Bitcoin software, etc. Of course, the downside of that is that it'll make using your wallet harder. If you isolate your wallet from the context where you use it, that makes transfers more difficult.
An advantage of open-source software is that it makes a backdoor that accesses your Bitcoin wallet somewhat hard to hide. You may not feel the need to inspect the software personally because overall a lot of people dig into the source code for this and that. That's in addition to the network traffic which you or others might detect. If there was a backdoor, there's a good chance that it would be detected.