Success with mint Cinnamon 18.3 and the newest amdgpu-pro driver (17.50)! This is with a Radeon Vega Frontier Edition card, though I have reason to believe this will work with other cards if you download the correct driver. OpenGL and OpenCL work and seem to be at least as fast as Windows (which is using a slightly older driver). Have not tested Vulkan or video acceleration.
The main trick is that xorg-server must be updated to xorg 1.19 or the modules won't load due to ABI mismatch. This may require you to enable the unstable packages repository:
apt install xserver-xorg-hwe-16.04
Download the driver from AMD. Go to http://support.amd.com/en-us/download and then use "manually select your driver". For me I had to choose "Professional Graphics", "Radeon Vega Frontier Edition", "Radeon Vega Frontier Edition", and "Linux", hit "Display Results". Point at the Download buttons to see what version you are getting, you want 17.50 or better for Ubuntu (note this is newer than the ones indicated by most help I could find on the web, the older ones work but they compile a kernel module and otherwise seem more complex, I believe 17.50 is the first "open source" one). You do not need the "preinstall" script. Hit the correct Download button.
Unpack (change the number to what you downloaded):
tar -Jxvf amdgpu-pro-17.50-511655.tar.xz
Fix the Ubuntu-specific script to work on Mint using some editor: nano amdgpu-pro-install. Replace
linuxmint), and in a new twist in 17.50, remove the
$APT (This appears to be for sending switches given to the script to apt, but it truncates the command line instead, I don't know enough bash to figure out why, and may be a bug in Mint's apt, or a bug in this script).
Run the install script like this:
./amdgpu-pro-install --opencl=rocm It says to use
legacy in place of
rocm for "pre Vega10". I wasted some time trying to figure out if my card is "Vega10", AMD manages to not tell me anywhere, but I suspect it is true of anything called "Vega".
Other installation instructions said to do
usermod -a -G video $LOGNAME. I don't know if it is necessary, you might want to try without it.
And then sudo reboot and enjoy!
If it screws up, look in
/var/log/Xorg.0.log. Search for "amd" just to see if it is trying to find it, and then look for "unloaded" to find failed modules. If it complains about "ABI major version" then you need to change which xorg you are using. If you can't get it to work you can run
amdgpu-pro-uninstall, note that for that to work you must edit then install script in-place as it copies it and runs it again to uninstall!