CapsLock as modifier key for application keymaps

Question

I am developer and currently I am using Ubuntu Linux with PyCharm IDE for development of website. In order to improve typing performance I consider change CapsLock to Hyper_L and assign it to mod3 using xmodmap. After this I want to map mod3 for 'i' key to 'Up'. So far what I did is:

xmodmap -e "keysym Caps_Lock = Hyper_L"

output is like this:

shift       Shift_L (0x32),  Shift_R (0x3e)
lock        Hyper_L (0x42)
control     Control_L (0x25),  Control_R (0x69)
mod1        Alt_L (0x40),  Alt_R (0x6c),  Meta_L (0xcd)
mod2        Num_Lock (0x4d)
mod3      
mod4        Super_L (0x85),  Super_R (0x86),  Super_L (0xce),  Hyper_L (0xcf)
mod5        ISO_Level3_Shift (0x5c),  Mode_switch (0xcb)

then I remove Hyper_L from mod4

xmodmap -e "remove Mod4 = Hyper_L"

Afterwards I tried assign Hyper_L to mod3

xmodmap -e "add Mod3 = Hyper_L"

However I got error as following:

X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  118 (X_SetModifierMapping)
  Value in failed request:  0x17
  Serial number of failed request:  11
  Current serial number in output stream:  11

So, what was the problem here? How I can make it work? Does my approach is right? Or I should use different approach in order to use mod3 for custom keys?

P.S. I am using Microsoft Natural Ergonomic 7000 keyboard

Answer 1

There are a bajillion answers on the internet, most of them confusing.

The key is you need to map 'keycode 66' to your hyper key, remove the mapping of that key from other mod{1,2,4,5}'s (only if present), and then set mod3.

Open ~/.Xmodmap and put the following:

! Unmap capslock
clear Lock
keycode 66 = Hyper_L

! Leave mod4 as windows key _only_
remove mod4 = Hyper_L

! Set mod3 to capslock
add mod3 = Hyper_L

Run this

xmodmap ~/.Xmodmap

If something goes wrong, you can reset your keyboard with:

setxkbmap -layout us

Likely causes of something exploding are that Hyper_L is mapped someplace else. Fix this by running xmodmap without args to figure out what else is bound to Hyper_L. When you know what you need to remove, chuck a 'remove BINDING = Hyper_L' in your .Xmodmap file.

The .Xmodmap file will be loaded in when you open X11 sessions, you should have this mapping in place next time you login or reboot.

This configuration works for my ubuntu 14.04 setup. If you have a different *nix and this doesn't work, please post what you did to get around it here and I'll incorporate it into the answer.