Are you sure what you want is happening?
When you run
ls /directory | grep '[^term]' you are essentially grepping for not the letters t e r m. This means if a file has other letters in its name it will still appear in the output of
ls. Take the following directory for instance:
alpha brave bravo charlie delta
Now if I run
ls |grep '^[brav]' I get the following:
$ ls |grep '^[brav]'
As you can see, not only did I get
bravo I also got
alpha because the character class
 will get any letter from that list.
Consequently, if I run
ls |grep '[^brav]' I will get all the files that do not contain the characters b r a v anywhere in the name.
$ ls |grep '[^brav]'
If you notice it included the entire directory listing because all the files had at least one letter that was not included in the character class.
So as Kanvuanza said, to grep for the inverse of "term" as opposed to the characters
t e r m you should do it using
$ ls |grep -v 'brav'
Also if you don't want the files that have any characters in the class use
grep -v '[term]'. That will keep any files from showing up that have any of those characters. (Kanvuanza's answer)
$ ls |grep -v '[brav]'
As you can see there were no files listed because all the files in this directory included at least one letter from that class.
I wanted to add that using PCRE it is possible to use just regex to filter out using negate expressions. To do this you would use something known as a negative look-ahead regex:
So using the example above, you can do something like this to get results you want without using
$ ls | grep -P '^(?!brav)'
To deconstruct that regex, it first matches on a start of a line
^ and then looks for strings that do not match
brav to follow afterwards. Only
delta match so those are the only ones that are printed.