I was posting big comments, and thought it might be best to post things as an answer, because the box is bigger and I can waffle all I want. :) I hope you'll excuse my insane ravings.
Also, I'll warn you now: I'm a complete bastard, and I'll be clipping your wings a bit.
The first thing you need to decide on is: how good are your programming skills? You can't write your own OS without good programming skills and a good understanding of hardware.
The second thing you need to decide is:
- Are you making a Linux distribution? (e.g. Debian, Ubuntu, Red Hat)
- Are you making a Linux-based operating system with a different runtime? (e.g. Android, Maemo)
- Are you writing an operating system from scratch? (e.g. VMS, CP/M, RT-11, Amoeba, Plan/9)
Before you even answer these questions, you should double check to make sure you've got your terms down, so you know what you're making.
1. A Linux Distribution (e.g. Debian)
If all you want to make is Yet Another Linux Distribution, compiling the kernel is the least of your worries. The kernel practically compiles itself, given enough time and a big enough computer.
Here is an article that discusses some of the aspects of making a Linux distributions.
Why this is harder than you might think: the more functionality you want to add to your distribution, the harder it'll get to get your distribution to build properly. You'll have to choose whether to re-invent the wheel (make your own package manager), or use an existing wheel (RPM? APT? Ports?). Either case, you'll be doing months of annoying work reconciling mutually incompatible packages, compiling, verifying things work using different computers with different CPUs (not just Intel or AMD), et cetera.
Depending on the scope, you may end up having to redo the work of thousands of people on your own. Would you learn a lot from it? Absolutely! But you'd also learn a lot by running an existing distribution of Linux, and occasionally compiling some packages on your own to see how things work. Why have to build libc and hundreds of other libraries if all you want to see is how an X server is built?
Think of it this way: if you want to learn how to assemble a PC, would you start with a bucket of transistors and a soldering iron, or get PC components and some documentation?
2. A Linux-based OS (e.g. Android)
There are quite a few Linux-based operating systems with runtimes fairly-to-drastically different from POSIX. Maemo, Meego, Android and other (most often embedded) operating systems come to mind.
The advantage of this is that you don't need to write your own kernel. The disadvantage of this is that you not only need to design your own distribution, but you'll also have to make most of the packages yourself. You thought compiling gcc and libc was a pain? Try writing your own ANSI-C-compatible C compiler and a full software stack from the ground up. Man years of writing annoying little functions (probably in Assembly) that provide tiny, tiny amounts of satisfaction. You'd also need to learn most of the internals of the Linux kernel, and how it talks to userspace, because that interface is what you'll be building on. You should know most syscall numbers by heart by the end of this. :)
Would you learn lots? Absolutely! More than you ever thought possible. Will it be frustrating and annoying? Oh yes. There's a reason Android wasn't built by a lone guy with a laptop in his spare time. Even gcc wasn't built that way.
3. An Operating System from scratch
This is where you start with an Assembler and C compiler and write your own kernel — if your OS even has a kernel. It's not universally the case. Download all the datasheets you can find for your hardware, and off you go. Once you have a basic kernel running, you can see about a runtime. You could write your own runtime, or you could make sure your kernel is partially or fully POSIX compatible, and then you'll have a full GNU runtime at your disposal. You'll ‘just’ have to make the first layer of libraries and get gcc running on your new OS.
Is this rewarding? Amazingly so. Will it be hard? If the names Donald Knuth, Andrew Tanenbaum mean nothing to you, you should probably reconsider. There'll be lots of studying theoretical computer science. There's an incredible amount of theory that goes into making an operating system truly from scratch. Unfortunately, there's also an amazing amount of tedium with modern systems.
It used to be, you just burned a ROM with a JMP instruction in the right place, socketed it, powered up, and you were done. Modern systems need serious amounts of work just to get a CPU core up and running, memory setup, an input device enumerated, and an output device started. There are entire books on each of these tasks (I have a book on the original VGA that's over 1,000 pages — growing up with 8-bit machines I'd assumed programming the 1987 VGA would have been simple, and I was seriously annoyed, but I learned a lot of stuff in the process!)
So, should you?
You absolutely should try. Don't have misconceptions about ‘succeeding’, unless your goals are set from the beginning. Maintaining an OS/distribution is an open-ended task that's usually done by large groups of people. But even attempting to do this will hone your skills and teach you new ones.
Btw, I'm speaking from (often painful) experience with respect to most of these things. I've (co-)written embedded low-level runtimes for a few CPUs (studies/work/fun), and I'm writing an toy OS for a toy CPU at the moment (that's all for fun).
