If I understand you correctly, you want to replace the inode number with the block address. That means (1) one inode per block, which wastes a lot of space (the inode isn't that large), and (2) it's not that different from using an inode number: An inode has a fixed size, so a block contains a known number
n of inodes. So if you devide the inode number by
n (which ideally is a power of two, so it's just a shift), the quotient is the block number of the inode (plus the disk address where the inode table starts), and the remainder is the index of the inode inside that block.
To understand the rationale behind the inode table, think about what data is stored in the inode table: It's attributes like owner, group, permissions and timestamps, and indices and indirect indices of the data blocks. You have to store those somewhere, and you can't store them together with the file data.
So if you want to design your own filesystem, the first question you have to answer is "how do I identify the data blocks that belong to a file?" and the second question is "where do I store attributes like ownership, permissions and timestamps?". And yes, you can use different schemes for that than inodes.
why not just use its address, like we do with main memory and objects therein?
As I wrote, basically you have the block address - you'll just have to divide first, and add an offset. If you add the offset to every inode on principle, the "inode number" will be much larger, and you'll have a constant value in the high bits that's repeated in every inode number. This in turn will make each directory entry larger.
Don't forget that the unix filesystem was invented when harddisk sizes where around 20 Mbytes or so. You don't want to waste space, so you pack everything densely, and you avoid redundancy. Adding an offset every time you access an inode is cheap. Storing this offset as part of every "inode number" reference is expensive.
And the interesting thing is that even though the inode scheme it was invented for small harddisks in today's terms, it scales well, and even on harddisks in the terabyte range in "just works".