34

I have a 22TB disk on /dev/sdb. How do I create a 22TB partition? Don't really care about the file system - ext4 or zfs is fine.

Running CentOS 6.2 - Partition will be used as a data dump. Only a single stream of data so being picky/choosy over what file system isn't really a concern right now. The disk is formed from 12x2TB nearline SAS drives and a Dell Perc controller.

I just want a 22TB partition.

Improve this question
2
  • 1
    a few more pieces of information would be helpful. what OS are you using? what kind of performance do you expect or need to get from it? what's the underlying hardware? it is already a raid set? does it have redundancy? do you need the extra data protection features of zfs?
    – Tim Kennedy
    Jan 14, 2012 at 1:46
  • 3
    Which tries have failed so far? fdisk, cfdisk?
    – Nils
    Jan 14, 2012 at 20:39

7 Answers 7

Reset to default
45

The simplest solution is to use GPT partitioning, a 64-bit version of Linux, and XFS:

  • GPT is necessary because the MS-DOS-style MBR partition table created by fdisk is limited to 2 TiB disks. So, you need to use parted or another GPT-aware partitioning program instead of fdisk. (gdisk, gparted, etc.)

  • A 64-bit kernel is necessary because 32-bit kernels limit you to filesystems smaller than you're asking for. You either hit a size limit based on 32-bit integers or end up not being able to address enough RAM to support the filesystem properly.

  • XFS is not the only solution, but in my opinion it is the easiest one for RHEL systems.

    You cannot use ext4 for this in RHEL 6. Although the filesystem was designed to support 1 EiB filesystems, there is an artificial 16 TiB volume size limit in the version of e2fsprogs included in RHEL 6 and its derivatives. Both Red Hat and CentOS call this out in their docs. (The ext4 16 TiB limit was raised considerably in RHEL 7 to 50 TiB.)

    ZFS may not be practical in your situation. Because of its several legal and technical restrictions, I can't outright recommend it unless you need something only ZFS gives you.

    Having ruled out your two chosen filesystems, I suggest XFS. It is the default filesystem in RHEL 7, it was available as a supported filesystem in all RHEL 6 versions, and was backported to the later RHEL 5 releases after RHEL 6 came out.

Here's the process:

  1. Check whether you have mkfs.xfs installed by running it without arguments. If it's not present, install the userland XFS tools:

    # yum install xfsprogs

    If that failed, it's probably because you're on an older OS that doesn't have this in its default package repository. You really should upgrade, but if that is impossible, you can get this from CentOSPlus or EPEL. You may also need to install the kmod_xfs package.

  2. Create the partition:

    Since you say your 22 TiB volume is on /dev/sdb, the commands for parted are:

    # parted /dev/sdb mklabel gpt
# parted -a optimal -- /dev/sdb mkpart primary xfs 1 -1

    That causes it to take over the entire volume with a single partition. Actually, it ignores the first 1 MiB of the volume, to achieve the 4 KiB alignment required to get the full performance from Advanced Format HDDs and SSDs.

    You could skip this step and format the entire volume with XFS. That is, you would use /dev/sdb in the example below instead of /dev/sdb1. This avoids the problem of sector alignment. In the case of a volume that only your Linux-based OS will see, there are no downsides worth speaking about, but I'd caution against doing this on a removable volume or on an internal volume in a multi-booting computer, since some OSes (Windows and macOS, for instance) will offer to format a partitionless hard drive for you every time it appears. Putting the filesystem on a partition solves this.

  3. Format the partition:

    # mkfs.xfs -L somelabel /dev/sdb1

  4. Add the /etc/fstab entry:

    LABEL=somelabel    /some/mount/point    xfs     defaults   0 0

  5. Mount up!

     # mount /some/mount/point

If you want to go down the LVM path, the above steps are basically just a more detailed version of the second set of commands in user bsd's answer below. You have to do his first set of commands before the ones above.

LVM offers certain advantages at a complexity cost. For instance, you can later "grow" an LVM volume group by adding more physical volumes to it, thereby making space to grow the logical volume ("partition" kinda, sorta), which in turn lets you grow the filesystem living on the logical volume. (See what I mean about complexity? :))

Improve this answer
3
  • 4
    The ZFS part is debatable. The Linux port has come a long way since 2010 and compared to XFS it has many advantages
    – TheLQ
    Jan 18, 2012 at 18:03
  • Regarding GPT/MBR, doesn't that only really apply if /boot is part of /? The MBR shouldn't care how big / is if it only has to mount a small /boot right? I could be wrong.
    – jonescb
    Jan 20, 2012 at 4:39
  • 1
    @jonescb: The location of /boot doesn't have any bearing on the limitations of MBR. If you need a partition over 2 TB, you can't use MBR partitioning. It is true, however, that it is possible to work around lack of BIOS support for booting from GPT by putting /boot on a smaller MBR-partitioned disk. Once the kernel is up, you don't have to worry about BIOS limitations because it knows how to interpret the GPT partition table. If your machine is EFI-based, you don't need to do this dance, because EFI understands GPT.
    – Warren Young
    Jan 20, 2012 at 6:32
16

Just as an alternative to the other suggestions.
You don't have to partition a disk at all.
You could simple create a Volume Group, with one or more Logical Volumes.

pvcreate /dev/sdb
vgcreate data /dev/sdb
lvcreate --name dump -L '100%VG' data

Now you have a logical volume that you can format with any filesystem type you wish.

mkfs.XXXX /dev/mapper/data-dump #<- XXXX can be ext4, xfs, btrfs, reiser
mount /dev/mapper/data-dump /mntpt
Improve this answer
5
  • LVM is basically an advanced form of partitioning. It is pointless to use LVM if you are just going to create a single LV using all of the space, so you may as well just mkfs directly on the whole disk device.
    – psusi
    Jan 17, 2012 at 18:42
  • 4
    no LV's add tools for snapshots of live data, resizable, multiple copies of metadata; far more flexible than a simple fs on a device.
    – bsd
    Jan 18, 2012 at 3:03
  • 1
    You don't need multiple copies of metadata when you have no metadata ( partition table ) in the first place. Snapshots require free space, as does adding/expanding volumes, hence why it is rendered pointless if you just create a single logical volume using all of the space right off the bat. If you want the features of LVM, then you should start with a smaller volume so you have plenty of free space to use later.
    – psusi
    Jan 18, 2012 at 16:39
  • I wasn't recommending LVM in this individual's case, simply listing it as an alternative to "other" partitioning, non-partitioning answers/solutions. It's up to her to read all the responses, follow up with her own research and then decide which course of action best suits her needs.
    – bsd
    Jan 18, 2012 at 17:57
  • 3
    One advantage of LVM even if you only have one LV is it lets you easily add storage later.
    – plugwash
    Dec 11, 2015 at 12:01
6

Question to the question: You asked 'how to partition 22TB disk' and then in the question again, you said, you just wanted a 22TB partition. So this is ambiguos in first place.

If you already have a single block device which can support 22TB of space on it, then you already posses whole 22TB partition. All you need is a filesystem on top of it, which will make the device mountable and usable for reading/writing by system processes. More ever, you need to have a Linux kernel running in 64-bit mode with a filesystem module/driver that supports and scales to 22TB of data growth, can handle the ins and outs of managing the data on the (single) block device with ease. Performance is altogether another dimension to it. In such case, I would opt to choose XFS as my filesystem, for the reason that it is a 64-bit filesystem and capable of handling filesystems as large as a million terabytes. It supports upto 9 EXABYTES.

2^63  = 9 x 1018 = 9 exabytes

For more details on XFS: http://oss.sgi.com/projects/xfs/

If you are looking for further partitioning the huge 22TB block device, then use gparted to split the device into usable partitions and then format them with the filesystems to make them mountable.

It seems that you have got hardware RAID controller, since you mention that you have got DELL perc RAID controller -- which will mean that, you have to tell which RAID configuration (precisely which RAID level are you using?) and in most cases, you are not going to get complete 22TB of space for use, I could be wrong though.

Improve this answer
5
  • I was going to suggest xfs as well, but xfs_check will take a large amount of memory, and a long time to run w/22G fs. Between physical and swap one would need at least 32G to check, that is if he ever cares to check the fs on a 'datadump' (whatever that is ;)
    – bsd
    Jan 14, 2012 at 11:43
  • @bdowning, you might want to correct it to 22T fs :)
    – Nikhil Mulley
    Jan 14, 2012 at 11:44
  • It is 22TB Raid 5. I have a lot of disks :)
    – LVLAaron
    Jan 14, 2012 at 13:20
  • That was a typo(brain fart). I have the same device, 10TB, with LSI Megaraid (same card as Dell Perc)
    – bsd
    Jan 14, 2012 at 13:27
  • @bdowning: xfs_check does indeed use a lot of memory, but the manual page(8) mentions: "Note that using xfs_check is NOT recommended. Please use xfs_repair -n instead, for better scalability and speed.".
    – Cristian Ciupitu
    Nov 29, 2012 at 6:10
4

You shouldn't need any partitioning when using ZFS, just create a ZFS pool on your 22 TB device and a file system in it if you don't want to use the default one and that's it. If for some reason, zpool doesn't support using the whole disk, first create an EFI label and a partition using the whole space available inside then use that partition to create the pool.

For several reasons, I wouldn't recommend using anything but ZFS for such a large file system. The most obvious is if you have a brutal power off (eg: kernel panic or power shortage), fsck can take a painful length of time to recover traditional file systems. ZFS on the other hand doesn't need fsck so will import the pool instantly.

Note that you'd better break the hardware RAID configuration and use the twelve devices as a JBOD to build a ZFS pool with taking advantage of its software RAID capabilities. If your goal is performance, you might mirror pairs of disks and if your goal is maximized space, you might use a RAIDZ, RAIDZ2 or RAIDZ3 configuration. Doing it will largely improve the reliability of your data and the fault tolerance of the solution.

Improve this answer
2
  • 1
    I know this is an old answer to an old question, but....if using ZFS, the best option is to destroy the RAID array, configure the RAID controller for JBOD so that linux sees each individual disk, and then create mirrored pairs (less capacity, great performance) or RAIDZ/Z2/Z3 (more capacity, lackluster performance) from the individual disks. You lose much of the benefit of ZFS if you layer it on top of existing RAID rather than let it handle the disks itself. The same applies to btrfs.
    – cas
    Jun 15, 2016 at 4:05
  • @cas You are definitely right, answer updated. I was focused to the OP question though "how to partition a 22TB disk" and not "what would you recommend to do with my disk configuration".
    – jlliagre
    Jun 15, 2016 at 5:45
3

I'm not sure this is currently possible using the standard partition table. In the standard partition table scheme, volumes are limited to 232 sectors. With 512 bytes per sector, you'd simply run out numbers to assign to sectors around 2TB.

However, you ought to be able to do this if you use a GUID Partition Table instead of a standard one. GUID partition tables allow for volumes to extend into the zettabyte range. Most Linux distros are bootable from a GUID volume, however no version of Windows (except for Windows 7 on EFI) currently is.

Some tools like fdisk can't work with GUID volumes, however other tools like GParted can. Once you create your GUID partition table, you ought to be able to create a volume using one of the several common filesystems which support a volume of that size (e.g. EXT4.)

Improve this answer
2
  • On a 32-bit system may be. I have been able to create a 3.5TB partition on a 64-bit Ubuntu server supported under version 8.04 and higher. Speaking of cyberciti.biz/tips/…
    – Karlson
    Jan 14, 2012 at 4:20
  • To my knowledge you may not even need a partition table to put a filesystem on a harddrive. Plus according to the OP it's not a boot drive just a large dump space.
    – Karlson
    Jan 14, 2012 at 5:47
3

For your partition table, as mentioned elsewhere, GPT is an excellent option, as it supports partitions up to 9.4 ZiB in size (9.4 × 1021 bytes), which is well beyond anything you'd need with 22 TiB.

For your filesystem, on Linux BTRFS is an excellent copy-on-write file-system:

  1. Its copy-on-write attribute means that no duplicate file is stored twice.
  2. It features on-the-fly compression so your data is run through LZO or GZip before being written to and read from disk, saving physical disk space.
  3. It supports redundancy in RAID-1, RAID-10, RAID-5, and RAID-6 configurations without any overhead.
  4. It also supports RAID-0, if speed is of the essence.
  5. It also features subvolumes, snapshots, and more.
  6. Almost all filesystem tasks are done online, so you won't usually ever have to unmount the filesystem to fix things.

It's similar to ZFS in features, but is a part of the mainline Linux kernel.

Improve this answer
1
1

If you are not looking for the redundancy or ability to back it up you can probably do:

mkfs -t ext4 /dev/sdb
Improve this answer
9
  • 4
    Have you tried it? I changed the mkfs to specify the filesystem
    – Karlson
    Jan 14, 2012 at 4:05
  • 1
    @AaronJAnderson an explanation would help.
    – n0pe
    Jan 14, 2012 at 5:26
  • That command will not work if the device is over 2TB
    – LVLAaron
    Jan 14, 2012 at 13:26
  • 1
    @AaronJAnderson I have created 3.5TB volume using reiserfs and ext3 so if the ext4 claims to have maximum size of the volume of 16 exabytes I don't see any reason that 22TB won't work.
    – Karlson
    Jan 14, 2012 at 20:04
  • 1
    the version of e2fsutils that ships w/ this version of linux (and most others at the moment do not support over 16TB)
    – LVLAaron
    Jan 17, 2012 at 8:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .