12

My friend was having problems with a USB flash drive, and I suggested he do a low-level format. Then it occurred to me that I don't even know how to do that in Linux. So... how can I low-level format flash memory in Linux? My friend tried a "full format" on the drive in Windows and it failed.

12

"Low level formatting" was done on floppies, where you could write at different densities by choosing to organize the tracks and sectors differently. But this makes no sense for most modern media. Its notion of how to organize the data on the device is fixed and unchangeable. It doesn't make any sense at all for flash, which has discrete bits, rather than magnetic domains. Higher level formatting is possible, which is mkfs in unix-land.

  • So if the filesystem on my friend's flash drive is messed up, mkfs is his only option? What if that doesn't work? There's nothing else he can try? – bmaupin Jan 31 '12 at 17:40
  • @Bryan: well, he could also try fsck, the equivalent of dos's chkdsk. – wnoise Jan 31 '12 at 20:18
  • 3
    @wnoise if mkfs fails, fsck isn't going to help. – derobert Dec 11 '12 at 17:59
  • 2
    I test tons of flash USB memory at work and the quality of that is all over the map, so I wouldn't be surprised if your friend's drive is just broken and can't work anymore. These things can break in many different ways and there's not much you can do about it, other than buying memory from reputable sources and then maybe testing it with f3 to make sure it's not broken. – unfa Nov 27 '17 at 14:17
8

Low-level formatting means many different things to different people and on different contexts.

The original meaning was a step needed in the formatting of disks - disk drives need header, sync and other patterns written on the media before it can store data to it. In this way the head can detect when it is A) on a track and B) where it is on the track. Low-level formatting a floppy prepares the disk to be able to read and write blocks. Early MFM and RLL PC hard drives could be low-level formatted, often using a utility built into the hard drive controller's (an ISA card) ROM. Modern IDE and SATA hard drives are low-level formatted too, but only at the factory.

Various other meanings include writing zeros to all blocks, configuring the drive to disable "hidden" areas such as HPA and DCO and then zeroing all blocks, or other things more related to partitioning than formatting.

Raw flash needs a different initial preparatory step at the factory - each flash "eraseblock" (analogus to a "block" on disks) needs to be tested and marked as bad if it is indeed bad. Each "eraseblock" has an additional small "OOB" block that holds error correcting information - and this is where it is marked as bad. You do NOT want to repeat this step as the act of writing to a bad block could prevent you from setting that particular bit again that identifies it as bad.

But you are not dealing with raw flash. You are dealing with a USB flash drive. There is a controller chip in all flash drives that accepts USB commands from the host and talks to the raw flash inside on the host's behalf. Some of these controller chips can be configured to report part of the flash as a separate CD-ROM partition, or act like two separate USB storage devices. Depending on the make and model of the controller chip, you may be able to find a recovery or configuration utility (likely Windows only) that could reset this controller chip. You would begin by opening the flash drive, looking for the smaller of (likely) two chips that are on the small PCB, and doing some Googling. The make and model printed on the outside of the case is not likely to help you find who made the controller inside of it.

6

use dd command for this dd if=/dev/zero of=/dev/hda

This will destroy ALL data on the hard drive, all boot sector info and all data on all partitions. It will not however render the disk useless, you simply have a clean disk that only needs to have partitions created and a new boot sector installed, which will happen when you install any OS including Linux or Windows. This is a good way to clean up any bad partitions, viruses, botched installs or data that you don't want to be seen.

  • 4
    You do not want to use /dev/zero to erase a flash memory device. See the entry on my blog: fakkelbrigade.eu/chris/blog/2012/01/… – Chris Down Jan 27 '12 at 19:46
  • 1
    It's not extraordinarily wrong to write /dev/zero to a flash device. The only advantage that writing 1s to the device gives is that you might get slightly faster write times the next time you write to the disk since it doesn't have to be erased first - and that depends on the flash translation layer being smart. – Shawn J. Goff Jan 28 '12 at 1:22
  • 2
    @ChrisDown, Not sure how it compares in performance, but you could use badblocks -s -w -t 0xff /dev/?d??. You get all 1's written, and tested too. – Zoredache Jan 28 '12 at 1:36
  • 2
    @ChrisDown your blog entry has gone 404-compliant. – derobert Dec 11 '12 at 18:01
  • 1
    If the flash internally erases to all ones, then the translation layer should be inverting the bits since it is customary to format disks with all zeroes. – psusi Dec 12 '12 at 1:56
5

There is no way to do a low-level format on most flash devices, since they have an additional translation layer from USB/ATA/SD/etc. to MTD which obscures the low-level MTD devices (which can be low-level formatted if gotten to directly [which you can't]).

4

mkfs.vfat /dev/hda1 will do the equivalent of the MSDOS "format" command.

2

I've come across devices in the past (although not for a while) which implement the SCSI FORMAT UNIT command - from Linux you can use:

sg_format --format /dev/sdX

to reset these devices to a factory state (USB storage spec is derived from the older SCSI standards).

I believe some newer USB 3 devices also implement ATA security commands, so you could use https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase which is likely to have the effect of resetting the flash translation layer to its factory state (as it typically does with SSDs).

1

There is such a thing as a low-level format for Flash memory, but it's done in factories with vendor-specific software.

To do that - you need to talk directly to the memory controller chip. Unfrotunately I couldn't find any programs that can do that on Linux.

There are manufacturer tools that perform low-level formatting and can write some settings (like LED behaviour, making the drive read-only) and data (like manufacturer name, model name, serial number) to the memory controller chipset, but these tools usually are made only for Windows, and are often in Chinese.

One such tool is called "FC MP Tools". As far as I can tell, these are designed by Alcor Micro, and manufactured by FirstChip (FC). Apparently "MP" stands for "Mass Production".

I'm using the version I have found here: http://down.upantool.com/file/software/mass/FristChip/2018/iTe_MpTools_20171130.zip

Other links you can follow: https://repusb.cubava.cu/?page_id=2052 https://www.rmprepusb.com/tutorials/repair-your-usb-flash-drive http://reboot.pro/topic/20865-alcor-micro-usb-pen-drive-repair/ http://reboot.pro/topic/19901-no-alcor-mptool-is-recognizing-my-usb/ https://www.elektroda.pl/rtvforum/topic3145335.html http://flashboot.ru/files/file/30/ http://www.flashdrive-repair.com/2014/05/download-fc-mptool-v402-for-fixing.html https://www.cdrinfo.pl/download/356133849

It runs in Wine, but will not detect the drive chipsets (it probably needs direct access to the USB controller to do it's thing).

Also - for this to work you need to have a flash drive with a specific Alcor chipset that is supported. Again - each vendor has his own software that talk to the controller chips via USB.

I have this:

enter image description here

Note the chipset part number on the microcontroller: FC1178 BC1 The FC MP Tools program will identify this as a 1178BC chipset:

enter image description here

And the software I linked works with it under Windows XP and 10. However - it will crash if you switch the program to English language! Set your preferences in English, then restart the program and don't change the language - it should work. Maybe it'll be fixed in a newer version.

Anyway - different vendors have different tools, it's a rabbit hole, there's no standard here AFAIK, and (at least this tool) will only work on Windows unfortunately.

From what I can tell, this will test the memory chip and write a bad sector map to the controller chip so it can present only good memory to the OS. This way even partially damaged chips can be used and sold. These are probably sorted in the factory by quality and low-level formatted to different capacities. This is called binning.

Maybe if someone can gather a lot of these tools, reverse engineer them and make a universal open-source tool for Linux we can do it - othwerwise I don't see that coming.

1

Although low level formatting the way it was done on hard disk drives and floppy drives do not make sense for a flash drive, there does exist manufacturer specific utilities to "low level format" flash drives: This is an example: Alcor low level format utility

I fell on this question while looking for such utility that would work on Linux. However, it seems that flash drive manufacturers do not care about making us happy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.