[beta] amtm v1.6_beta, now with Disk formatting automated

[Zonkd]

It's not proper yet, hence the beta.
What works 99% is formatting an new FAT device to ext*. I struggle with re-formatting existing ext* devices here.

You layed the groundwork. The formatting code is a step by step adaption of your tutorial.
What I'm struggling with is that while reformatting, after the fdisk partition creation a ghost device is mounted, named in the format like /tmp/mnt/sda(1) that prevents running mke2fs. I have not been entirely successful to umount that device to make mke2fs work every time.

I think I once had the problem you’re describing and couldn’t format a newly partitioned drive. After several tries I realised I had 3 ghost devices and tested umount to find it wasn’t unmounting anything. I rebooted to clear them and started over without any problems. I couldn’t reproduce the issue and thought I must’ve made a typo while my brain was on autopilot. Now it seems like a bug or limitation in umount...

Are you able to reproduce this problem?

Do you have multiple USB disks attached at the same time or do you use multiple USB disks on the router in one boot session?

 

[thelonelycoder]

I think I once had the problem you’re describing and couldn’t format a newly partitioned drive. After several tries I realised I had 3 ghost devices and tested umount to find it wasn’t unmounting anything. I rebooted to clear them and started over without any problems. I couldn’t reproduce the issue and thought I must’ve made a typo while my brain was on autopilot. Now it seems like a bug or limitation in umount...

Are you able to reproduce this problem?

Do you have multiple USB disks attached at the same time or do you use multiple USB disks on the router in one boot session?

I only had one device attached, with one single partition on it. I can reproduce the ghost device any time I want.
I'm thinking of doing the fdisk part first, then reboot the router and set label and perhaps journalling at pre-mount.
But before I do that I will do some more tests.

 

[ColinTaylor]

I only had one device attached, with one single partition on it. I can reproduce the ghost device any time I want.
I'm thinking of doing the fdisk part first, then reboot the router and set label and perhaps journalling at pre-mount.
But before I do that I will do some more tests.

The problem is that when fdisk does its write it triggers a hotplug event. Unfortunately this can't be stopped because even if you kill the hotplug2 process the watchdog will just restart it.[See Update below]

hotplug2 will look at the partitions on the "new" disk and try and identify any filesystems. If it finds any it will mount them. In our case it's almost certain that the disk previously contained a valid filesystem starting at the beginning of partition 1. So even though we haven't issued our mke2fs yet it doesn't matter, because the filesystem is still intact from before and hotplug2 will mount it.

The only way I can think of stopping this is to wipe the beginning of the partition so that it no longer contains a valid filesystem for hotplug2 to mount. The first partition typically begins at sector 63, but just to be sure we should wipe the first cylinder which will also include the MBR. For example:

# dd if=/dev/zero of=/dev/sdb count=16065 bs=512

This works in my testing, but usually doesn't stop hotplug from restarting all the USB apps .

Jan 14 21:02:22 kernel: sd 1:0:0:0: [sdb] Assuming drive cache: write through
Jan 14 21:02:22 kernel:  sdb: sdb1
Jan 14 21:02:22 hotplug[20437]: USB /dev/sdb1(unknown) failed to mount at the first try!

P.S. Also bear in mind that it is quite common for flash drives not to contain a partition table at all. In which case the entire drive is just a single filesystem (i.e. exFAT) as /dev/sdb rather than /dev/sdb1.

Update 24/4/2019: Although the hotplug2 process cannot be permanently killed it can be reconfigured to do nothing. Immediately prior to running the fdisk command do this:

# rm /etc/hotplug2.rules
# killall hotplug2

After the fdisk command has finished you can reinstate the operation of hotplug2 as follows:

# ln -sf /rom/etc/hotplug2.rules /etc/hotplug2.rules
# killall hotplug2

Note that it will still be necessary to wipe the disk as shown above to avoid other problems. But this at least will stop hotplug2 from interfering with fdisk and restarting the USB apps.

 

[faria]

feedback :
Router: rt-ac86u 384.8_2
Usb drive: Samsung MUF-BB 128 GB USB 3.0 MicroFit Flash Drive

Hi,
I formatted a drive with the with this script (ext4 with journaling ) with no problems, but i found the following in my logs:

Jan 14 23:16:23 kernel: EXT4-fs (sda): couldn't mount as ext2 due to feature incompatibilities
Jan 14 23:16:23 usb: USB /dev/sda(ext) failed to mount at the first try!
Jan 14 23:16:23 kernel: EXT4-fs (sda): couldn't mount as ext3 due to feature incompatibilities
Jan 14 23:16:23 usb: USB ext fs at /dev/sda mounted on /tmp/mnt/MYUSB.

Thanks for all your hard work.

 

[cmkelley]

feedback :
Router: rt-ac86u 384.8_2
Usb drive: Samsung MUF-BB 128 GB USB 3.0 MicroFit Flash Drive

Hi,
I formatted a drive with the with this script (ext4 with journaling ) with no problems, but i found the following in my logs:

Jan 14 23:16:23 kernel: EXT4-fs (sda): couldn't mount as ext2 due to feature incompatibilities
Jan 14 23:16:23 usb: USB /dev/sda(ext) failed to mount at the first try!
Jan 14 23:16:23 kernel: EXT4-fs (sda): couldn't mount as ext3 due to feature incompatibilities
Jan 14 23:16:23 usb: USB ext fs at /dev/sda mounted on /tmp/mnt/MYUSB.

Thanks for all your hard work.

That's normal. It has a set sequence of filesystem types it tries to mount until it finds one that works.

 

[faria]

That's normal. It has a set sequence of filesystem types it tries to mount until it finds one that works.

Thanks, i see it now, Doh!

 

[ColinTaylor]

P.S. Also bear in mind that it is quite common for flash drives not to contain a partition table at all. In which case the entire drive is just a single filesystem (i.e. exFAT) as /dev/sdb rather than /dev/sdb1.

OK this is an example of that situation:

I formatted a drive with the with this script (ext4 with journaling ) with no problems, but i found the following in my logs:

Jan 14 23:16:23 kernel: EXT4-fs (sda): couldn't mount as ext2 due to feature incompatibilities
Jan 14 23:16:23 usb: USB /dev/sda(ext) failed to mount at the first try!
Jan 14 23:16:23 kernel: EXT4-fs (sda): couldn't mount as ext3 due to feature incompatibilities
Jan 14 23:16:23 usb: USB ext fs at /dev/sda mounted on /tmp/mnt/MYUSB.

And here is me doing something similar with /dev/sdb. As you can see it actually crashes through the fdisk prompts with errors because it's trying to fdisk /dev/sd !!! Luckily the following mke2fs still works and the preceding dd just creates a new file called /dev/sd (rather than damaging anything).

 Do you want to set a device label?

 1. No label
 2. Yes, set label

 Enter selection [1-2] 2

 Enter label: test

 Formatting /tmp/mnt/sdb now

 At the eventual promt: Proceed anyway? (y,n)
 Answer with y.
1+0 records in
1+0 records out
Device contains neither a valid DOS partition table, nor Sun, SGI, OSF or GPT disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that the previous content
won't be recoverable.

Unknown value(s) for: cylinders (settable in the extra functions menu)

Command (m for help): Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that the previous content
won't be recoverable.

Unknown value(s) for: cylinders (settable in the extra functions menu)

Command (m for help): Unknown value(s) for: cylinders (settable in the extra functions menu)

Command (m for help):
Disk /dev/sd: 0 MB, 512 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks  Id System

Command (m for help): 1: unknown command
Command Action
a       toggle a bootable flag
b       edit bsd disklabel
c       toggle the dos compatibility flag
d       delete a partition
l       list known partition types
n       add a new partition
o       create a new empty DOS partition table
p       print the partition table
q       quit without saving changes
s       create a new empty Sun disklabel
t       change a partition's system id
u       change display/entry units
v       verify the partition table
w       write table to disk and exit
x       extra functionality (experts only)

Command (m for help): Command (m for help): Command (m for help): The partition table has been altered.
Calling ioctl() to re-read partition table
fdisk: WARNING: rereading partition table failed, kernel still uses old table: Inappropriate ioctl for device
mke2fs 1.42.13 (17-May-2015)
/dev/sdb contains a vfat file system
Proceed anyway? (y,n) y
Creating filesystem with 7577600 4k blocks and 1896832 inodes
Filesystem UUID: 0ccc20a4-67a9-48da-b0c9-3212dd8aa0b3
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

tune2fs 1.42.13 (17-May-2015)
___________________________________________

 Done formatting device.

 Your router will now reboot
 for the changes to take effect.

EDIT:
The script also makes no particular provision for devices that contain more than one partition. So for example, I have a device with 3 partitions (sdb1, sdb2 and sdb3) which are all mounted. I run the script and choose sdb2 to be formatted. It attempts to unmount sdb2 but makes no attempt for sdb1 or sdb3. It then wipes out the partition table and creates a new one with just partition 1. It then formats sdb2 (even though the partition entry doesn't exist the old device mapping still does). The end result is that the new partition hasn't actually been formatted at all.

 

[skeal]

I have been playing with the formatting process for about a year and found the above will work every time if the volume is fat16 or fat32. Maybe quickly format to fat and then proceed with the ext* formatting process. Just a thought...

 

[thelonelycoder]

amtm 1.6_beta2 is now available

What's new in 1.6_beta2
- Corrected logical flow in df
- Disk (re)formatting works for all available options reliable now
- Reliable zeroing and unmount of involved disk/partition(s). Thanks again @ColinTaylor for your code improvements and insight.
- df creates now a log file, view it with dfl
- General code improvement throughout amtm

How to update from 1.6_beta to 1.6_beta2
Use u to update to this latest beta version

To install and participate in the amtm beta testing, read post #1 of this thread.

 

[ColinTaylor]

Hi @thelonelycoder , there's a bug in the check for the total size of the disk. Here's a patch:

# diff amtm amtm.colin
--- amtm
+++ amtm.colin
@@ -409,10 +409,10 @@

                eval mtddev="\$mounts$device"

-               devtf=$(df -h | grep "$mtddev" | awk '{print $1}')
-               devtfsize=$(fdisk -l | grep -w "${mtddev%?}" | awk '{print $5}')
+               devtf=$(df $mtddev | xargs | awk '{print $8}')
+               devtfsize=$(fdisk -l | grep -w "${devtf:0:8}" | awk '{print $5}')

-               if [ "$devtfsize" ] && [ "$devtfsize" -gt "2000000000000" ]; then
+               if [ "$devtfsize" ] && [ $devtfsize -gt 2000000000000 ]; then
                        show_amtm " Device is over 2TB and cannot be\\n formatted by $thisScript."
                fi
        }

 

[ColinTaylor]

@thelonelycoder I've added one more changed line to my previous post. It fixes an edge case where the name of the selected filesystem is a substring of another filesystem. For example;

 Select your device to format

  Again, this will erase all data and
  partitions on the selected device!

 1.  /tmp/mnt/VERBATIM
 2.  /tmp/mnt/aaaa
 3.  /tmp/mnt/a
 4.  /tmp/mnt/aa
 5.  /tmp/mnt/aaa

 Select device [1-5 e=Exit] 3

 

[ColinTaylor]

I think there's an unexpected logic bug in the device selection process.

After you select a device you can then enter 2 to "Return to device selection". Now when you select the device it doesn't ask you that question again but goes straight through to the "Select new filesystem" dialog.

 Select your device to format

  Again, this will erase all data and
  partitions on the selected device!

 1.  /tmp/mnt/VERBATIM
 2.  /tmp/mnt/aaaa
 3.  /tmp/mnt/a
 4.  /tmp/mnt/aa
 5.  /tmp/mnt/aaa

 Select device [1-5 e=Exit] 3
___________________________________________

 You selected the following device:

  /tmp/mnt/a

 This will delete ALL of the following:

 /dev/sdb3 mounted as /tmp/mnt/aaaa
 /dev/sdb4 mounted as /tmp/mnt/a
 /dev/sdb2 mounted as /tmp/mnt/aa
 /dev/sdb1 mounted as /tmp/mnt/aaa

 1. Continue
 2. Return to device selection

 Enter selection [1-2 e=Exit] 2
___________________________________________

 Select your device to format

  Again, this will erase all data and
  partitions on the selected device!

 1.  /tmp/mnt/VERBATIM
 2.  /tmp/mnt/aaaa
 3.  /tmp/mnt/a
 4.  /tmp/mnt/aa
 5.  /tmp/mnt/aaa

 Select device [1-5 e=Exit] 4
___________________________________________

 Select new file system

 2. ext2
 3. ext3
 4. ext4

 Set file system [2-4]

 

[thelonelycoder]

I think there's an unexpected logic bug in the device selection process.

After you select a device you can then enter 2 to "Return to device selection". Now when you select the device it doesn't ask you that question again but goes straight through to the "Select new filesystem" dialog.

Thanks, I've uploaded the fixed and patched file.
Use u to update, no version change.

 

[JacquesR]

The "remove" means it's installed at /jffs/scripts/disk-check. Looks like /jffs/amtm-disk-check.log went out for a long walk. It will return when next plugging in a device or rebooting the router. The disk checker is only run just before a device is mounted.
If not, just remove dc and re-install it.

Strange - DCL is working now, but returns "Unknown filesystem type on /dev/sda - skipping check." This is an 86U, with a 2Gb Sandisk, that I formatted at journalled Ext4. But everything works fine - AMTM, Skynet and Diversion, and all the expected files are present on the Sandisk.

With everything working, I'm reluctant to reformat again, but in case that DCL message is something that I should attend to, or something that indicates a code tweak you need to make, I thought I'd report it. Thanks for the feature-addition, in any event!

 

[thelonelycoder]

Strange - DCL is working now, but returns "Unknown filesystem type on /dev/sda - skipping check." This is an 86U, with a 2Gb Sandisk, that I formatted at journalled Ext4. But everything works fine - AMTM, Skynet and Diversion, and all the expected files are present on the Sandisk.

With everything working, I'm reluctant to reformat again, but in case that DCL message is something that I should attend to, or something that indicates a code tweak you need to make, I thought I'd report it. Thanks for the feature-addition, in any event!

You did that with beta1, right? Can you post output of

fdisk -l /dev/sda

 

[JacquesR]

You did that with beta1, right? Can you post output of

fdisk -l /dev/sda

Formatted using beta1, yes (though running beta2 now). Output of fdisk:

Disk /dev/sda: 4004 MB, 4004511744 bytes
124 heads, 62 sectors/track, 1017 cylinders
Units = cylinders of 7688 * 512 = 3936256 bytes

Disk /dev/sda doesn't contain a valid partition table

So I guess I should probably reformat, though it's confusing to me that everything works without a partition table!

 

[thelonelycoder]

Formatted using beta1, yes (though running beta2 now). Output of fdisk:

Disk /dev/sda: 4004 MB, 4004511744 bytes
124 heads, 62 sectors/track, 1017 cylinders
Units = cylinders of 7688 * 512 = 3936256 bytes

Disk /dev/sda doesn't contain a valid partition table

So I guess I should probably reformat, though it's confusing to me that everything works without a partition table!

There's likely an non-destroying a fix for that. Or do you have a Linux system to do the formatting on?
If so, copy everything over to a safe place and reformat in Linux, then paste it back onto the device.

 

[JacquesR]

There's likely an non-destroying a fix for that. Or do you have a Linux system to do the formatting on?
If so, copy everything over to a safe place and reformat in Linux, then paste it back onto the device.

Thanks. Just MacOS and Win10 available, at least for today, so I'll try in the week from a Linux system, unless I get impatient today (it's not a big deal to reinstall, at least after backing up whitelists etc.).

 

[ColinTaylor]

So I guess I should probably reformat, though it's confusing to me that everything works without a partition table!

My guess is you had the situation described in post #27, where the utility appeared to work but didn't completely. The current version fixes that problem. So if you backup your data and use the new version of amtm to format the device it should work properly.

 

[thelonelycoder]

Thanks. Just MacOS and Win10 available, at least for today, so I'll try in the week from a Linux system, unless I get impatient today (it's not a big deal to reinstall, at least after backing up whitelists etc.).

The missing partition table is not a problem, it'll work without it. But the disk checker does not recognize it, unfortunately.