Is there a limit to total files # in NTFS driver? (Merlin v.384.13/ AC68U)

[Hale_JP]

Here, "Samba bug with admin login", I came to conclusion, that all my problems with SAMBA are related to NTFS partition, or to how it is accessed by the router, to be precise.

I want to confirm, is there a limitation to total number of files in NTFS driver, used in Merlin firmware? At the moment I have v384.13 installed.

If yes, is there a workaround?

P.S. for those who always offer to get rid of router or NTFS partition: there are situations, when it is the most appropriate combination, like... you know, backup of windows machine! Who would think about that!? But windows understands only NTFS. And you can not restore a dead windows-based laptop by wifi in most cases, so you just unplug your usb-disk and restore directly.

 

[Adamm]

And you can not restore a dead windows-based laptop by wifi in most cases, so you just unplug your usb-disk and restore directly.

You can always image your hard drive and restore it on another device (a lot of backup software also have "live usb" support with wifi drivers for restoring). Due to the nature of NTFS and Linux you are never going to match the performance/compatibility of ext*

 

[bbunge]

Here, "Samba bug with admin login", I came to conclusion, that all my problems with SAMBA are related to NTFS partition, or to how it is accessed by the router, to be precise.

I want to confirm, is there a limitation to total number of files in NTFS driver, used in Merlin firmware? At the moment I have v384.13 installed.

If yes, is there a workaround?

P.S. for those who always offer to get rid of router or NTFS partition: there are situations, when it is the most appropriate combination, like... you know, backup of windows machine! Who would think about that!? But windows understands only NTFS. And you can not restore a dead windows-based laptop by wifi in most cases, so you just unplug your usb-disk and restore directly.

Yes, there is a limit. Somewhere over 4 billion files. However, your issue is likely with the router resources. Each Asus router has its external drive limitation which is published. A swap file or partition on a thumb drive may help extend the router memory. It may sound counter productive but using USB2 is recommended.
Knowing all these things from experience and trying drives formatted with ext3 and 4 as well as ntfs/far32 I spent the bucks on a Synology NAS. Family is much happier!

 

[Hale_JP]

@Adamm
You can always image your hard drive
1. Of course I can't . Nobody can do this "always". But a scheduled task can do it to available NAS, which is always available when you come home.
2. This is OFF TOPIC, that's why it was written in small letters, specifically to give a reasoned answer to unreasonable offtopic suggestions.

@bbunge
Yes, there is a limit. Somewhere over 4 billion files.
well, it is a limit of NTFS. It does not mean a driver cannot have its own limits, maybe just because of programmer's laziness, who did not expect more than 100,000 files on one partition, or maybe by performance reasons, like fetching all the tree into a fixed-size structure...
so, that is the question, does anybody know about documented, nondocumented, or "happened by mistake" limitations?
Because my latest experience tells that I can not create more files at router anymore, after filling the partition already with a huge number of files. The partition itself is not something special, just 1Tb primary MBR NTFS, filled by 60%.

>Synology NAS.
Whom did I write the first post for? Drop down your laptop, and you can drop your synology with it. Because it will not help you restoring the backup on replaced laptop without hard 肛門性交. And it also requires extra spending, and extra power cord with extra power socket.

Coming back to topic. I examined google, and found that original ASUS driver had problems with NTFS partitions larger than 1Tb.. and maybe it extends to ~900Gb partitions as well.
here
https://www.snbforums.com/threads/usb-hard-drive-and-87u.29696/#post-321148
it was suggested to replace the driver to ntfs-3g.
But the information is very outdated, the step-by-step walkthrough does not work. And I have no enough linux-box experience to reach the goal other way.

 

[ColinTaylor]

I want to confirm, is there a limitation to total number of files in NTFS driver, used in Merlin firmware?

I don't think anyone can answer that for you. The NTFS driver is closed source and supplied by Tuxera, the version has also been updated over the years. So to get a definitive answer you'd have to ask Tuxera.

But you're probably in the best position to answer your own question. You say that you can't add any more files after the 79,000+ you've already added. I'd say that's likely to be all the proof you need.

As hinted at by @bbunge it's possible that you're hitting some sort of memory limitation, which of course would vary from router to router. We already know that Asus routers don't have enough RAM to perform an ntfsck on partially filled NTFS drives. But in that case the problem is obvious from the out of memory errors in the syslog.

 

[Greg72]

exFat is the better way to go if this drive is going to be connected to a Windows machine. ext4 if the drive is always connected to a Linux or Android device. When OpenZFS finally hits the router kernels, it is going to be even better.

 

[ColinTaylor]

exFat is the better way to go if this drive is going to be connected to a Windows machine.

The router doesn't support exFAT so that's not currently an option.

 

[Greg72]

The router doesn't support exFAT so that's not currently an option.

Actually Merlin does support exFat. https://github.com/RMerl/asuswrt-me...c/router/busybox/util-linux/volume_id/exfat.c

If you instal Debian, you extend the capabilities.

 

[ColinTaylor]

Actually Merlin does support exFat. https://github.com/RMerl/asuswrt-me...c/router/busybox/util-linux/volume_id/exfat.c

No it doesn't. That's just some volume_id code that's bundled with busybox and not used, it's not a filesystem driver. You're also looking at the wrong repository.

 

[Hale_JP]

"in the best position to answer your own question. You say that you can't add any more files"
My experience with linux boxes is not enough to tell for sure. technically, it was an added pile of files, there also were other backups before that, and something in sysvol-folder. so I guess, that is around 100,000. What I can not understand is why it relies on memory so much? NTFS was developed in the end of 80s, when computers were limited with 1Mb RAM. Driver should traverse nodes on request dynamically, and should not have a large footprint... maybe it is Samba, which tries to fetch all the shares in one RAM at a time?

 

[ColinTaylor]

maybe it is Samba, which tries to fetch all the shares in one RAM at a time?

It would be easy to test. Just log into the router through SSH and try to create some files from the command line. If you still have the problem then it's nothing to do with Samba.

 

[L&LD]

But do keep in mind that in the 80's we didn't have TB's of data on tap either.

Remember single-sided floppy disks? Cheap, relatively affordable storage 'memory' for the masses back then. And a grand total of around 180KB of storage!

The NTFS of 1980 is a far cry from the NTFS of today's OS'.

 

[Greg72]

But do keep in mind that in the 80's we didn't have TB's of data on tap either.

Remember single-sided floppy disks? Cheap, relatively affordable storage 'memory' for the masses back then. And a grand total of around 180KB of storage!

The NTFS of 1980 is a far cry from the NTFS of today's OS'.

The Air Force finally moved away from 8” floppies. The Railroads were using 5 1/4” disks to run their switch control computers into the mid 2000’s.

 

[Hale_JP]

we

But do keep in mind that in the 80's we didn't have TB's of data on tap either.

We did not, but NTFS and NT itself were a certain attempt to create a non-orthodox foundation for the future. So, today's volumes are inside the design scopes. It should work.

BTW, I tried attaching it back. Between last experiment and this time, another K of files were added to the archive.

And what do you think?

It could not even mount the volume! Tried rebooting, it took 10, or 12 minutes (probably lost on checking the disk). Nothing!
Even disk check failed, router answers "cannot read the log file".

Samba was disabled in GUI.

I guess, the problem is in NTFS driver.

Any suggestion how to replace it is appreciated.

 

[RMerlin]

Any suggestion how to replace it is appreciated.

You can't. All you an do is switch to ext4.

 

[Hale_JP]

You can't. All you an do is switch to ext4.

Oh, thank you for attention. Do you mean that even this will not work?
https://www.snbforums.com/threads/usb-hard-drive-and-87u.29696/#post-522110
I did not make it yet.

 

[RMerlin]

Oh, thank you for attention. Do you mean that even this will not work?
https://www.snbforums.com/threads/usb-hard-drive-and-87u.29696/#post-522110
I did not make it yet.

It won't work automatically, as the firmware itself is designed to mount the disk using tntfs. At best it might allow you to manually mount it perhaps (no idea if it will actually work or not), but it won't be simple. Best to just switch to ext3/ext4.

You could also try running a chkdsk from a Windows PC, in case there is a problem with the filesystem indexes. The Windows chkdsk will be much more reliable than the one developed by Tuxera, which also has to deal with a very limited amount of RAM.

 

[bits]

Sounds easy to test the limits if you think you are on them.
Move a few files off the drive, then does it allow you to create that many new files.
Eg 4 files off, does that allow 4 new files on?
Delete a folder, does that allow you to create 1 more file?
Then select all files and get a full file and folder count and tell us the limit.

 

[Hale_JP]

It won't work automatically, as the firmware itself is designed to mount the disk using tntfs. At best it might allow you to manually mount it perhaps (no idea if it will actually work or not), but it won't be simple. Best to just switch to ext3/ext4.

You could also try running a chkdsk from a Windows PC, in case there is a problem with the filesystem indexes. The Windows chkdsk will be much more reliable than the one developed by Tuxera, which also has to deal with a very limited amount of RAM.

Oh, I thought it acts as drop-in replacement. Is there a way to reconfigure automount?
Unfortunately, ext-FS are useless by concept for windows backups. And using external USB drive is the most convenient way to restore anything.

It looks like I have to abandon the idea of synching files there and return to image backups.

Do you plan to examine the problem, or upgrade the driver in future firmwares?

Sounds easy to test the limits if you think you are on them.
Move a few files off the drive, then does it allow you to create that many new files.
Eg 4 files off, does that allow 4 new files on?
Delete a folder, does that allow you to create 1 more file?
Then select all files and get a full file and folder count and tell us the limit.

That is exactly how I hit the limit.
I was synching there some user directories.
When added a photo archive, which is HUGE. At some point, I got error messages in synching software. My first suspicion was that the software fails to access SMB. Seems, Admin can have only one session there.
So I tried making a new user with exclusive ritghs for the folder. And could not assign the rights. ACL definitions in the embedded samba are stored on the drive in separate files. And since the limit was reached, I could not create one anymore.
I could only create new file, or folder after deleting another.
When I copied more in Windows, the partition failed to mount with errors. But Windows did not find any errors.
So the problem is in tntfs.

 

[RMerlin]

Oh, I thought it acts as drop-in replacement. Is there a way to reconfigure automount?

No, it's part of the firmware code.

Unfortunately, ext-FS are useless by concept for windows backups.

Why would that be? When using SMB, the underlying filesystem doesn't really matter.

Do you plan to examine the problem, or upgrade the driver in future firmwares?

The driver is always kept up-to-date with whatever Asus uses. There is nothing for me to investigate or upgrade.