[TESTING] File transfers (Samba/FTP/NFS) from USB drive to client intermittently fail

[ColinTaylor]

As per the title, there have been frequent reports in the forum of file transfers from the router’s USB drive being unreliable. Often this is when people are trying to use the router as a NAS. The problem can manifest in various ways and its intermittent nature has made understanding the cause difficult.

Having come across a simple set of circumstances that appeared to allow me to recreate the problem on demand I thought it would be interesting to diagnose it further. What follows is a summary of my observations, experimentation and one possible solution.

As my test setup is very simple I would be interested to hear from people experiencing this issue whether the solution below was effective for them.

Observations and Packet Capture​

Packet capture on router shows cksum is incorrect (because it's offloaded).

When the problem occurs the router sends a single "TCP Out-Of-Order" followed by multiple "TCP Retransmission"s. The client sends lots of "TCP Dup ACK" back to router before failing.

Only seems to affect data that's been read from a USB drive, even if it's already cached, as it doesn't appear to effect iperf.
EDIT: Retested iperf with 10 parallel streams and it does appear to suffer some sort of problem as tx error messages appear in the syslog.

Only specific files seem to trigger the problem.

SCP transfers don’t appear to suffer from this problem but SMB, FTP and NFS do.

Only affects router to LAN transfers. Does not affect WAN to LAN or LAN to LAN traffic.

Doesn't affect Wi-Fi clients as badly, although I still see Dup ACK’s. But client seems able to recover using normal TCP methods.

Potential Solution – Disable TCP segmentation offload on bridge interface​

To disable TCP segmentation offload, checksumming and scatter gather must also be disabled (which in turn disables generic segmentation offload).

Turning off checksumming alone or gso alone does not fix the problem.

Use this command on your router to turn off TCP segmentation offload (this change will be undone if the router is rebooted):

ethtool -K br0 tx off sg off tso off gso off

N.B. ethtool must be installed from the Entware repository (opkg install ethtool).

Note: Disabling offloading increases the load on the router’s processor. As such your router to LAN throughput may be restricted, especially on routers with weak CPU’s (e.g. RT-AC68U). This may also impact other CPU-intensive processes running on the router like VPN encryption.

 

[SteverinoLA]

When this happens, do you see many SMBClient errors in Windows event log (scroll down the long list to SMBClient)?

 

[drinkingbird]

When this happens, do you see many SMBClient errors in Windows event log (scroll down the long list to SMBClient)?

Call me paranoid but I want none of my data sitting plugged into the internet facing router. It would be nice to make use of AiDisk functionality (though it can be quite problematic as seen here) but do not trust it. I have an old PC that I use for local storage (I don't do it enough to invest in NAS, it is mostly just backups and hosting my music and video collection which I typically don't use that often since it is all on my phone).

Ironically I actually trust my 1TB Onedrive more - but I do still encrypt anything sensitive that I back up there. While Microsoft is certainly not immune to vulnerabilities, it at least has a lot more enterprise level protection around it.

If I did want a NAS I might consider a second Asus router (maybe upgrade my main one to AX and keep my AC) as a dedicated NAS setup behind that router. Cheaper solution than full fledged NAS assuming you don't need massive throughput, >1G connections, etc.

 

[Yota]

Is it possible to report this issue to Asus? Because NAS is a marketing gimmick, and nowadays that gimmick doesn't work anymore, it's their responsibility to fix it, right?

 

[Jeffrey Young]

+1 with @Yota on this one.

Still, in the end, Asus stands to be a better product thanks to this kind of initative and research (thanks @ColinTaylor)

 

[Yota]

I think @Tech9 has a lot to say on this topic, Tech9 can you share your way of reproducing the problem?

We need more people to pay attention to this issue because when marketing gimmicks don't work, it's fraud.


Every time a forum member comes here complaining that the SMB/NAS feature doesn't work, instead of helping them figure out what's wrong, people tell them to give up. stop talking here about routers not supposed to act as NAS, I've heard this statement on here for countless years and it's true, but it also prevents people from testing what the router can do, which helps Asus cover up the problem.

Feedback is valid as always, maybe a single feedback won't do much, but if 5% of Asuswrt users' feedback is enough for Asus to take this issue seriously.

We need to clarify the steps to reproduce the problem, preferably 100% on each router, and then we need to mobilize everyone to report the issue to Asus.

Asus puts a lot of development into NAS, even buying paid NTFS drives just for better performance, so I'm sure they'll take this seriously.

let us start doing something, doing.

 

[Tech9]

I tested standard configurations only Asuswrt and Asuswrt-Merlin with no additional fixes. This is what Asus advertises as working and Asuswrt-Merlin has available with some additional features. The routers crash from out of memory condition. They crash faster with faster USB interface or device (USB 3.0 SSD), larger volume (in GB at once) and multiple small size files (in thousands at once). Usually there is nothing much in logs, but the side effects are lost WAN or WLAN, unable to login to UI, SMB crash etc. whatever was terminated first. AC86U running 386.9 was the first one clearly showing Out of Memory in logs. Chances to complete the transfer increase with USB 2.0 and slow USB stick and on 1GB RAM models.

Bad working features are also WAN aggregation, Link Aggregation, Dual WAN, Adaptive QoS, I also found Traditional QoS and Bandwidth Limiter broken on some firmware versions with or without IPv6 enabled, IPv6 in Passthrough mode, IPv6 Dynamic DNS, firewall rules exposing devices with IPv6 enabled was discovered too, AiMesh Status page broken showing 0 traffic, Ookla speedtest crashing saying WAN is disconnected, VPN Fusion crashing the router in recent firmware, sluggish WebUI with TrendMicro components enabled, AiProtection not registering blocked pages, wrong DST time zones, multiple copy/paste WebUI bugs, about 30% reduced OpenVPN performance 384->386 base on ARMv8 + AES models... from memory only, there is more.

I used to do target testing for things I know were broken at some point and every firmware just fixes one and breaks another. Some of the above are broken for years. What is Asus mostly fixing is what sells more routers - AiMesh. This is what the entire 386 base firmware was about + some security fixes. What was broken in 384 is broken in 388 as well. I just give up on testing because it's really disappointing and I don't really know what works and what doesn't on every new firmware release unless I spend hours going through all the features advertised. In some cases my routers were running 4 different firmware versions in one day just to compare what's going on and find eventual workarounds. I just can't do this anymore and I don't want to because nobody seems to care.

 

[Tech9]

Feedback is valid as always

No. After many weeks total time of testing my findings were called limited datapoints. I have no reasons to continue testing. It is what it is.

Is it possible to report this issue to Asus?

I did report 4 times obvious and easy to reproduce bugs via Feedback Form in Asuswrt with logs and test scenario. It was few firmware releases back in about a year time period. They are still present in latest 388 firmware release plus more things broken on top. Good luck with reporting to Asus.

 

[SteverinoLA]

Hmmm, I have a couple threads from this week that your commenting in, one about 2.2G 160mhz WiFi still half the speed of my gigabit network, and the other about 86U crapping out and disconnecting during Samba large transfers.

Now after reading this thread, I suspect those 2 problems are related -- my 86U is getting overwhelmed and samba is crapping out when transferring big files at gigabit 100Mb/s speeds. I have no problems transferring same large files via WiFi which maxes out at 50MB/s.

 

[Tech9]

Use whatever you can. For me this is just wrong hardware with wrong software on top. If you have no budget for real NAS get one used mini PC from eBay with USB 3.0 ports, run whatever OS you like better on it and plug your USB drives there. It will guarantee you Gigabit speed transfers on LAN and whatever you can squeeze on Wi-Fi. The one below is $100 on eBay - Lenovo M93 with Intel i5-4570T CPU, 4GB RAM, 500GB HDD, 4x USB 3.0 ports, Intel Gigabit NIC and Windows 10 Pro pre-installed. It also has one empty RAM slot for memory expansion. Power consumption at idle is about 10W. There are other similar mini PCs available from HP and DELL. You can even get second Gigabit NIC on specific models with some DIY creativity.

Just look around and find a solution for your needs. If you wait for Asus to find it your router will perhaps enter End-of-Life state before it happens.

 

[Yota]

I tested standard configurations only Asuswrt and Asuswrt-Merlin with no additional fixes. This is what Asus advertises as working and Asuswrt-Merlin has available with some additional features. The routers crash from out of memory condition. They crash faster with faster USB interface or device (USB 3.0 SSD), larger volume (in GB at once) and multiple small size files (in thousands at once). Usually there is nothing much in logs, but the side effects are lost WAN or WLAN, unable to login to UI, SMB crash etc. whatever was terminated first. AC86U running 386.9 was the first one clearly showing Out of Memory in logs. Chances to complete the transfer increase with USB 2.0 and slow USB stick and on 1GB RAM models.

Can you point out specifically how to reproduce the error pointed out by this thread?

How big a file does it need? How many gigabytes? If it is a small file, what kind of small file is required? How many bytes, approximately how many small files are required to trigger the error?

I hope I can make a sample file that can trigger the error 100%, or files.

No. After many weeks total time of testing my findings were called limited datapoints. I have no reasons to continue testing. It is what it is.



I did report 4 times obvious and easy to reproduce bugs via Feedback Form in Asuswrt with logs and test scenario. It was few firmware releases back in about a year time period. They are still present in latest 388 firmware release plus more things broken on top. Good luck with reporting to Asus.

Don't want to complain, but this forum somehow doesn't allow people to point out Asus' mistakes. This is what I often see among some Apple fanatic users: "If there is any problem there, it must be a user's fault".

I'm sorry to see that your feedback isn't being understood at all by others, but I think it's partly because there's so little this forum can do, and it's not RMerlin's responsibility to fix these bugs, it's just that Asus should provide quality assurance for its products.

We should make the problem reproducible as much as possible, and then mobilize as many people as possible to directly report to Asus. It is difficult for one person to bring about some changes, but together, I think we can do something.

I will not recommend any relief measures in this thread or let people buy real NAS. we can no longer avoid the NAS issue, burying our heads in the sand will only lead to worse firmware, we have to reproduce the issue, and feedback to Asus together.

 

[SteverinoLA]

My repeatable test to produce SMB errors is:

1) Need to use a LAN connection, or possibly WiFi w/at least a 100MB/s true throughput
2 ) Put a folder on USB drive attached to router with a lot of files great than 1G, eg:

3) Attempt to copy this folder from USB drive to local PC via samba networked drive

4) Observe the throughput starting off at 100MB/s then quickly see throughput start dropping and then hit 0MB/s.

5) At this point, check the SMBClient event logs in windows and see:

 

[Tech9]

Can you point out specifically how to reproduce the error

@Yota, I have gone back 4 years of firmware releases and it's all the same on different model routers. How I test is very simple and close to what an Asus router user would do looking at USB 3.0 port(s) available, USB Share in firmware and some test results online - Asus promises >60-100MB/sec file transfers, so must be in USB 3.0 mode; I don't want the drive to be the bottleneck, so SSD instead of USB stick; Asus offers drive format in WebUI, so drive formatted to NTFS in WebUI; Asuswrt supports SMB 1.0, Asuswrt-Merlin SMB 2.0 as well, so the necessary adjustments for Windows; then I just attempt a transfer of about 20GB in size and 20.000 mixed size files from 1GB to 1KB. I have tried different drives as well including fast USB sticks (Sandisk Ultra Flair was about to melt, seriously), external portable drives from WD (My Passport) and Seagate (Backup Slim), 2 types of external 3.5" enclosures with own power supply, WD Elements with own power supply... you name it. It was done to eliminate eventual controller or device brand/make issues.

I have access to hardware and can rinse and repeat multiple times. I strongly believe very few users around can do that while actually using the routers for Internet access. Still, every time I report an issue and show screenshots someone who tried it once and it worked or tested it for 15min shows up and we are going nowhere up to open "I don't care" statements. In this case I don't care either. You guys figure it out, I can't help.

 

[Tech9]

I'm sorry to see that your feedback isn't being understood at all by others

This very thread is an excellent example of what usually happens. We have under 15 comments on something @ColinTaylor spend the time to investigate, found an issue and even offered a solution back in March 2022. The second post in this thread is from yesterday in Jan 2023. Colin basically wasted his time and no one cares. Thank you @SteverinoLA for finding the old thread so someone may eventually look at Colin's work.

 

[SteverinoLA]

Well, I appreciate your work and Colin's work. Unfortunately, that doesn't help us get this resolved.

 

[Tech9]

It's been like this for years and this is why the general recommendation is don't use the router as NAS. I have actually tried the same transfer on my ISP provided modem/router with USB ports and USB share and it went through on a first attempt for >2h time. It was slow, but steady and the router didn't crash. So it's possible, but not on Asus routers at the moment. I remember testing Archer C2300 (the same hardware as RT-AC86U) and it had no issues as well. Synology RT2600ac also does USB share properly, but it's kind of expected from a NAS manufacturer. A little surprise was the old RT-AC68U where sending Windows PC lost the destination few times as well, but the router recovered and kept going without crashing.

 

[thecheapseats]

@ColinTaylor - as always, thanks for your diligent time and testing...
@Tech9 - cogent conclusions regarding the situation... thanks as well...

 

[GHammer]

You know, I understand that some bought a router based partly upon 'Hey I can use it for file share too!'
Way back, I thought the same. Took less than a week to decide to buy my first actual NAS.
Personally, I use Synology and have moved up their foodchain 3 times in ~10 years. If you want to run other things on the NAS it's likely there as well. Can't swing a retail NAS? @Tech9 has good suggestions for investment of a little time and learning.

Yep, I'm one of those 'Quit tilting at windmills' people. A router is not now and is not going to be a do-all box.
Be happy if it does its core function well. For me, it does what I need. I don't need QoS, VPN, or ad blocking on the router either.

 

[Yota]

This very thread is an excellent example of what usually happens. We have under 15 comments on something @ColinTaylor spend the time to investigate, found an issue and even offered a solution back in March 2022. The second post in this thread is from yesterday in Jan 2023. Colin basically wasted his time and no one cares. Thank you @SteverinoLA for finding the old thread so someone may eventually look at Colin's work.

I really didn't realize this thread was from last year, I thought it was from yesterday.

Well, if everyone reading this isn't going to join me in giving feedback to Asus, it's hard to change anything on my own.

"Why should I care if everyone else doesn't? I don't use the USB functionality." this is what I see here.

 

[ColinTaylor]

I really didn't realize this thread was from last year, I thought it was from yesterday.

Well, if everyone reading this isn't going to join me in giving feedback to Asus, it's hard to change anything on my own.

"Why should I care if everyone else doesn't? I don't use the USB functionality." this is what I see here.

Have you been able to recreate this problem? ***

I've never seen the out-of-memory errors that @Tech9 describes. Notwithstanding my original post, I've also tried to recreate his failure scenario on numerous occasions by copying nearly 500GB of files (including 10's of thousands of small files) back and forth between a USB HDD and it never failed once.

Given that in the 10 months since I posted this thread not a single person replied, let alone attempted to verify my observations I could only conclude that the problem was an extremely rare edge case and/or it was fixed in a subsequent firmware update. Indeed there are multiple posts from people that are regularly transferring large volumes of data to their router's HDD without issue.

*** There are numerous reasons why network file transfers might fail. It would be a mistake to lump them all together as a single problem.