Thunderbolt / Enclosure

[Tech Junky]

TLDR; Results - https://www.snbforums.com/threads/thunderbolt-enclosure.79526/post-775493


So, I have a TB4 port on the laptop and ordered a TB4 AIC for the server.

Now, the problem is how to test / verify. The current enclosure works great but it's a C-10gbps that gets full 1GB/s.

Well, that's nice and all but, TB3/4 should hit ~2GB/s+ and require different HW.

I would prefer to forego the TB3 option as it's rated for 22gbps and TB4 is 32gbps and supposed to roll in USB support w/o a bridge chip.

Also, TB4 should bring prices down w/o the royalty to Intel for licensing / certification.

The next thing coming to mind is whether or not OEM's will be adding Gen4 support to max out the theoretical 5GB/s possible as TB3 uses a Gen3 interface on the board.

I've been down the rabbit hole looking at enclosures and the best I can tell is there's 2 types. TB only and TB+USB-10gbps. Obviously USB is important for using it on other systems that don't have TB.

The speed difference between enclosures seems more related to the drive being used rather than a technical limitation of the controller. Though I've seen plenty of OWC enclosures using 6th gen TB controllers while more recent using the prior 7th gen controller. Seems in ~2 years time since the current 8th gen controller has been released by Intel no one has made a product yet and on the USB4 side ASM4242 likely won't be released until the fall.

I guess since up until recently TB has been mostly a MAC enthusiast / needed option since they typically can't crack open a case and add anything themselves. Hopefully being more open sourced and deployed on more PC based systems things will change where the pricing for an enclosure drops from ~$130 average down to USB like prices closer to $50. I paid $50 for the TB4 AIC used off Amazon figuring some idiot couldn't figure out how to cable it and turn on the features int he bios to make it work or expected it to exceed the speed of whatever they were trying to connect to it. Which leads to another issue I have with TB is the inconsistency of the connections needed varies by which OEM makes the card / MOBO. Asus uses a 14-pin / Gigabyte 5+3 pin / ASRock 5-pin. So,, I have an ASR MOBO w/ a 5-pin but the market is dry for ASR AIC's and they only provide 27W of power on the ports which is kind of dumb. The card I have enroute though is the GB Maple Ridge with the 2 x 6 pin power ports to provide 100W of PD. I dug through everything I could to figure out what the addition 3-pin is used for and couldn't find any documentation on it anywhere. Maybe it's a legacy connection being used for some security protocol or hot-plug or ? *shrug*

Long story short.... any recommendations on enclosures? Any insider info on TB4? USB4 specs beyond bundling USB + TB into the same connector / waiving the royalty fee?

 

[Tech Junky]

Card arrived for the 2nd attempt since the first one got cancelled due to inventory aka the seller sold it before Amazon could remove the listing. The card arrive all sealed up w/ all cables included. The static bag had a smudge where you could see where it was opened and then resealed which is fine.

Cracked the PC case and added the USB / 5-in TB / 2 x 6-pin PCIE PSU leads for 100W PD and then closed things up.

Had to edit some things to get the network working again since the IF numbers change with different PCI slots being populated.

Rebooted and spotted the TB4 come up in the dmesg scrolling across the screen.

lspci shows the card / interfaces working as they should be. Still need to get an enclosure to test them for max speeds but, all appears to be working. I expected to have to change something in the BIOS but, didn't spot anything relevant to TB4 but will check the manual later for any insights.

0c:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0d:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0d:01.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0d:02.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0d:03.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0e:00.0 USB controller: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020]
10:00.0 USB controller: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020]

dmesg | grep -i Thunderbolt
[    0.834411] ACPI: bus type thunderbolt registered
[    0.834439] thunderbolt 0000:0e:00.0: enabling device (0000 -> 0002)

 

[Tech Junky]

Been digging around on the subject at hand and noticing some things.

JHL6xxx - only do TB3 and aren't all that fast
JHL7440 - seems be the release everyone is using for the TB side and on the USB side JMS583/RTL9210/B
JHL8440 - is the latest Goshen Ridge / Maple Ridge peripheral release that's only mentioned with some indiegogo product that's past it's release date by a year or two now

So, I reached out to Acasis because they're hyped up all of the Mac sites as being the best option to confirm their controllers because it's not listed anywhere and this what they responded with:

TBU401 and TBU405 use same dual chip. i.e. thunderbolt: JHL7440, USB: RTL9210B

ACASIS

Quality, Creativity, Efficiency - ACASIS

www.acasis.com

I pinged the originally because I had some thoughts on why they would be releasing a new model (405) in hopes it might be a controller update and not just putting lipstick on a pig. I've also been considering the Konyead below in the capture because it's immediately available within a couple of days w/ Prime. Ultimately though I think the TBu401 is probably the best option unless TB4 controller based options are being released in the near future. USB ASM controllers coming out this fall around the release of Intel/AMD new options.

 

[Tech Junky]

Ordered the TBU401 for the tool free access to swap drives as needed quicker. The Acasis website is a bit interesting as sometimes it wouldn't take the discount for for $100 off (TBU401-10) or (TBU405-10) depending on the model but, it could just be me w/ pihole / chrome blocking things at the time.

US shipping for $6 isn't too bad either w/ 2-5 day delivery / no sales tax. Will have to see if there's a FTX fee though since I think their PayPal account is based out of Hong Kong, Total was $124.99 shipped. This beats dealing with Amazon UK w/ expedited shipping ~$140 w/ tax or the gouging from a US seller at $229 w/o tax which is now showing out of stock anyway.

The Konyead is still appealing though as a quick ship @ ~$140 w/ tax on Amazon.

Everything I came across with Acasis though indicates it should perform well and very little hassle compared to the other options. Using the RTL9210B for USB is familiar territory and nothing but glowing reviews across different sites / users. The JMS583 as I've tested in the past draws critical reviews worse than my testing experience with the controller. When I did my testing I tried out the JMS / VL / RTL versions and JMS/VL worked but underperformed considerably while the RTL was hitting 800's consistently. Not quite as high as some tests show with different drives but, good enough for a $100 1TB drive.

It should be interesting to see what this new enclosure can do as I have a few different drives I can test with Gen3 & 4 options to see how much bandwidth it unlocks. A couple of Phison based options and a SN850 as my primary drive currently. I'm also considering an additional drive for the enclosure being a SN770 2TB which shows some reasonable numbers in reviews and has pretty good numbers for a mid tier drive for $200. It should pair nicely with the enclosure seeing as though TB speeds are hobbled to 3GB/s there's no need to go much higher up the ladder for specs.

Cheaper option by $20 would be the SN570 or going primo SN850 extra $40.

Either way $125 for the enclosure is a steal when the average price right now is hovering around $170

 

[Tech Junky]

This is interesting buying from acasis.com and shipping with Amazon even through they don't have stock listed on Amazon for either TB enclosure. And..... no tracking on Amazon even with a tracking ID being provided.

https://www.amazon.com/?tag=snbforums-20 -- add the shipping ID

Never have I ever had a shipment from Amazon w/o ordering through Amazon so this is a bit interesting seeing the logistics from an outside source / order using Amazon as the carrier / delivered by Amazon similar to Prime.

 

[Tech Junky]

Did some initial testing with 3 drives - BPX Pro 1TB / PNT CS3030 (Phison - E12 based) / SN850

Results are interesting but, still want to up my game with higher Write speeds to fully utilize the drive to make it worthwhile compared to USB-10gbps.

Laptop ENC

Server Enclosure - BPX

 

[Tech Junky]

Server Enclosure - PNY
View attachment 42542

Server Enclosure SN850

 

[Tech Junky]

Internal Tests Laptop

 

[Tech Junky]

Had picked up a couple of these a year ago for some additional cables w/ potentially TB in mind in the future but the focus was 10gbps /100W.
https://www.amazon.com/gp/product/B093FQSYLB/?tag=snbforums-20 - $13 / 2.6ft where most TB cables are short / $20

Testing them in comparison with the TB3 cable that came with the enclosure and the speeds look just as good for the read vs the slower USB cable that tops out at 20gbps. The Write speeds though are faster with the Nekteck USB cable. Looks like NT released a TB 20gbps certified cable to replace the original w/ the TB stamp.

Looks like more cables are on the horizon for TB to boost those Write speeds + a new drive to speed things up.

 

[Clark Griswald]

@Tech Junky
Thank You for the shared data!

 

[Tech Junky]

@Tech Junky
Thank You for the shared data!

Most reviews lack real data and end up in bad real world performance because they don't test end to end and just paste the specs form the advert usually. I'm putting in a little more effort to find the right cable and drive for max performance. Have found a couple of candidates for cables and drives just digging deeper before ordering and testing.

 

[Tech Junky]

So, I found some more concrete speed reviews... of course this is probably where I should have looked first.

Reviews	R	W
SK Hynix Gold P31	2450	2250
SN770	2750	2690
Sabrent Rocket NVMe 4.0	2600	2700
Samsung 980 Pro	2700	2700
SN750	up to 2000	up to 2000
Pioneer 2TB (PS5012-E12)	1700	2000
SN750	2384.4	1645.3
SN750	2450	2250

The SN770 it is then when I spot a good deal on the drive. Right now it's sitting at $200 New for a 2TB

 

[L&LD]

To me, the speeds are within margins of error (run to run), particularly with the drive being 90% full (I suspect you have another partition or two).

 

[Tech Junky]

@L&LD
Same drive, same enclosure, different cable. I reran the same test w/ the Acasis cable just now that came with it which I was using for all of the other tests for TB3.

Wiped the drive completely / closed apps and running the test again showed slow writes on the Acasis cable as well. Clearly something odd is happening since the speeds dropped on both cables. I have more cables showing up tomorrow for testing but also ordered 2 x SN770 1TB drives which should max things out per reviews.

Original tests.....

Todays testing. retests have the date appended.

 

[Tech Junky]

I have two other cables showing up tomorrow for testing and started testing with HWINFO to log the speeds instead of relying on eyeballing them while testing copies. I also have 2 x SN770 1TB drives showing up Sunday for unbridled speed testing as indicated by their reviews.

Prelim HWINFO logging shows...

	BPX Pro << SN850
Anker	975MB/s
Acasis	894MB/s

Drag and drop from one drive to the other various folders with different sized files. Instead of relying on CDM to do synthetic testing this is more accurate for real world moves. The thing that's confusing how the drive dropped from 3GB to 1.8GB/s for reads and 600MB/s to ~300MB/s on writes or on the smaller sizes from 1.7GB/s to 700MB/s on the one test with rolling one end of the cable.

Testing by dropping separate folders vs a group doesn't multitask very well in terms of writes hitting ~200MB/s to the BPX. I think when I was doing initial testing with this method the other day though they were going faster. It's something I'll be trying with the new drives when they show up but, apples to apples I'll do the group of folders to the BPX with the other cables and see what they produce for throughput.

This TB experiment has a few moving parts to complete the puzzle with satisfactory results.
Controller on each side of the cable <enclosure JHL7440> / server <JHL8540> / laptop??
the cable
the drive / controller / ram / chips -- taking out of the equation with the 770's is DRAM being absent

Laptop info from Linux / TBT software was installable on the server but errors on the laptop for some reason so wanted to verify the chips vs USB4

00:07.0 PCI bridge: Intel Corporation Alder Lake-P Thunderbolt 4 PCI Express Root Port #0 (rev 02)
00:0d.0 USB controller: Intel Corporation Alder Lake-P Thunderbolt 4 USB Controller (rev 02)
00:0d.2 USB controller: Intel Corporation Alder Lake-P Thunderbolt 4 NHI #0 (rev 02)


server TB info

0c:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0d:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0d:01.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0d:02.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0d:03.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0e:00.0 USB controller: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020]
32:00.0 USB controller: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020]

Seeing as though the server card has 2x USB Type-C/ 1x DisplayPort 1.4/ 2x Mini DisplayPort I expect there to be more output as shown. aka 1 PCIE bridge per port

I purged the drivers from the laptop trying to get TB apps to recognize TB but the apps aren't working to see the controller / enclosure though it works fine. I did install a TB4 driver though that caused DMA BSOD and needed to be purged as well to stabilize things again. Post reboots / drivers / etc. though the speeds picked back up to where they were.

Seems both cables at this point are producing similar results. Extended 64GB size testing though shows a slight edge over the Anker cable though using the one from the box provided with the enclosure. A significant edge when it comes to small writes though @ nearly double the speed in the 64GB test.

Still some more fiddling to be done to see if there's anything else that can be done to get cable tests to sync with each other.

 

[Tech Junky]

Observations with another day of testing. The average speed of Writes for longer duration actual copies is 450-500MB/s. In the prior tests the Acasis cable performed just as well as the OWC shown in the capture. Anker / Cable Matters not so much. Getting the drives tomorrow to really see what this setup is capable of though.

	Measured with HWINFO / counters reset between tests
Temps	Min 25C / Avg 60C / Max 70C
	BPX Pro << SN850	CDM 16GBx5
	W MB/s	R/W GB/s
Anker (TB3)	921	3x1.7
Acasis (TB3)	961	3x1.8
Cable Matters (TB4)	998	3x1.7
OWC (TB4)	946	3.1x1.8
Avg Write Extended	450-500

 

[Tech Junky]

@L&LD

So, I got the new drives and tested them thoroughly and the bottleneck was the drives not the cables or enclosure. Something potentially with the controller / DRAM setup causing slightly lower numbers? Not exactly sure especially when looking at the SN850 being a top tier Gen4 drive. I expected it to perform a bit better in the enclosure vs the other 2 drives.

I tested with 5 different cables and here's what I came up with.

Temps	25C/70C			29C/80C
			SN770-443		SN770-211
	BPX Pro << SN850	CDM 16GBx5	SN770 << SN850	CDM 16GBx5	SN770 << SN850	CDM 16GBx5
	W MB/s	R/W GB/s	W GB/s	R/W GB/s	W GB/s	R/W GB/s
Acasis (TB3)	961	3x1.8	2.6	3.1x2.8
Anker (TB3)	921	3x1.7	2.6	3.1x2.8
Cable Matters (TB4)	998	3x1.7	2.6	3.1x2.8
OWC (TB4)	946	3.1x1.8	2.6	3.1x2.6	2.6	3.1x2.7
Sniokco (USB4)	970	3.1x1.8	2.4	3.1x2.8	2.4	3.1x2.8
Avg Write Extended	450-500
SN770-443 Internal				3.5x3.4
HWINFO not working	internally

I'll keep testing the #211 drive as time permits but, the Sniokco / OWC cables match up between the two SN770's which leads me to believe they will with the other 3 cables.

 

[avtella]

Thanks for the posts, pretty informative.


Just some side notes:

Samsung in my experience generally performs better than the competitors in 1/2 and full drive tests as their controllers seem to be the best at least in the consumer space, though probably not worth the premium over Phison based drives. Phison’s not too bad and some vendors like MyDigital and Corsair (overprovision these Phison drives more by default, hence lower advertised drive numbers like 960 & 1920 GB vs 1000 & 2000 but better performance under more full conditions.

SK Hynix has done a pretty good job with their in house controllers, their drives seem to have an excellent balance between performance and battery life vs the competition (if Mobile is I guess)

WD with the 750 series I think had issues with sleep states if I recall (more important for Mobile not so much for desktop)

Overall though only a few conpanies are fully vertically integrated designing/making their own controlller and flash, Samsung, Micron, Sk Hynix, WD as far as I recall. Others pretty much buy flash from these guys plus Toshiba (Kioxia) and off the shelf controllers like the Phison E12, E16, E18 etc, Silicon Motion 2262 etc. (Ex: Sabrent, Corsair, MyDigital, HP, Adata, PNY etc). I’d avoid the Silicon Motion controller based drives though those were relatively awful sustaining performance as the drives filled, vs similarly priced Phison models.

 

[Tech Junky]

Yeah, the 2 Phison drives work fine until I had something fast enough externally to show their short comings. Internally they work at speed as designed. Funny thing about all of them though in the 1TB size show up in the system as 931.xMB. So, even with the overprovisioning they're all the same usable space.

Initially Phison drives had some FW issues that required a couple of upgrades to get them up to speed before using them. FW 12.3 works at speed and AFAIK is the latest even a couple of years later.

WD SN850 had a FW upgrade I wasn't even aware of and not sure it really did anything performance wise on Intel but might have solved some issues on the AMD side. The "dashboard" app for WD does the upgrade in mere seconds while the disk is online w/o needing to reboot to activate it.

WD uses SanDisk controllers though when looking at the disk. Not sure which chip/s though with the whole sticker deal / warranty. The Phison / SN850's though are double sided and the 770 is just 1 big chip and a couple of smaller ones for the controllers.

I've seen nothing good to be said about the SM controllers either. The Mushkin's use Inogrit which isn't used commonly by anyone else which keeps me on my toes when looking at them.

Anyway, there were 3 brands / models that worked at speed looking at the reviews WD 770 / Sabrent / 908 Pro. Looking at these numbers now makes me wonder why they're not able to breach 3GB/s on their systems. Maybe it's the TB3 only port vs my TB4 port or they're lacking other resources like CPU/RAM sufficient to saturate the line / drive?

Acasis Reviews	R	W
SK Hynix Gold P31	2450	2250
SN770	2750	2690
Sabrent Rocket NVMe 4.0	2600	2700
Samsung 980 Pro	2700	2700
SN750	up to 2000	up to 2000
Pioneer 2TB (PS5012-E12)	1700	2000
SN750	2384.4	1645.3
SN750	2450	2250

A little detective work goes a long way when getting things to work as designed and of course Acasis recommended drive list doesn't even mention half of them. Getting it running at 3x3GB feels better than most of the lopsided results others posted. Some of the other enclosures went with 3x2 lanes on their boards to keep the energy low to keep the heat throttling out of the picture but results in 1.5GB top end speeds and charging the same as a 4 lane model.

If I didn't already have the 850 though the 770 would be a great stand in on the gen4 slot.

Additionally I tried booting a cloned image of my Windows laptop to update the firmware on the server drive / 850 and it booted up on TB and BSOD'd on USB for some reason but, then again I didn't expect anything less from MS on that front as they don't just make things work. I do have another flash drive thought hat shows up as a SSD that hits ~400MB/s which is less than the 10gbps and works fine for booting windows. It's just a bit dumb to deal with when you need to for a particular reason.

 

[Tech Junky]

This has to be the cheapest cable in terms of costs when it comes to this testing though @ $9.99
https://www.amazon.com/dp/B093FQSYLB/?tag=snbforums-20

Of course with PD coming up in a day or two everyone jacks up their listing price for when the discount hammer falls to generate traffic / sales. I might pick a couple more of these up when they drop in price again.