questions about Synology SHR ....

[zero7404]

i currently have what would be considered a barebones NAS (WD EX2 Ultra) .... and I want to migrate to a Synology in order to benefit from advanced RAID features and other features offered by DSM.

have some general questions about DSM, and how it treats disks .... that would help me decide what I can do with the drives I have to use.

regarding SHR (Synology Hybrid Raid), SHR2, what does the portion of the volume that is allocated to protection actually represent ? does it mean that the array is allocating enough space to rebuild ANY data that is lost from a single drive failure ? i am pretty confident in my drives and doubt i will have any of them fail, but in case one (or 2) do fail, would the protection offered by SHR/SHR2 be enough to restore the data lost by the [up to] 2 disks lost ?

my drives don't all have matched capacity, which i think is fine for most modern software-RAID setups. in particular I have the following drives:

Toshiba N300 6TB
Toshiba N300 6TB
Toshiba N300 8TB
Toshiba N300 8TB
Toshiba MG08 16TB
Toshiba MG08 16TB

My current content/storage usage on my nas is a little more than 8TB, and the rate data is added is not drastic enough to say that i could consume another 8TB in the next few years. Would I be better off with SHR2, or RAID10 if I threw all these drives in a Synology 6-bay NAS ?

Other thing that's interesting i'd like to know more about is how the unused space (SHR2) would allow me to swap out for larger disks in the future .... would like to know how DSM works in that regard when the time comes to upgrade the disks.

thanks in advance.

 

[L&LD]

All HDDs fail. Planning a NAS without acknowledgement of that fact will lead to drastic data loss, sooner or later.

Already having 8TB+ of data, I would suggest you set up a proper RAID on the new NAS (RAID5) , then copy the data to it. Continue using the old set up as a backup of your new, primary NAS.

 

[zero7404]

understand and noted these points. i keep appropriate # of duplicates of my data for redundancy, but being a home user, eyeballing a server-grade raid setup would be overkill and expensive for me.

simple thinking, that a Synology would provide just as good data handling/protection as my current arrangement, with the benefits of easy expansion, disk diagnostic monitoring and not having to worry about an external raid1 enclosure as I have now. to do smartctl now on my current NAS, requires ssh each time. as I understand the Synology has a lot of functionality built into DSM, in addition to being able to break in via ssh for command line work.

either way, i am interested in learning more about DSM and how SHR/SHR2 works, if anyone has experience using the platform for some time.

 

[L&LD]

Understand. I'm sure others will give their opinions here and more directly address your questions.

Do note that a WD EX2 Ultra isn't a NAS. Nor is it cheap it seems. You're already playing in the same budget space as a 'server-grade raid setup', I suggest you simply follow through and do it right. Overall, a QNAP or Synology NAS is much cheaper than those toy-like solutions (they're built to last a decade or more).

What are you using to ensure data redundancy? Is that backup current and up-to-date? You may be able to use the bigger drives in your new NAS if you truly have a full backup available to fall back to when you disassemble your current working setup.

The proper way to get into a QNAP or Synology is to buy a unit with as many drive-bays as possible, set up at least a RAID5 array (3 drives minimum), then add drives as needed in the future. A six-drive bay unit will allow you to have 2 fault-tolerant RAID5 arrays and simply replacing the drives on each array will increase the capacity (when all have been replaced).

 

[zero7404]

thanks for that advice .... one of the things i was interested in knowing about is the ability to create 2 or more RAID volumes with multiple disks installed.

not sure where you're looking, but almost all single bay and 2 bay WD NAS's are bottom of the barrel in terms of the NAS industry. does mine have enough features to get the basic job done ? yes. it also has a media server, as well as other features. but what's offered on them are the most basic/watered down versions of what one can have with higher end NAS units. it's a fixed-hardware setup that serves content at the rate it can, nothing more (no transcoding ability or hdmi output to a TV, etc.).

the NAS is setup as software RAID0, whereas my external backup is hardware RAID1 (an OWC enclosure).

although i have 2 duplicates of my entire NAS on the RAID1, whatever data corruption goes unnoticed on the NAS - is copied to the RAID1 during a backup. so that is one risk i run using RAID0 for the master source of data. i am relying on the WD and it's software RAID to run the show. one positive aspect is that the NAS does use ext4 internally on the disks and so far after some months of running N300's in it - SMART diagnostics for each disk show me no issues to be concerned about and they are healthy.

on the outside of the NAS, the RAID1 enclosure is behind hardware raid - and i have no way of checking disk health without removing the drives and mounting them separately on a computer. which would break the RAID1 array when I put them back (according to OWC).

so as you can see, I need some additional functionality that a Synology or QNAP offer, and never have to man-handle the disks until I am ready to swap out. their beefy hardware doesn't really phase me because i'm not demanding on my usage of the content and it is a small family that the NAS is serving (not an organization/business). but having the extra bottom-end on the hardware like an HDMI port and LAN aggregation is more appealing for longevity - i would no longer need to rely on streaming my media via a roku box or an xbox .... i could just connect the NAS to the TV and let the NAS do the decoding for playback.

 

[zero7404]

i would also caution anyone who relies on spindle-based storage, to stay away from SMR hard drives.

i have 2 older WD products (a MyBook and a MyCloud), in which both unit's hdd's failed. they put SMR's in them to sell them cheaper. Shingled Magnetic Recording can lead to allocation errors, overallocation, etc. lucky i had another good copy of my data when i spotted the problem. that also triggered the whole re-do effort on my home NAS/storage needs.

any NAS-rated or Enterprise hdd is CMR and either SATA or SAS. the very best ones have shock and vibe sensors good enough to get the flying heads clear of the platters in case of unexpected shock to the drive.

 

[L&LD]

To me, a NAS is just simple Network Attached Storage, with a proper RAID, Redundant Array of Inexpensive Drives. The other features are secondary and seldom if ever, used in my home.

RAID0 as your primary data source? I don't believe you have data you care too much about.

My final advice here. Don't rely on smart diagnostics or other numbers that show how healthy everything is. These states change at a moment's notice.

Use a proper RAID level, in a generally available (and therefore 'standardized' hardware) such as QNAP (much more hardware vs. Synology for the same or lesser cost).

Use a proper UPS to protect your data and NAS hardware from not just power outages, electrical storms, or surges, but also from brownouts that you may not even know your equipment experiences (and is just as harmful, if not more so).

Don't merely rely on technology to save your data. Rely rather on best practices too. You may not be a corporation, but it won't hurt to operate as one with regard to your data.

Assuming it is important to you at all.

 

[zero7404]

all great advice !

strategy-wise, keeping my raid1 powered down until a backup is needed seems to make sense. used to be backup-fanatic and ran jobs nightly. it occurred to me that i could be aggravating that unnoticed data corruption nerve by forging ahead that way.

so, backed off a bit to give time to use/access whats on the nas, increasing the chance i could find an issue with a file BEFORE i backup.

having raid0 allows me to run more storage, the primary reason behind the decision, along with the limitation of a 2-bay unit and before i had access to the larger disks i listed above.

so i would consider now a 6 bay and throw all my disks into it, with mirror+stripe or shr on a few volumes to take advantage of the protection features offered while giving back added functionality and use for arrays with mismatched disk sizes.

still, a risk runs that everything resides in a single enclosure, which could be an issue if software/hardware fail on the nas for some reason.

i have my networking gear plugged into a ups so there is some safety there in addition to having time to manually spin down in the event of power outage. nas has a line of communication with the ups as well.

 

[L&LD]

I thought this may be appropriate in this thread too.

Release of QNAP and ULINK DA Drive Analyzer, an AI-powered tool for predicting drive failures on NAS (guru3d.com)

Not free, but possibly a beneficial use of 'AI' to better predict drive failures than S.M.A.R.T. thresholds.

 

[zero7404]

interesting find … will check it out.

though not much of a fan of the bleeding edge of software technology, that includes AI as a platform.

software updates are the cat and mouse diet that all these vendors and manufacturers are pitching to the public to accelerate planned Obsolescence and much of them fix something that ain’t broke to begin with.

i stopped fanatically updating my gear in general, even the xbox - where that fondly crummy seagate 1TB disk that failed on me because of the daily blizzard of console and game updates and microsoft’s stupidity that allows someone to play a game while it’s being updated in the background.

hope you see where i’m coming from.

software managed storage i keep a keen eye on with regard to manufacturer updates and not WD nor Synology nor QNAP can convince me that their latest update is the cat’s meow.

 

[L&LD]

Your call, of course. But this isn't QNAP specific. As I understand it, it is a third-party offering that is targeting software-based hardware monitoring.

Not something I am willing to jump on today either, but now that I know of it, it will be interesting to follow its development over the next few months/years.

 

[sfx2000]

so i would consider now a 6 bay and throw all my disks into it, with mirror+stripe or shr on a few volumes to take advantage of the protection features offered while giving back added functionality and use for arrays with mismatched disk sizes.

If one is purchasing/deploying a new NAS, don't use old drives... just asking for trouble a year down the road when drives start dying...

Invest in new ones, and match them up as a set

 

[John Davis]

yes you can SHR all those drives ( 2 * 6tb, 2 * 8tb and 2 * 16tb ) - however the actual capacity yielded will be kind of low ( it’ll create a 6 member raid5 array using 6tb partitions across all drives, then a 4 member raid5 array using 2tb partitions across the 8tb and 16tb drives and finally a raid1 array using 8tb partitions on the 16tb drives - then join them all together using lvm - shr is a nice term but it’s really just the linux md array driver plus lvm)

so actual usable capacity will be ~30tb+6tb+8tb=44tb

the first question is do you actually NEED all that space in one large volume - if not raid1 on the 2 16tb drives ( which are presumably your newest drives?) and shr across the 6tb and 8tb drives probably makes more sense ( one 16tb volume and one 20tb volume). Or even shr-2 (2 disk redundancy same as raid6 ) on the old/small drives.

If you want maximum resiliency just run raid1 on each pair ( 6tb + 8tb + 16tb volume )

As someone has already said, building arrays on old drives can be fraught - the risk is when you have a drive failure the increased workload of rebuilding onto a replacement drive will push another drive over the edge - and with raid5/shr only having single disk redundancy a 2nd drive failure during rebuild will result in COMPLETE loss of all data

And that leads to the second point - RAID is an availability tool ( i.e. your server can stay up and functional even when it looses a drive and whilst you rebuild onto a new drive ) - it does NOT replace the need for backups!!! At the least you should have your NAS setup as 2 volumes and run backups between the two arrays ( synology have good tools for this ) - ideally two separate units at separate places in your house. And you should think about off-site backup as well - either to cloud backup service, or just something as simple as periodically backing up to a couple of large external drives which you keep at a friend/familiy member’s house

 

[L&LD]

Particularly with important data, never take a working system down. Even more important when it is a large amount of it (in TB's).

Use what is working as a backup of the new, you're building, today.

 

[zero7404]

If one is purchasing/deploying a new NAS, don't use old drives... just asking for trouble a year down the road when drives start dying...

Invest in new ones, and match them up as a set

i have had fairly good experience with most spinning platters in the past, save for a few that went sour over the years.

i recently went to some expenses (over $1K) to have all the drives i have right now, and i am sure it's completely unnecessary to go thru that expense all over again for an occasion - like a new enclosure. the drives are babies, in light of what they were designed to do. all of them NAS or Enterprise-grade CMR platters and all of them have SMART health info that is spotless so far.

 

[John Davis]

i have had fairly good experience with most spinning platters in the past, save for a few that went sour over the years.

i recently went to some expenses (over $1K) to have all the drives i have right now, and i am sure it's completely unnecessary to go thru that expense all over again for an occasion - like a new enclosure. the drives are babies, in light of what they were designed to do. all of them NAS or Enterprise-grade CMR platters and all of them have SMART health info that is spotless so far.

we've got a bit of disk in use at work ( about 0.75pb ) spread across about 10 years worth of servers ( mix of synology rackmounts and old intel storage servers which have been gutted and upgraded with new raid cards etc).

Disks range from current gen ( mainly 16tb hc550s and exos ) right back to old 2tb/4tb sas units - but they're all enterprise grade, and what we've learned over the years is we get little to NO warning via smart of a unit going bad.

The oldest servers/drives are generally just running backups now ( so relatively light workloads compared to our production units ) but even then we still see failures - we're just about to pull one old 12 bay full of 4tb drives as we've already had a few of the drives in it die and it's obvious ALL those drives have a deifnite 'use-by' date ( they had 5 year warranties and were good up to then - now they're closer to 7 years and definitely not long for this world)

 

[zero7404]

yes you can SHR all those drives ( 2 * 6tb, 2 * 8tb and 2 * 16tb ) - however the actual capacity yielded will be kind of low ( it’ll create a 6 member raid5 array using 6tb partitions across all drives, then a 4 member raid5 array using 2tb partitions across the 8tb and 16tb drives and finally a raid1 array using 8tb partitions on the 16tb drives - then join them all together using lvm - shr is a nice term but it’s really just the linux md array driver plus lvm)

so actual usable capacity will be ~30tb+6tb+8tb=44tb

the first question is do you actually NEED all that space in one large volume - if not raid1 on the 2 16tb drives ( which are presumably your newest drives?) and shr across the 6tb and 8tb drives probably makes more sense ( one 16tb volume and one 20tb volume). Or even shr-2 (2 disk redundancy same as raid6 ) on the old/small drives.

If you want maximum resiliency just run raid1 on each pair ( 6tb + 8tb + 16tb volume )

As someone has already said, building arrays on old drives can be fraught - the risk is when you have a drive failure the increased workload of rebuilding onto a replacement drive will push another drive over the edge - and with raid5/shr only having single disk redundancy a 2nd drive failure during rebuild will result in COMPLETE loss of all data

And that leads to the second point - RAID is an availability tool ( i.e. your server can stay up and functional even when it looses a drive and whilst you rebuild onto a new drive ) - it does NOT replace the need for backups!!! At the least you should have your NAS setup as 2 volumes and run backups between the two arrays ( synology have good tools for this ) - ideally two separate units at separate places in your house. And you should think about off-site backup as well - either to cloud backup service, or just something as simple as periodically backing up to a couple of large external drives which you keep at a friend/familiy member’s house

noted on your points, thanks for the detail it is good to reference. it is also helpful trying to understand what i'm getting with SHR and SHR2. my initial notions is that there is some redundancy. i suppose that SHR doesn't qualify as having a true duplicate of a single drive in the array, should it fail ?

what i have right now is 'old reliable' .... basically a hardware raid1 for 2 identical disks, that serves as the primary backup of my entire NAS. relying on the reliability of the controller + the disks in this backup enclosure is basically what i am banking on right now.

the other thing i am risking is the raid0 configuration inside the NAS. should one disk fail, i am hosed. so some compensation for anticipating this is that i use a staggered interval for how i perform backups. i send backups of my PC to the NAS 2 times per week, and then manually run the rsync backup job from the NAS --> backup drives, also 2 times per week (but some time interval AFTER the primary PC --> NAS backup jobs).

i'm not made of money, but not poor either. a 1K investment in disks is about as much as i'm willing to stomach for my personal data safety. more than that and i'm just indulging in server speak, with no real justifaction for having server-grade equipment and data center-grade backup schemes.

if a Synology brings only just the convenience of checking SMART health on my disks without elaborate ssh or physically breaking an array .... i'm happy with that even if i don't use SHR. SHR i saw as a benefit to expansion in the future .... where 'if' 16TB suits me fine today ... it would be nice to know i can expand to another 16TB easily later on without too much headache and no need to buy new disks.

 

[John Davis]

noted on your points, thanks for the detail it is good to reference. it is also helpful trying to understand what i'm getting with SHR and SHR2. my initial notions is that there is some redundancy. i suppose that SHR doesn't qualify as having a true duplicate of a single drive in the array, should it fail ?

raid1 is easy to understand/visualise as it just mirrors all data to each drive in a paired manner. Higher raid levels are harder to get your head around - but yes, you have full redundancy ( either 1 drive in shr/raid5 or 2 drives in shr-2/raid6 ), so if a drive fails no data is lost

What you gain with higher raid levels is better efficiency - with raid1/raid10 your usable storage is 1/2 the total amount of drive space. With 'classic' raid5 (all drives the same size) that's much better - with a minimum size array ( 3 drives ) you've got 66% usable, with 4 drives that goes up to 75%, 80% with 5 drives etc. What normally limits practical raid5/6 array size is rebuild time - the older server we're retiring has all 12 4tb drives in a single raid5 array - incredible efficiency, but you don't want to know how long it takes to rebuild when a drive dies! Our new servers we're generally building 4-6 drive raid5 arrays of 16tb drives - any bigger and rebuild time again goes thru the roof. Since we have nightly backups to dedicated backup servers we can live with the level of risk with raid5 - otherwise we'd be running raid6 or raid10.

And yes, synology allows you to 'grow' existing arrays - so you could put the current 2 * 16tb in a 4-bay as a raid1 array, then later add another 16tb to go to raid5 ( taking it from 16tb usable to 32tb usable ) and then add a 4th to either go bigger again ( 48tb ) or increase redundancy by going from raid5 to raid6 ( although raid6 on a 4 drive array doesn't make much sense - might as well just run raid10 as it'll yield the same space and have better write performance )

 

[zero7404]

thank you everyone for the expert advice, will be digesting it over time to figure out what my next steps should be ....

it's nice to get expert and knowledgable feedback, especially on forums these days ...

 

[sfx2000]

i have had fairly good experience with most spinning platters in the past, save for a few that went sour over the years.

I went thru the Seagate 3TB mess back when I was managing core network hosts at telecom provider (modest size for the US, only 7M subs) - I had a 30 percent failure rate on those drives.

Spinning rust does eventually die - and since one of the key things about a NAS is to centrally host most of the files one would like to keep around...

Backups are handy - and something to consider with any NAS, as RAID is not a backup...