Which way should I attempt to go - retail NAS or DIY?

[Melontes]

I would like to benefit from this forum's knowledge base, a selfish request, I know. My ultimate goal is to be able to download content, then store, organize, and update it automatically for eventual streaming to my devices. I want to be able to transcode the most demanding codecs (as I understand things). Please keep in mind that I am very new (read unknowledgeable) to this stuff but I am a bit stubborn and relentless...

I have read somewhat (I think a lot) on retail NAS units and DIY NAS units but I am very confused as to what items I really need to accomplish my ultimate goal. It is my understanding, and perhaps a false one, that retail NAS units come with many "apps" that are completely able to accomplish my goals. Please correct me if I am wrong on this one fact. Another understanding, and perhaps false as well, is that a DIY unit requires the use of VMs and a Server OS of some sort. Is this correct?

Initially, I was thinking to acquire/build a DIY NAS tower that would be able hold my storage needs with some kind of a RAID configuration. Recently, I have begun to think about an enclosure of some kind for my storage needs attached to a computer with the processing power needed for all the other stuff, but I am tending to think this might not be feasible. I think this way because of the interface (USB 3, eSata) speed concerns to the computer. Is it feasible? What about a Gigabit Ethernet connection to the enclosure (a NAS without a CPU)?

What about an enclosure that can handle its own raid via a web interface? (the Ethernet is not for data transfer)

Like this 8-bay enclosure for example (at a reasonable price):

RAID: 0, 1, 0+1, 3, 5, 6, JBOD, 10, 30
Interface: Ultra 320 SCSI <--------- (further explanation appreciated)
SATA: SATA I and II
Processor: Intel IOP80331 64-Bit RISC Processor
Memory: 512mb SODIMM
Management: Web Browser & SNMP via Ethernet Port​

The speed concern is another item that is confusing. What is the weakest (slowest) chain in the link, with the link being the entire system. With SATA II, SATA III, SCSI, USB 3.0, USB 3.1, eSATA. Ehternet, etc., etc.

I know nothing about Server OSs, VMWare, or ESXi. I also know very little about the retail NAS apps. Budget is a concern too.

I am thinking that 8 x 3.5" bays is what I'd like to have to start. Can SATA II ports support SATA III 8TB HDDs? I know about the drop in speed but haven't gotten a definitive answer to that question?

Given what I have stated above, what is my first priority? i.e. where do I start?

If you kind folks could answer my individual questions as it pertains to your own level of expertise, but PLEASE, if you use an ancronym not stated here, please follow it with the long version within parentheses.

If anyone wants to tackle how GPU transcoding would affects my system requirements, please be my guest!

THANKS IN ADVANCE!

P.S. - I have read some articles in this forum, but for some reason, they get technical really fast, and I am just not ready for that

P.S.S. - If this is posted in the wrong section, please move it accordingly.

 

[sfx2000]

If one is uncertain - probably better to go "off-the-shelf" with a retail NAS vendor - QNAP/Synology/AsusStor and others have some very good solutions here...

BTW - the specs you mentioned with the ex-Intel IOP chipset - that's a dead-end - they sold off the X-Scale biz to Marvell, and Qualcomm grabbed a lot of the ex-DEC guys that worked on StrongARM...

 

[Melontes]

Thanks, sfx...not able to afford those super NAS units.
And didn't understand your second sentence.

 

[ColinTaylor]

I think you are approaching the problem from the wrong direction. If you're just wanting something to store and stream your pirated movie/TV collection you're probably looking for a server with sufficient processing power for the transcoding, fast network connections and sufficient local storage. It makes no sense to store the source material on a NAS, only to have to move it across the network to transcode it and then send the result back out across the network. Of course the server can also act as a general purpose NAS for the rest of your network.

I suggest you think about requirements for CPU, RAM, number of concurrent users, bandwidth, software (DLNA, Plex, etc?) as well as storage capacity.

 

[sfx2000]

Extended post I wrote last spring... Repurpose an old PC, or build from scratch...

(yes, it's a PDF, and yes, it's safe...)

 

[Cake]

question, what are your devices? phones and wireless stuff? I suggest avoiding DLNA.
I have good luck with kodi on raspi B's, however they can not decode x265 codecs. Every TV of mine has a raspi behind it that connects to my low powered NAS. It is really smooth. You can set up kodi to use a central database for thumbs, mysql for remembering where you last stopped watching a show....
I am guessing you want to re-encode some older codecs so you can stream? I think you might need a higher end cpu in the box if using DLNA to handle multiple connections, especially with 1080 or the newer 4k. IMO I would avoid it and install kodi type stuff, or a video/audio player that can fetch files on the lan and decodes it. Stick to x264 codec and 1080 or smaller.

 

[Melontes]

you're probably looking for a server with sufficient processing power for the transcoding, fast network connections and sufficient local storage. It makes no sense to store the source material on a NAS, only to have to move it across the network to transcode it and then send the result back out across the network. Of course the server can also act as a general purpose NAS for the rest of your network.

I suggest you think about requirements for CPU, RAM, number of concurrent users, bandwidth, software (DLNA, Plex, etc?) as well as storage capacity.

I agree Colin. Transcoding and storage must be the closest possible. A super NAS makes this easy, but as I have said a few times now, they are super expensive. Perhaps a DAS connected to a good computer. If I had to run a few virtual machines (apparently I must) and stream from a maximum of two clients at one time (very seldom), what would my CPU and RAM requirements be?

What are the pros and cons of having an 8-bay DAS connected to the computer instead of buying a computer tower with 8 bays and enough SATA ports on the motherboard? If we could flesh this one point out, it would make me very happy!

 

[Melontes]

question, what are your devices? phones and wireless stuff? I suggest avoiding DLNA.
I have good luck with kodi on raspi B's, however they can not decode x265 codecs. Every TV of mine has a raspi behind it that connects to my low powered NAS. It is really smooth. You can set up kodi to use a central database for thumbs, mysql for remembering where you last stopped watching a show....
I am guessing you want to re-encode some older codecs so you can stream? I think you might need a higher end cpu in the box if using DLNA to handle multiple connections, especially with 1080 or the newer 4k. IMO I would avoid it and install kodi type stuff, or a video/audio player that can fetch files on the lan and decodes it. Stick to x264 codec and 1080 or smaller.

Cake, Sorry, but have no idea what a raspi b is...thumbs?? MySQL???
DLNA not viable...1080P is what my TV is and will be for a long time (I hope)
A listing of my devices will be forthcoming as soon as I catch up on the replies...

 

[Melontes]

Extended post I wrote last spring... Repurpose an old PC, or build from scratch...

(yes, it's a PDF, and yes, it's safe...)

Sfx...yousers, man! I am still in the womb for your comments...maybe in 6 months or so it will make sense.

 

[Melontes]

My current set-up:

Upstair’s office: Router (Asus RT-AC68U), temporary budget NAS (Asustor AS-202TE) [TO BE REPLACED WITH TBD PRODUCT]

Downstair’s Rec Room: Panasonic Viera Plasma TC-P60ZT60, Yamaha A/V Receiver RX-V671, and other stuff.

(Other stuff = Cable Box –HDMI, Blu-ray player-HDMI, Roku 3-HDMI, Wii-HDMI, and a fried xBox 360-Component)

From the upstair’s router, I have connected Cat6E Ethernet to an 8-port gigabit switch behind the TV and other stuff. Everything than has an Ethernet port is connected to this switch, which includes the TV, receiver, and a bunch of the other stuff. All video content goes into the receiver via HDMI and then thru to the TV via HDMI.

Right now, temporarily connected to the office router is an Asustor 2-bay AS-202TE NAS which has Plex Media SERVER installed on it. The Plex Media PLAYER is installed on my Gateway NE522 laptop. When I connect my laptop to the LAN and to the receiver via HDMI, it seems like I have no problem playing 1080P content from my 3TB HDDs inside the Asustor NAS via the installed Media Player on the laptop. I have not tried to play anything wirelessly (and really have no desire to). This setup seems a bit clunky but I could not tell you why…no one seems to mention exactly how the physical devices should be set up in conjunction with each other.

How should the physical devices be set up?

Phones and tablets aren't an issue at this time.

 

[ColinTaylor]

I agree Colin. Transcoding and storage must be the closest possible. A super NAS makes this easy, but as I have said a few times now, they are super expensive.

That's the first time you've mentioned it. What is your budget?

Perhaps a DAS connected to a good computer. If I had to run a few virtual machines (apparently I must)

There's no particular reason to run VM's. I just use a 4-bay micro server running native Linux.

and stream from a maximum of two clients at one time (very seldom), what would my CPU and RAM requirements be?

Here's the most important part if you're transcoding HD video***. If we're talking PC-like specs I'd say 8GB of RAM and any reasonably modern CPU (Intel i3 or above). Note that these specs are easy to achieve with any PC but be wary of appliances like your Asustor because the cheap ones have weak processors (Marvell, Intel Atom, etc.) that don't have the processing power for transcoding.

What are the pros and cons of having an 8-bay DAS connected to the computer instead of buying a computer tower with 8 bays and enough SATA ports on the motherboard? If we could flesh this one point out, it would make me very happy!

How the storage is attached is largely irrelevant. First you need to determine what capacity you want, now and in the future. Then decide on the hardware. Why 8-bay, why not 4 or 2 or 16?

*** You said at the very beginning that "I want to be able to transcode the most demanding codecs". When you later described your setup, you appear to only have 2 display devices. One is a Roku attached to a TV. In this case the Roku is doing the transcoding itself. The second is a laptop running Plex client. The Plex server is almost certainly using Direct Play or Direct Stream to send the video. This avoids the need for the Asustor to transcode and keeps the CPU demands to a minimum.

Finally, you said "My ultimate goal is to be able to download content, then store, organize, and update it automatically for eventual streaming to my devices". Do you have any idea how you will do this? If you have some preferred software package, that might dictate what hardware you need.

 

[Melontes]

Colin, thanks again for taking the time...

Budget is $500

I thought (or have read in other forums) that running VMs was necessary for apps like CouchPotato, and a few others that venture forth and retrieve. It was also pointed out that running these VMs would require additional CPU power and RAM. Unfortunately, I know nothing about this area...I am just reading and attempting to memorize in the hopes that it will one day make sense.

Yesterday, I was informed that transcoding involved "dummying down" the original codec so that a device could handle it (like H.265 to mp4). Is this correct?

I am aware of the budget NAS's lack of performance. This is one of my main concerns. The good NAS boxes are too expensive. Hence my inquiry into large DAS enclosures...attached to another (TBD) computer.

"Why 8-bay, why not 4 or 2 or 16?" 8-bay was just a guess. Once I start to gather content, I am sure the storage will quickly increase. Hopefully, it is a bit of future proofing. 8 X 8TB is a fair bit of info...the enclosures should last a fairly long time, especially if they do not have their own RAID - again, my guess. Please correct me if I am wrong. I need a basis of understanding!

I would rather get an 8-bay enclosure and populate it with my 4 X 3TB HDDs and gradually add more to the RAID as I go along...and then eventually increase the capacity of the drives as well. What this would do to the RAID is unknown to me 'cause I don't have much understanding in that area as well. I suppose I could go out and get another piece of crap and learn the hard way and spend well over my budget experimenting, but that is why I am here. I want to start right and waste as little cash as possible.

You said, "The Plex server is almost certainly using Direct Play or Direct Stream to send the video." You are not sure? I was under the impression (rightly or wrongly) that the Plex Media Server was doing the work on its own. How does Direct Play or Direct Stream come into this? Where do they come from?

I thought the individual virtual machines + scripts are the "software" to download, oraganize, etc.

I just wish I could get some definite, black and white basic understanding of what is needed. It seems that everything from all sorts of sources is in disagreement with each other and it is driving me mad.

What is my first step?

 

[ColinTaylor]

I just wish I could get some definite, black and white basic understanding of what is needed. It seems that everything from all sorts of sources is in disagreement with each other and it is driving me mad.

What is my first step?

That's the $64M question isn't it? Unfortunately there is no "correct" answer. In a sense there is too much choice and everyone has their own opinion. This is why it is important to be absolutely sure of exactly what you want to achieve, before going down the wrong path.

For example, you say that mobile devices are not a concern at the moment, but will they be in the future? Conversely, creating at setup that works well for mobile devices might compromise the functionality of your wired devices.

You say that your current setup works well. So what is it that you are unhappy about? Having the Roku doing all the hard work (transcoding) for the TV is an excellent choice, so no need to change that. (Having a Raspberry Pi micro-computer running Kodi is basically a DIY Roku ).

Plex is an excellent product (but nothing is perfect). The Plex server streams video to the client in one of three modes, Direct Play, Direct Stream or Transcoding. The first 2 modes require very little CPU power for the server as the data is sent more-or-less "as is" to the client. This relies to the client being able to natively play the video. This is not normally a problem for PC's. If the client device doesn't support the video format/codec (think Blu-ray players, phones, tablets) then the server has to transcode the video into a format that the client can understand before it is sent. This is extremely CPU intensive.

I haven't used couchpotato or similar products so I can't comment on them. Plex as you know is excellent at identifying source material and presenting it in an organised manner, even if the actual files are just chucked into a single folder . Last time I looked some of the Plex clients cost money, but the Linux server was free.

 

[Melontes]

Keep up the good work, Colin. You are able to speak my kindergartenese...

Phones and tablets do not play into this at all...other than using my android phone as a mouse for my laptop.

I am not using the Roku at all. The interface (Plex channel) is terrible.

I would rather not do video streaming wirelessly, except maybe for audio.

Yes or no, please. As long as the Plex Media PLAYER is on my laptop and is linked to the Plex Media SERVER on my NAS/Enclosure, there should be no problem playing 1080P Blu-ray videos on my TV.

I am getting very mixed up on the word "server" when used as a one-word construct. I don't know if you are talking about Plex Media Server software installation, the computer server, the NAS as a server, or something else. This is why I painstakingly use long hand for all my comments.

I am also mixed up as to what the "client" is. I thought the Plex Media PLAYER was the client to Plex Media SERVER.

1. Further to my question in red...this setup works okay on content that has already been retrieved. But I need a set-up in which FUTURE retrieving is done automatically. But since I can't get a definite answer as to how this gets done, it is utterly impossible for me to figure out the hardware requirements, and this is so #%^&*( frustrating!

2. Also further to my question in red. As my content increases, I need to be able to better preserve my data. This brings about the question of a good RAID system. Budget NAS boxes are not very powerful in this area...OR ARE THEY? Making sure I have adequate RAID control also adds to the hardware requirements, right?

I am thinking of getting a non-RAID enclosure. That way, any economical enclosure should do the trick. IS THIS CORRECT

What is the best interface to connect the non-RAID enclosure to the computer that will handle the RAID configuration?

Does the enclosure need to be connected DIRECTLY to the LAN or will the computer it is connected take care of that?

Which RAID "system" is economical and good enough for my needs (think 8-bay 8TBs). The DIY NAS people have me spending over $300 on a RAID card along with a $200 motherboard, and, and, and...there must be a more economical way.

All the Plex objects that I have mentioned are free.

I am almost ready to give up.

I must have answers to the questions in red to establish some kind of base from which to proceed from.

 

[ColinTaylor]

I am not using the Roku at all. The interface (Plex channel) is terrible.

Sorry, I thought the Roku could pick up and play video content stored on the local network (SMB shares). Apparently that is not the case. Apologies for the confusion. So that presents a thorny problem of how to stream stuff to your TV with a user interface you find acceptable. I note that your Viera supports DLNA which is the ubiquitous standard. Have you tried that? DLNA interfaces tend to be pretty basic though.

Yes or no, please. As long as the Plex Media PLAYER is on my laptop and is linked to the Plex Media SERVER on my NAS/Enclosure, there should be no problem playing 1080P Blu-ray videos on my TV.

Provided that your laptop can play the video files if they are stored on it's local hard drive then there's no reason to think they won't play when streamed from the Plex Media Server. (Although nothing is ever 100% guaranteed).

I am getting very mixed up on the word "server" when used as a one-word construct. I don't know if you are talking about Plex Media Server software installation, the computer server, the NAS as a server, or something else. This is why I painstakingly use long hand for all my comments.

I am also mixed up as to what the "client" is. I thought the Plex Media PLAYER was the client to Plex Media SERVER.

I'll try to be more clear, but I think you've got the idea. The server hardware is usually just a box (usually without a screen or keyboard) that is connected to your network that "serves up" some resource, usually files. The server software would be the software packages that run continuously on the server hardware, in your case Plex Media Server and quite possibly other things like Samba for creating Windows network shares.

Client devices are things like TV's or PC's, and usually run client software, like the Plex Media Player. Just to make things more confusing it is possible to install some "server software" like Plex Media Server on a client device like a Windows PC. Of course this isn't that useful unless you want to leave your PC on all the time.

This brings up an important question. What is you skill level/knowledge area? I'm sure there are a lot of people here (myself included) that might want to suggest that you just buy on old PC, install Linux, configure your software and off you go. Simple, right? Maybe not for you? More familiar with Windows? All the software you talking about is probably available for Windows and you might already have a PC lying around. All you need to do is attach... more... storage... to... it... Doh!

1. Further to my question in red...this setup works okay on content that has already been retrieved. But I need a set-up in which FUTURE retrieving is done automatically. But since I can't get a definite answer as to how this gets done, it is utterly impossible for me to figure out the hardware requirements, and this is so #%^&*( frustrating!

This is not something I have any experience of I'm afraid. I'm also not sure whether it is a suitable topic for these forums.

2. Also further to my question in red. As my content increases, I need to be able to better preserve my data. This brings about the question of a good RAID system. Budget NAS boxes are not very powerful in this area...OR ARE THEY? Making sure I have adequate RAID control also adds to the hardware requirements, right?

Think very carefully here, you can end up spending a lot of money for no good reason. First, what are you really protecting. When a hard drive fails, which it will at some point, without redundancy or backup the data is lost. If the data is just a bunch of TV shows and movies why not just download them again once you've replaced the faulty hard drive? If the data contains precious irreplaceable home movies you'd be foolish not to have another copy backed up in another location (so you could restore from that copy).

I am thinking of getting a non-RAID enclosure. That way, any economical enclosure should do the trick. IS THIS CORRECT

What is the best interface to connect the non-RAID enclosure to the computer that will handle the RAID configuration?

Does the enclosure need to be connected DIRECTLY to the LAN or will the computer it is connected take care of that?

I think you might be confused about what a RAID enclosure (like the one you described earlier) is. It is just a bunch of storage, albeit with hardware redundancy. Think of it as one giant hard disk. It has no network capability itself, it has to be locally attached to some sort of computer. If that computer happens to be a NAS, then it is the NAS that is "serving up" the storage to the network, not the enclosure.

Which RAID "system" is economical and good enough for my needs (think 8-bay 8TBs).

RAID 5 should be more then adequate.

The DIY NAS people have me spending over $300 on a RAID card along with a $200 motherboard, and, and, and...there must be a more economical way.

That seems rather overkill for a bunch of videos. If you were storing your company database then fair enough. Modern operating systems like Linux (and probably Windows) have the ability to do RAID in software (for free). This is probably more than adequate for your needs. Sure, it's not as fast or bullet proof as a hardware solution, but it's a lot cheaper.

 

[sfx2000]

Sorry, I thought the Roku could pick up and play video content stored on the local network (SMB shares). Apparently that is not the case. Apologies for the confusion. So that presents a thorny problem of how to stream stuff to your TV with a user interface you find acceptable. I note that your Viera supports DLNA which is the ubiquitous standard. Have you tried that? DLNA interfaces tend to be pretty basic though.

Roku can't do SMB mounts - but they do support DLNA...

https://support.roku.com/article/20...a-player-to-play-your-videos-music-and-photos

 

[Melontes]

I tried playing a few videos using DLNA from my Blu-ray. It worked once (I think) but once I added my content to the already existing content (bringing it up to just over 3TB) the DLNA wouldn't even connect. Haven't tried it via Roku, but that doesn't matter. I need to be able to view it with the filters (genre, name, etc.) I can open VLC and track it to the HDDs on the NAS, but sometimes it takes a long time to load the list, but again, I need the filters...

You said, "This brings up an important question. What is you skill level/knowledge area? I'm sure there are a lot of people here (myself included) that might want to suggest that you just buy on old PC, install Linux, configure your software and off you go. Simple, right? Maybe not for you? More familiar with Windows? All the software you talking about is probably available for Windows and you might already have a PC lying around. All you need to do is attach... more... storage... to... it... Doh! "

Hmmm...well, I do have a bunch of PCs lying around, Some IDE and maybe a couple of SATA I (maybe a SATA II but I doubt it). I was re-assembling them, adding software, and putting them back on Kijiji...nothing major, upgrading HDDs, adding a burner, and adding RAM - I have yet to experience a motherboard and CPU install. I have done some Visual Basic programming, a few Access databases, dabbled in Dos.

Re-downloading hundreds of movies would be extremely time consuming.

I likely messed up by saying that an enclosure could be hooked up to a computer by Ethernet. I was thinking that NAS meant it had a CPU to do RAID and have apps, etc (and it may have) but it simply means "Network attached storage." I guess a RAID enclosure is a plain enclosure with disks that has become subject to some kind of RAID control (either from the enclosure itself or by the computer it is attached to. Is this correct?

And when I asked about what RAID system I should get, I meant on-board integration, PCI RAID card, or RAID assist, not a particular RAID configuration. So I need to establish what kind of RAID "system" I need...

I need to get started.

I will acquire an 8-bay barebones enclosure without any RAID ability by the enclosure. I will let the computer it gets attached to decide the RAID control.

OR should I get an enclosure that offers some RAID functionality? Could the RAID be disabled if I decided to go with the main computer

Let's get this one aspect figured out. Let's dwell here...

Accepting all recommendations for enclosure makes and models. You certainly may add your pros and cons with the recommends...all ears are open.

 

[ColinTaylor]

Hardware RAID (assuming it's reasonably modern) is preferable to software RAID because the storage unit isn't reliant on a host computer to work. Software RAID can lock you into a particular operating system. For example, if you configured your enclosure using Linux's LVM but later wanted to connect it to a Windows PC, it's likely that Windows would not understand what it was seeing. Hardware RAID gives you a certain amount of independence, but usually costs more.

As you have no idea what type of computer you will eventually be attaching it to you'll have to play safe with the choice of external interface. Don't buy old kit that uses SCSI interfaces, you'll end up scouring eBay looking for HBA's and the drivers probably won't exist for anything more recent than Windows 95. Ideally something with eSATA or USB 3.

 

[sfx2000]

Hardware RAID (assuming it's reasonably modern) is preferable to software RAID because the storage unit isn't reliant on a host computer to work. Software RAID can lock you into a particular operating system. For example, if you configured your enclosure using Linux's LVM but later wanted to connect it to a Windows PC, it's likely that Windows would not understand what it was seeing. Hardware RAID gives you a certain amount of independence, but usually costs more.

HW RAID locks one in to a specific controller - and even there - some of those are fake-raid dependent on OS level drivers...

These days - SW RAID is more adaptable and flexible across different OS's - and CPU's are still faster than many ASIC oriented host controllers...

mdadm/lvm are your friends here... and then one can layer on whatever filesystems one wants....

 

[Melontes]

Hardware RAID (assuming it's reasonably modern) is preferable to software RAID because the storage unit isn't reliant on a host computer to work. Software RAID can lock you into a particular operating system. For example, if you configured your enclosure using Linux's LVM but later wanted to connect it to a Windows PC, it's likely that Windows would not understand what it was seeing. Hardware RAID gives you a certain amount of independence, but usually costs more.

As you have no idea what type of computer you will eventually be attaching it to you'll have to play safe with the choice of external interface. Don't buy old kit that uses SCSI interfaces, you'll end up scouring eBay looking for HBA's and the drivers probably won't exist for anything more recent than Windows 95. Ideally something with eSATA or USB 3.

Nice tip on old units and drivers...

I am still a bit lost on RAID. I understand the configurations (RAID 0,1,5,6,10, etc). I do not understand the hardware or software side of things.

(Am reading this article)

Quick question to all...a barebones DAS enclosure can be connected directly to a hardware raid controller in a PCI slot? Probably a stupid question but I have never seen a hdw raid controller or a DAS...I am guessing that it must be a yes. And if so, what type of interface is supplied by the hdw raid controller?

What about ROC (RAID on Chip) ??? How does this work with the DAS?

I am pathetic...