NAS advice

[Neverland]

Hello, first of all this forum seems to be amazing with advice... First I was going through posts to see what would work for me, but I think I might be better off laying out what I intend to do and listen to the advice

I was looking for a solution for my newly setup business. I have a few requirements

I run a small hedge fund with low frequency trading. Currently the hardware configuration is around several cheap small unit servers (Rasberry Pi likes) that each monitor 1 strategy. They operate all off the same code library and data infrastructure. Currently I have about 10 of those running and they all mounted the same drive from a NAS to get position coordination. In a mature setup, this might increase to multiples of this.

1) Data security: much of my business is around my own intellectual property, which is why all models are centralised onto my NAS
2) IO speed: a lot of information gets read from files, whether it be model information or small sets of data. Nothing is really huge amounts of data that is read in 1 go, but rather many many calls to small subsets. Currently the NAS I have is not fast enough to run a DB on so all data has been stored in a binary format as a file and gets read.
3) I do run into problems with too many servers reading the same data, so I need to go to a DB solution quickly

So I was thinking of getting a good NAS (but open to challenge if that is the right solution) with eg 8 or 10 bays.

4 (or 6 if I get 10 in total) of those I was thinking of running in raid 10 setup with SSD. I wouldnt need so much disk space actually, so would probably take smaller SSD drives, eg 256 GB each.
the other 4 I would run in raid 6 and would install the backup software for the 4/6 critical drives. Those could be 4 or 6 TB drives.

I currently have 2 very old synology NAS systems. One is a 1 bay (that I would try and sell) and the other one is a 2 bay. I would sell this if I get something for it, or use this one as the back up server instead. Both come with 1 TB drives (so 3 in total) as well as 2 USB drives plugged into them for extension (1TB each). Currently I have them pumped with video and music and if I upgrade to a new one, I think I would use the large volume in Raid 6 to copy all my stuff onto for my home media server. All this is being accessed by a separate streamer (use Amazon fire + RPI Kodi for that). This is not critical but would just give me a bit of space and less equipment in the house.

So the main use will be a DB setup that will service many small LAN based servers as well as a file server for the actual models as I don't want to copy or run anything from the local small servers as they need to remain plug and play, so all is to be centralised.

I read a lot online so far, and Synology seems to be a bit weaker with the Atom processors. I don't want to spend money today and then run into issues or need an upgrade later. Better to be spending once and go for the real stuff.

So I was looking for recommendations on anything that comes to your mind, but given that eg. the TVS-EC1080+ is quite expensive, I prefer asking advice...

Thanks for your time

 

[sfx2000]

I read a lot online so far, and Synology seems to be a bit weaker with the Atom processors. I don't want to spend money today and then run into issues or need an upgrade later. Better to be spending once and go for the real stuff.

A lot of folks might have had bad experiences with the first generations of Atoms (anybody remember the netbook craze a few years back) with Bonnel cores - the current Silvermont cores perform much, much better on that level - esp. the Celeron J1900 Baytrail-D Quads, along with the C2558 Rangley Quads.. the performance on them belie the name, and in some use cases, they're just as capable as their bigger, more power hungry elder cores..

Running a DB server, that might be justification moving from the small cores over to the big Haswell/Broadwell based options, if running the DB server on the NAS, as they can generally utilize more RAM.

 

[stevech]

My DS212 Synology isn't Atom, isn't a real fast CPU. But I use it as a NAS, not an app server. So speed is usually constrained by PC file system overhead and the LAN speed.
It has been problem free since 2012, and it runs the latest DSM OS upgrades minus 1 (!).

When I buy a new one, it'll likely be Synology or QNAP, though I'd look at how ASUS is maturing their NAS.
The retail hyped ones, I'll again take a pass on them.

 

[Neverland]

Thanks for the responses so far. I am inclined to go with Qnap so far.

I forgot to add that I will also use the Cloud software to sync a whole bunch of files

 

[stevech]

Thanks for the responses so far. I am inclined to go with Qnap so far.

I forgot to add that I will also use the Cloud software to sync a whole bunch of files

No doubt, you've Synology and QNAP's on line demos. Synology's new software DSM 6.0 is on now.

 

[sfx2000]

Before you jump - QNAP and Synology have excellent pre-sales teams that will ensure that what you purchase meets your needs - so kick the tires on the online demos, and see what works, and then reach out.

 

[stevech]

I forgot to add that I will also use the Cloud software to sync a whole bunch of files

I tried most all low cost cloud file servers - about 8 of them that are US-centric. Used some for long time (e.g., OpenDrive). Settled on ADrive and their 100GB for $25/year. Been using it a year now. Good.

I use cloud only for the "1" in the 3-2-1 backup strategy, and use cloud ONLY for VIP data EXCLUDING personally sensitive information, e.g., no financial data, no medical. Family photos, etc. Risk is disgruntled employee/contractor. Happens several times a year.

Upload speed of ISP is also a limitation, though some costly cloud services do allow you to pay them to receive a USB drive with GBytes on it, and upload one time.

You could use, say, Safehouse Software or winzip, with strong encryption, to create a big file of sensitive info and upload that to the cloud - as the service cannot know the key. Most cloud services encrypt before transmittal over the internet, decrypt, the reencrypt for storage. They try to control those 2nd set of keys. And compliance with a court order uses the 2nd set of keys.

But my "1" in 3-2-1 is a 2TB 2.5 inch USB3 drive/enclosure. Fast. I control it.

 

[Neverland]

Before you jump - QNAP and Synology have excellent pre-sales teams that will ensure that what you purchase meets your needs - so kick the tires on the online demos, and see what works, and then reach out.

Thanks i will try that

 

[kamina]

I used to have a Synology, now have Qnap. I prefer the Qnap management UI and it's plenty fast. It seems in some ways to be more featured from a home "do it all" angle.

However I had some issues where the whole RAID just melted after being used for a while. After resetting, installing new firmware and rebuilding the disk it's worked ok, but somehow that seemed strange as it was pretty much using stock settings with one share and a few terabytes of files copied over. I also have two colleagues with Qnaps and both had issues with firmware (one while flashing, other had the raid go bad like me).

I also have colleagues with Synology and nobody has ever had any issues there... Luckily I don't have to decide on what I'll get next, so Qnap has plenty of time to win me back but if I was deciding today I would probably get a Synology.

Whatever you get, I'd see that you can expand the ram. If you are mainly having a few files accessed then they should end up cached in ram so it's not hitting the disk all the time. Intel CPU's are generally better with hard IO then arms, and of course SSD can make the biggest difference if it does not come from cache in memory. Both Qnap and Synology offer the possibility for SSD cache, but comparing them is not very easy. From Synology white papers I got the feeling it would work best as a block device (iscsi) which could be a good idea for you anyway as then the clients might end up caching it it memory, I got the feeling Qnap's solution is different and would work better via smb (but documentation is vague at best).

 

[Shabbir Rao]

Well of course according to the requirements mentioned above you should go with QNAP along with any good third party back up software available in the market. Third party software and QNAP makes a good back up combination because in QNAP creating packages with these third party software is quite simple and workable.

 

[kamina]

Well, I think the requirements are quite simple and I don't see why Qnap would be clearly better then Synology or others... There's a lot of details missing from the original post, but here is a bit more comments:

- If the raspery pi's are running Linux they could quite easily use iScsi which could really fit this well (who has better iscsi)
- Having a separate volume for backups is a good idea for starters, but I always remember a friend who I finally managed to talk into taking backups. His office was burgled and they stole all the computers and external drives including the ones with the backups. That was a bleak start for a working week... Personally I keep one copy of backups off site
- You don't mention what kind of DB you would be using. We have lot's of old legacy servers working with binary data and small IO (millions of clients reading mainly very small binary files over a custom protocol) and the key is getting the files that are being read to be cached in memory. We also run a lot of various DB's and it's not actually so different; if the clients don't lock each other out then you can handle a lot of reads (like millions of rows per second) if the commonly accessed rows fit in memory. For something like this I would rather consider running a VM on the NAS they running the databases there unless you can manage with the default config the NAS comes with...
- If all the clients have read / write access to the NAS share with all critical data and they work with different datasets then you might not be automatically increasing security by centralising all the data. Sure it will be easier to handle and coordinate from one central space, but it also means any client can read or delete the other clients data.

 

[Shabbir Rao]

Well you have a lot of options for this. How about you go for a public cloud? Private cloud is better and secure of course but it is costly as compare to the public cloud. You can try Syncribox for public cloud or you can check out Dropbox for business as well both are very good to use as I am using both. As for private cloud you can check out QNAP or Synology. I am using both of them and both are good to use and easy to setup along with my NAS devices I am using Syncrify. It is a back up software with all the latest back up features which includes Incremental back up, Remote access, Schedule back up and many other useful and most importantly workable features. Moreover it is also compatible with any known OS available in the market I am personally using it with both Linux and Win and I am not having any problem so far. Plus the best of part about Sycnrify is that it is a cost effective workable back up solution as compare to other like products available in the market these days.

 

[sfx2000]

- You don't mention what kind of DB you would be using. We have lot's of old legacy servers working with binary data and small IO (millions of clients reading mainly very small binary files over a custom protocol) and the key is getting the files that are being read to be cached in memory. We also run a lot of various DB's and it's not actually so different; if the clients don't lock each other out then you can handle a lot of reads (like millions of rows per second) if the commonly accessed rows fit in memory. For something like this I would rather consider running a VM on the NAS they running the databases there unless you can manage with the default config the NAS comes with...

Hmmm... something I hadn't considered before, but now since QTS 4.2.1 supports Docker/LXC, along with the KVM/QEMU - running some mysql benchmarks against native (mysql is a qpkg) vs. a Docker container vs. Debian/Ubuntu/Centos/Fedora on a VM... this might evolve into it's own thread ')

 

[kamina]

Hmmm... something I hadn't considered before, but now since QTS 4.2.1 supports Docker/LXC, along with the KVM/QEMU - running some mysql benchmarks against native (mysql is a qpkg) vs. a Docker container vs. Debian/Ubuntu/Centos/Fedora on a VM... this might evolve into it's own thread ')

I was just testing generating some stats based on a DB dump on my work computer (mbp, quad core, 16GB ram) and found it to be faster in a Linux VM with sufficient memory... So you never know.

 

[whsbuss]

One of the places I volunteer at (animal shelter) is looking to get their technical act together. They have 2 desktop PCs that they use for all their stuff..... email, MS office suite, access database, quickbooks, and more. One of the PCs is the main office computer that most of the work is done on. The other is just an overflow for the other officer workers.

I know I could deploy a NAS and setup their PCs for auto backup. But I was wondering what would be a good NAS for having each PC access the files from the applications over the local LAN? For example, keeping the access databases, word files, quickbooks database on the NAS and having the PC access the appropriate file when working in the app.

I use a MyCloud at home because mine is about sharing and backup storage, including family members who have their own shares. At the shelter that would not be required.

 

[sfx2000]

I know I could deploy a NAS and setup their PCs for auto backup.

NAS's are handy for that - but also remember to have a backup strategy for the NAS... a lot of people seem to miss this one...

RAID is nice, but one of the issues with RAID - disks fail - without RAID, they'll fail and hopefully a backup is current and available.

The risks increase with every disk added to a RAID (Mean Time Between Failures) - so 4 disks, more risk - and then it's a statistical exercise from there.

RAID10 is the safest, RAID5 was next, followed closely by RAID1 - RAID0, while very fast, had a 100 percent chance of failing suddenly...

Always have a backup of that NAS...

 

[stevech]

preaching: 3-2-1 backup. NAS or not.

 

[whsbuss]

NAS's are handy for that - but also remember to have a backup strategy for the NAS... a lot of people seem to miss this one...

RAID is nice, but one of the issues with RAID - disks fail - without RAID, they'll fail and hopefully a backup is current and available.

The risks increase with every disk added to a RAID (Mean Time Between Failures) - so 4 disks, more risk - and then it's a statistical exercise from there.

RAID10 is the safest, RAID5 was next, followed closely by RAID1 - RAID0, while very fast, had a 100 percent chance of failing suddenly...

Always have a backup of that NAS...

I have a NAS here on my home network..... LAN backup to another NAS weekly, monthly USB backup and taken off-site.

 

[stevech]

I'd backup a bit more frequently, unless you have something else at work, as I do.

The forums are replete with users' horror stories about RAID, for reasons other than drive failure, failing to rebuild and user had no backups. And there's theft or human admin error.

 

[sfx2000]

http://storagemojo.com/2007/02/26/netapp-weighs-in-on-disks/

Interesting read - not the blog post - but the netapp discussion within it - (NetAPP builds big enterprise grade SAN's, and they have a lot of data on drives, RAID, and failure modes)