NAS and IP-surveillance storage

[JNA]

I'm currently working on a project regarding NAS and IP-surveillance storage, I'm testing different kinds of SOHO price ranged NAS with multiple camera setups ranging from 4-16 cameras on a Gbit-network, while monitoring and logging performance.

Since this is the premiere NAS-website I've found I thought this would be the place to go for questions and input.

What should I look out for when logging? Aka, should I log anything that isn't obvious at first glance?

I'm currently testing a Qnap NAS, and logging/monitoring with dstat, are there any other similar apps/software for other manafacturers' NAS?

 

[stevech]

Synology and QNAP each have a modest video archive/search/replay capability.

The archiving software from Axis, Pelco and others in that field are much more robust but costly.

FreeNAV might work too.

 

[JNA]

Maybe I should explain myself further, my explanation probably wasn't the best.

I already have a system for managing/viewing and setting up cameras and I'm currently trying to find the where the bottleneck lies when the system starts dropping frames when recording video to the NAS, looking around online this hasn't been done before at all except by the sureveillance company Intransa, they really don't go into detail what specific component, in general, is the culprit.

My goal is to try and pinpoint that component, so I'm mainly searching for tools that can monitor the stats on the NAS-system that can provide me with data to analyze. I'm currently testing this on a Qnap 219P II, and so far I've found dstat, htop, nmon and nload. Are there any similiar tools and availability on other NAS manafucterers' platforms?

The best would be if I could write my own tool but sadly I'm not that tech savvy.

Any advice on what to look out for when analyzing data and general advice would be appreciated.

 

[stevech]

Maybe I should explain myself further, my explanation probably wasn't the best.

I already have a system for managing/viewing and setting up cameras and I'm currently trying to find the where the bottleneck lies when the system starts dropping frames when recording video to the NAS, looking around online this hasn't been done before at all except by the sureveillance company Intransa, they really don't go into detail what specific component, in general, is the culprit..

Make/model of the DVR system you're speaking of, above?

 

[JNA]

I'm using Axis ACC

 

[egreen]

If you're talking about Linux, the single most useful tool we used when qual'ing hardware for camera capacity at Intransa was latencytop, followed by iostat. On the windows side of things, perfmon gives you a lot of good stats, queue depth is the most important one. dstat gives a nice overview but doesn't tell you about latency, which is the most important thing when it comes to dropped frames. The big problem with RAID6 is random writes, which require pulling a full stripe frame into memory, modifying it, and writing the changed disk sectors on three drives back out, basically reducing the throughput on random writes to that of one drive. RAID10 works much better in that situation, but of course wastes a lot more space. You may not think that video storage would do a lot of random writes, but the way that most VMS software stores data does have a significant random write component because they are continually appending data to the ends of multiple files.

Another issue a lot of people overlook is switch quality. One reason why Intransa shipped a SMC switch with every storage system was that we found that typical consumer switches simply cannot handle full speed traffic on multiple ports on a continuous basis, they're more for consumer type operation with short bursts of traffic. The end result with a consumer was typically pretty gruesome, the switch would start dropping packets and things... would... get... very... very... slow. Not to mention that under the most severe conditions the resulting dropped connections pollute the Linux network stack skbuf pool until the Linux storage system runs out of memory and either panics or just quits serving data. So make sure you have a good quality switch -- I generally recommend HP Procurve or Dell enterprise switches nowadays, but the latest Netgear "Pro" gear works pretty well too, SMC/Ericcson-LG seems to have pretty much given up selling switches directly in the USA -- or you may have "mysterious" issues even if your storage system is up to snuff.

 

[JNA]

Thank you for a really informative answer, I was quite stumped on where to look after testing the obvious (cpu, mem, net, disk sat/util/IO) since the information found online is very limited on this subject. The 3 intransa documents found was the only sources I found and they didn't go much into detail what they really found just that there was a problem.

Sadly there is no latencytop package for use in the IPKG manager that I'm using at the moment, and I doubt I could get one running manually, which probably means I need to install Debian on the NAS to use that tool. Problems are made to be solved so I need to find a solution for that. Another problem is how to monitor the frameloss, as of now I'm just observing visible drops when watching the recordings after capturing footage.

I'm currently using Netgear Prosafe switches and judging from the switch logs and my Wireshark captures indicates the net seems like it's rolling along nicely.

As I said initially in this post your post was very helpful as this subject is totally new to me, as is understanding how data flows through the system. You seem very knowledgeable on the subject, are there any other sources you could point me towards for information? and would you mind me PMing you if I have any questions?

 

[egreen]

The Intransa studies were done primarily for internal consumption, Intransa had an internal system configurator application that would allow SE's to punch in how many cameras the customer intended to support and what kind of bandwidth they were (VGA, 720p, etc.), then it would spit out how much and what kind of storage the customer needed. That required gathering a lot of data, Intransa also had a large camera simulator rack that would simulate hundreds upon hundreds of cameras. There was a guy who was full-time doing things like performance characterization. I was not that guy, though I discussed things with him because I was the platform lead responsible for making sure that the tools he needed were part of the platform. I guess I'm saying good luck .

 

[stevech]

If the camera video doesn't flow on WiFi, I'd doubt that the ethernet or switches are at fault. The camera's mpeg or equiv. streams use only a fraction of a 100BT ethernet LAN.
More likely, a software issue.

Try viewing software (from Axis) on a PC and see if there are any frame drops. The software might be transcoding and at times, the CPU load may slow things. Watch the CPU load display.

 

[BigAl.nz]

Mobotix

Might not be a fix for your current install but, Mobotix cameras write directly to the NAS. They also have a buffer built in to each camera.

The benefit is they dont drop frames.

I can reboot my NAS and still not drop any frames!!

Having said all that - the comments about switches, AND hard disk are valid. make sure you use Enterprise grade disks, ie Seagate Constellation etc.

HTH

-Al