CCTV recording server failover: an acceptance worksheet

High-availability claims are easy to place in a CCTV proposal and surprisingly easy to misunderstand. A second server may protect the recording service, but it does not automatically duplicate old archives, eliminate every transition gap or guarantee that all cameras can reconnect under peak load. The acceptance test must therefore measure what happens to live view, recording, playback, alarms and retained video during both takeover and failback.

1. Define the promised recovery outcome

Start with the operational requirement, not the server count. Record which failure is being protected: recorder service crash, operating-system failure, server power loss, storage failure, access-switch loss or complete room outage. A standby server in the same rack does not protect against every one of these events.

2. Size the standby for the failure domain

Do not compare only the standby's licensed camera count with the cameras on one primary server. A shared standby may need to absorb more than one failure, especially when the protected servers share a rack, UPS, switch or storage system. Include the recorder's existing workload if it is not dedicated.

Use measured or conservatively specified peak recording bitrates. Variable bitrate cameras, scene motion, audio, metadata and secondary streams can make an average figure too optimistic. Genetec explicitly advises load planning for the extra cameras and warns that other roles on a standby can reduce its archiving capability. Hanwha's WAVE guidance similarly makes failover dependent on spare device capacity.

3. Understand what failover does not protect

Milestone documents that clients may temporarily lose live and recording access during takeover, and that older archives held only on the failed recorder are unavailable until it returns. Genetec distinguishes role failover from redundant archiving: a standby can restore the Archiver role, while simultaneous redundant archiving creates another copy of video. These are different controls.

• Service availability: another server takes responsibility for cameras.
• Archive redundancy: another independent copy of recorded video exists.
• Gap recovery: camera edge storage fills video missing during transition.
• Disaster recovery: configuration, keys, databases and archives can be restored after a wider site loss.

4. Measure the central recording gap

Use at least three representative cameras: one continuous, one motion-based and one high-bitrate or high-priority camera. Place a clock or a repeatable moving subject in view. Note the last centrally recorded frame on the primary, the first frame on the standby and the first merged or retrieved frame after recovery.

Keep the two figures separate. A VMS timeline that later becomes continuous after edge retrieval does not mean takeover itself was gap-free. Milestone states that cold standby and hot standby have different takeover behavior, and that edge storage can cover the small interval between primary failure and failover recording. Axis also documents prerequisites and limitations for camera-side failover recording, including time synchronization and bandwidth for later transfer.

5. Test six controlled states

Normal operation

Confirm recording, live view, playback, bookmarks, alarms, camera time and health status. Export the active recorder mapping and record baseline CPU, memory, disk latency, write throughput and network ingress.

Recorder service failure

Stop only the recording service using the vendor-supported method. Measure detection, camera reassignment, live-view recovery, first failover recording and alarm delivery. Verify that the management platform identifies the correct failed component.

Hard server or network loss

After service-failure recovery is complete, perform the approved hard-failure test, such as disconnecting the recorder's test network link or removing its protected power feed. Do not pull production disks or corrupt a live database. Compare results with the service-only failure.

Standby saturation check

While failed over, inspect every transferred camera and the high-priority subset. Check dropped frames, recording errors, disk queue, database errors, network utilization and client responsiveness. Confirm that configured priorities are honored if the standby cannot accept all cameras.

Primary return and failback

Restore the primary. Verify which server currently owns each camera, whether failback is automatic, and whether recent failover recordings are temporarily unavailable during merging. Confirm that both servers do not continue recording the same assigned role unexpectedly.

Edge retrieval and timeline reconciliation

If cameras have edge storage, verify retrieval job state, rate and completion. Milestone notes that retrieval can take time because historical video competes with current live and recording traffic. Confirm that the retrieved interval is playable and correctly ordered across the transition.

6. Use a recovery bandwidth worksheet

Edge retrieval can overload a constrained uplink or delay other cameras. Estimate the minimum retrieval time before the test, then compare it with the measured result.

If a 20 Mbps retrieval flow is permitted on a 100 Mbps shared path, the utilization factor is 0.20. Preserve bandwidth for current recordings, control traffic and live users. Record whether the VMS automatically limits retrieval or requires an engineering schedule.

7. Acceptance checklist

1. Protected failure scenarios and excluded failure domains are documented.
2. Primary, secondary and optional tertiary roles are mapped.
3. Camera licenses and standby device capacity are confirmed.
4. Peak camera ingest and current standby workload are calculated.
5. Temporary failover storage is sufficient for the agreed duration.
6. Databases, logical disks and archive paths have clear ownership.
7. Server, switch, storage and power dependencies are mapped.
8. Continuous, event-based and priority camera samples are selected.
9. Normal-operation health and performance baselines are saved.
10. Service-failure takeover time and alarms are measured.
11. Approved hard-failure takeover time is measured.
12. All protected cameras are checked while the standby is loaded.
13. Live view, recording, playback and export are tested in failover.
14. Pre-failure archive availability is checked during takeover.
15. Primary return and camera ownership are verified.
16. Failover recordings merge or remain accessible as designed.
17. Edge retrieval completes without harming current recording.
18. Unrecovered gaps, duplicate intervals and deviations are signed off.

8. Handover record

The final pack should include the architecture diagram, failure-domain statement, server and storage inventory, licenses, camera-to-recorder matrix, capacity worksheet, configuration export, test timestamps, VMS and operating-system logs, screenshots of alarms, performance counters, sampled timelines, edge-retrieval status, deviations and retest evidence. Record the vendor software versions because failover behavior and licensing can change between releases.

Primary technical references

• Milestone XProtect: Failover recording server
• Milestone XProtect: Edge storage used as failover recording
• Genetec Security Center: Archiver failover
• Hanwha Vision WAVE: Configuring failover
• Axis Camera Station 5: Failover recording and troubleshooting