By Jemadhar · republished on June 12, 2026 (original version: April 3, 2026)
“No plan survives contact with the enemy.” — Helmuth von Moltke, 1871
Why this article is being republished
The article you’re reading already existed. I had published it in April, proud of my two PowerShell scripts that switched my personal infrastructure over to the DRP server and brought it back into production, all with service continuity that I believed was rock-solid.
Then came June.
During a return to production after maintenance, I realized that changes made while the infrastructure was running on the DRP had disappeared after the return to production. Recent files were missing; a day’s work had vanished. On my personal infrastructure, the damagewas minor—just a few blog posts to redo. In a corporate setting, it could have been a major incident: data silently lost, with not a single error in the logs to warn us.
So I started from scratch, with Veeam documentation to guide me. And the verdict is harsh: three of the “solutions ” I presented in the April version were actually time bombs. Not just rough approximations—commands that actively destroy data.
This article follows the same structure as the original, but each section now includes a record of what was corrected, why, and what unexpected effects it caused in production. The scripts published here cancel and replace those from previous versions.
I believe this is a better article than before. You forget about a script that works. A script that nearly cost you your data teaches you Veeam for good.
Architecture
HYPERV1 (Hyper-V prod)
├── 2 AD domains with trust
│ ├── Domain1: SRV-PDC1 (PDC + AD-integrated DNS)
│ │ SRV-DC1 (Secondary DC + AD-integrated DNS)
│ └── Domain2: SRV-PDC2 (PDC + AD-integrated DNS + DHCP)
│ SRV-DC2 (Secondary DC + AD-integrated DNS + DHCP)
├── 2 Linux BIND9 DNS forwarders + NTP
│ ├── SRV-DNS1 (Primary DNS forwarder + NTP)
│ └── SRV-DNS2 (Secondary DNS forwarder + NTP)
├── RADIUS, Proxy, SMTP, SIEM, Monitoring, dev, internal DNS...
└── virtual workstations
HYPERV2 (Hyper-V DRP)
└── Veeam B&R; 12 + replicas of all VMs
The infrastructure has grown since April: we’ve gone from 19 to 22 VMs (addition of a dev server that replicates the blog’s production environment, and an internal Pi-hole/Unbound DNS server). Replication runs 4 times a week with a retention of 2 restore points — a maximum RPO of 24 hours.
The two scenarios
Two radically different use cases, and this distinction hasn’t changed one bit:
FULL DRP — Actual crash. Production is down. We switch everything over immediately. The PDCs start up first, the rest follows. No continuity to manage since everything is already down.
MCO DRP — Planned maintenance. Production is still running. We must switch over without any service interruption. The order becomes critical: there must always be at least one active DC per domain and one active DNS forwarder somewhere on the network.
What has changed, however, is how we trigger the switchover. The April release relied on two Veeam Failover Plans (“FULL-DRP” and “MCO-DRP”). The new version replaces them with individual VM-by-VM failover via PowerShell. I’ll come back to this later—it’s this change that unlocks the real performance gain.
MCO Continuity: The Order That Changes Everything
The continuity principle still relies still on two groups that switch over alternately, ensuring that at any given moment Domain1, Domain2, and the DNS always have at least one active service somewhere.
Group 1 — Domain1 secondary DC + Domain2 secondary DC + primary DNS forwarder: SRV-DC1 · SRV-DC2 · SRV-DNS1
Group 2 — Domain1 PDC + Domain2 PDC + Secondary DNS forwarder: SRV-PDC1 · SRV-PDC2 · SRV-DNS2
┌────────────────────────────────────────────────────────────────────────┐
│ HYPERV1 (prod) HYPERV2 (DRP) │
│ Domain1 Domain2 DNS Domain1 Domain2 DNS │
├────────────────────────────────────────────────────────────────────────┤
│ Start PDC1 ✅ PDC2 ✅ DNS1 ✅ — — — │
│ DC1 ✅ DC2 ✅ DNS2 ✅ │
├────────────────────────────────────────────────────────────────────────┤
│ Group 1 PDC1 ✅ PDC2 ✅ DNS1 ⬇️ DC1 🔄 DC2 🔄 DNS1 🔄 │
│ failover DC1 ⬇️ DC2 ⬇️ DNS2 ✅ │
│ ↑PDC1+PDC2+DNS2 still UP on HYPERV1 → continuity ✅ │
├────────────────────────────────────────────────────────────────────────┤
│ Group 1 PDC1 ✅ PDC2 ✅ DNS2 ✅ DC1 ✅ DC2 ✅ DNS1 ✅ │
│ confirmed ↑Group 1 Running confirmed on HYPERV2 BEFORE continuing │
├────────────────────────────────────────────────────────────────────────┤
│ Group 2 PDC1 ⬇️ PDC2 ⬇️ DNS2 ⬇️ DC1 ✅ DC2 ✅ DNS1 ✅ │
│ failover — — — PDC1 🔄 PDC2 🔄 DNS2 🔄 │
│ ↑DC1+DC2+DNS1 up on HYPERV2 → continuity ✅ │
├────────────────────────────────────────────────────────────────────────┤
│ Final — — — PDC1 ✅ PDC2 ✅ DNS1 ✅ │
│ DC1 ✅ DC2 ✅ DNS2 ✅ │
└────────────────────────────────────────────────────────────────────────┘
At each stage, Domain1 always has an active DC, as does Domain2, and the DNS forwarder always responds. No AD, DNS, or DHCP outages during the entire failover.
It’sI didn’t have in April.** In the initial version, the script logged a warning if Group 1 failed to start, but continued anyway to shut down Group 2. Potential consequence: if Group 1 failed to start on HYPERV2, we would end up with Group 1 DCs down everywhere and Group 2 DCs shutting down → no domain controllers anywhere. The new version stops immediately if Group 1is confirmed as Running. Worst-case scenario: a clean shutdown in degraded state (PDCs remain up on production), never a total outage.
The architecture change: from Failover Plan to individual failover
This is the core of the failover-side redesign. The April versiontriggered Start-VBRFailoverPlan—a monolithic Veeam object that starts the entire group at once with its own internal timers. Problem: it was impossible to control it VM by VM, so the script had to shut down all non-critical VMs on production before launching the plan. Meanwhile, services remained down.
The new version controls each failover individually with Start-VBRHvReplicaFailover -RunAsync. This changes everything for non-critical VMs: we can pipeline.
For each non-critical VM:
1. Shutdown on HYPERV1 (blocking, we wait for the Off)
2. Start-VBRHvReplicaFailover -RunAsync (we do NOT wait for the boot)
3. Next VM immediately
The -RunAsync option is key: VM N boots on HYPERV2 while VM N+1 is shutting down on HYPERV1. Downtime for each service is reduced to its own stop/boot cycle, rather than “all shutdowns + position in the plan ".
The DCs/DNS servers, however, are not pipelined—this is intentional. They retain group-based processing with confirmed wait and the safeguard described above. Speed for workstations, caution for controllers.
Side benefit: the script no longer depends on Failover Plans. We can keep them in Veeam as a manual fallback solution, but they are no longer a synchronization point to maintain.
Return to production: where things went wrong
If you only read one section, make it this one. The three bugs were hidden in the failback, and it was the failback that cost me data in June.
Veeam 12 pitfalls — corrected version
Pitfall 1 — Get-VBRSession requires a mandatory parameter
Contrary to what the documentation suggests, Get-VBRSession without arguments opens an interactive prompt (Job:) instead of returning null — even with -ErrorAction SilentlyContinue, which has no effect on a required parameter. In an automated script, this freezes everything. The fix: use Get-VBRBackupSession, filtered by job name.
$lastSession = Get-VBRBackupSession -ErrorAction SilentlyContinue |
Where-Object { $_.JobName -eq $VeeamReplicaJobName } |
Sort-Object CreationTime -Descending |
Select-Object -First 1
Pitfall 2 — The “commit” that was actually an undo
This is the most serious bug, and I had published it as a solution. To finalize a failback, my code from April did the following:
# WHAT I WAS DOING — THIS IS WRONG
Stop-VBRHvReplicaFailback -RestorePoint $rpCommit
Stop-VBRHvReplicaFailback is not a commit. The Veeam documentation is unambiguous: “ Undoes Hyper-V replica failback.” It is the cancellation of the failback. Instead of validating the resync that had just brought the data back to production, this command reverses that resync. The actual PowerShell commit has no dedicated cmdlet—it’s the same Start-VBRHvReplicaFailback with the -Complete switch, applied to the restore point prior to the failback (index 1):
# THE REAL COMMIT
$rpCommit = Get-VBRRestorePoint |
Where-Object { $_.IsReplica() -and $_.VmName -eq $vmName } |
Sort-Object CreationTime -Descending |
Select-Object -Skip 1 -First 1 # index 1 = pre-failback RP
Start-VBRHvReplicaFailback -RestorePoint $rpCommit -Complete
The index detail remains valid: after a failback, Veeam creates a new restore point (index 0). Committing to index 0 locks the VM as LockedItem. You need index 1. I had that part right—it was the cmdlet that was wrong.
Pitfall 3 — Stop-VBRReplicaFailover before failback: an undo failover
Second destructive command, presented as a fix for the “Failover Plan that restarts thereplicas." My code from April did the following for each VM, before failback:
# WHAT I WAS DOING — THIS IS WRONG TOO
Stop-VBRReplicaFailover -RestorePoint $rpFailover
Here again, the documentation is crystal clear: “All changes that were made to the replicas during failover are discarded.” Stop-VBRReplicaFailover is an undo failover: it discards all changes made to the replicas during the DRP period, which is exactly the work we want to preserve. I was destroying the data even before running the resync that was supposed to restore it.
The fix: nothing. We simply remove this step. Start-VBRHvReplicaFailback handles the clean shutdown of the replica itself. The sequence is simplified as well as being fixed.
Pitfall 4 — -QuickRollback, the optimization that has no place here
My April failback passed -QuickRollback to Start-VBRHvReplicaFailback to speed up the resync. Quick Rollback relies on the CBT (Changed Block Tracking) of the source VM to transfer only a delta. The Veeam documentation is clear: use it only for an issue that occurred at the guest OS level (application error, deleted file). Never after a hardware incident, a restore from backup, or a VM recreation.
Yet a DRP is triggered by definition for these very cases. After my platform change on production (motherboard + CPU replacement) and a RAID restore, the source’s CBT was invalid. Quick Rollback considered blocks that were not actually unchanged to be “unchanged,” blocks that were not, and never rewrote them. Silent corruption, no errors.
The fix: complete removal. No opt-in setting, no “are you sure.” Full failback (calculating digests on entire disks) is slower, but reliable regardless of the source’s state. It is the only acceptable mode for a DRP failback.
Pitfall 5 — The VHDX is still locked
This one was correct as of April; I’m keeping it. After the commit, Hyper-V did not immediately release the VHDX; an immediate Start-VM fails. Solution: Wait 15 seconds between the commit and startup.
The corrected final sequence
For each VM (in pairs, DC/DNS continuity):
1. Start-VBRHvReplicaFailback (blocking, WITHOUT QuickRollback)
→ FULL resync via digest calculation to HYPERV1
→ Veeam shuts down the replica and creates a new RP
2. Verify thata new RP has been created
→ safeguard: no blind commit if failback failed
3. Start-VBRHvReplicaFailback -RestorePoint <index 1=""> -Complete
→ COMMIT (the real one)
4. Start-Sleep 15
→ VHDX release
5. Start-VM on HYPERV1
→ in case of failback failure: VM NOT started, added to the
failure list, final summary (better to have a service visibly down
than a service up with stale data)
Three fewer commands (undo failover and Quick Rollback removed, fake commit replaced), two additional safeguards. The sequence is shorter and safer.
The recovery order for AD/DNS continuity
SRV-PDC1 (PDC Domain1) 120s ┐ while PDC1 comes back online,
SRV-DC1 (DC2 Domain1) 60s ┘ DC1 still covers on HYPERV2
SRV-PDC2 (PDC Domain2) 90s ┐ same for Domain2
SRV-DC2 (DC2 Domain2) 60s ┘
SRV-DNS1 (forwarder) 60s ┐ same for DNS
SRV-DNS2 (forwarder) 30s ┘
... application services ...
... workstations ...
During failback, continuity is not ensured by groups but by the interleaving of pairs: when a PDC comes back online, its secondary DC is still in failover on HYPERV2, so the domain remains covered. At no point are both DCs of the same domain down at the same time.
Why I Didn’t Notice Anything
A legitimate question: how could three destructive commands have made it into production and remained there for two months?
Because none of them produce an error. The undo failover runs without a hitch. The fake commit returns a success. The Quick Rollback finishes with “OK.” The scripts were logging green [OK]s everywhere. The VMs were rebooting, the services were responding. Everything seemed perfect.
The only symptom was missing data—and even then, only the data modified during the DRP window, so a subset you’d only notice if you went looking for it specifically.
And that’s where my testing protocol was flawed. Every time Ivalidated my scripts, I’d do the round trip within the same hour: failover to the DRP, verify that everything was up, then failback right away. Between the moment a VM went to HYPERV2 and the moment it came back to HYPERV1, nothing had changed internally. But the three bugs only destroy changes made during the DRP window. No changes, no symptoms. My tests passed becausethey never tested the one thing that was broken.
In June, it was different. The infrastructure had run three days on the DRP during maintenance—three days of real changes: articles written on the blog server, logs, live databases. This time, there was something to lose. And that’s exactly what happened.
The lesson: to validate a failback, it’s not enough for the VMs to reboot, nor is it enough to do a dry run. You have to create test data while theinfrastructure is running on the DRP, wait a while, then verify that it survived the switchover. This is the only test that truly exercises the resync chain—the one I wasn’t doing; the one I didn’t think to do.
Validation via a test file
This is the method I should have used from the start. Before trusting the corrected scripts again, I tested them on a single disposable VM (a workstation), through a full cycle:
- Failover the VM to the DRP via the new individual failover.
- Create a timestamped file on the VM, while it is running on the DRP:
"DRP Proof - created on HYPERV2 on $(Get-Date)" | Out-File C:\drp-proof.txt - Complete failback with the actual commit.
- Verification: Is the file present on the VM that has returned to production, with its timestamp?
The timestamp is the key: it proves that the file was created after the failover, meaning on the replica. If it survives the return, it means the resync successfully retrieved the changes from the DRP window—which was precisely what was broken.
The witness file survived. And the time taken for the full failback (3 minutes for a lightweight workstation, without Quick Rollback) confirmed, incidentally, that the resync was actually working, whereas the old fake commit was nearly instantaneous — because it didn’t do anything useful.
The fourth pitfall: a bug that didn’t come from Veeam
With the corrected scripts validated, I treated myself to a full cycle “just for fun” — failover and then back, under real-world conditions. And that’s when a fourth pitfall jumped out at me. Not a Veeam bug this time: a PowerShell / Windows bug, and the sneakiest one of all.
During the MCO switchover, the script started waiting indefinitely for a VM from Group 1 to start up on the DRP. Except that it was already running — I was even connected to it via RDP. The same command, typed manually in my console, returned Running; when run by the script, it returned NotFound. Same machine, same account, same moment.
The explanation: Get-VM on Hyper-V requires elevated administrator rights. In a non-elevated PowerShell window—even when opened with a domain admin account—Get-VM doesn’t throw an error: it returns an empty collection. The script interprets this emptiness as “VM not found ” and waits indefinitely. I had run the morning script in an elevated window; the afternoon one, I hadn’t. That was the whole difference.
The safeguard held. The script waited for the timeout on the three VMs in Group 1, then refused to switch Group 2—exactly as it was designed: never leaving both domains without a controller. The infrastructure remained in a mixed but stable state (Group 1 on the DRP, Group 2 + PDC still in production), with zero downtime. I completed the failover manually and cleanly, then added an elevation check at the beginning of both scripts: they now refuse to run in a non-elevated window, displaying a clear message.
$isAdmin = ([Security.Principal.WindowsPrincipal]`
[Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole(`
[Security.Principal.WindowsBuiltInRole]::Administrator)
if (-not $isAdmin) {
Write-Host "ERROR: Non-elevated window. Restart as an administrator." -ForegroundColor Red
exit 1
}
The lesson is exactly the same as for the three Veeam bugs, and that’s why I’m sharing it: a command that fails silently is a thousand times more dangerous than a command that crashes. A crash, we can see. Silence, on the other hand, lies — NotFound looked like a with the VM, when it was actually a permissions issue. Three times on the Veeam side, once on the Windows side: the common thread throughout this whole story is detecting silence.
Results
After the overhaul and a live MCO validation on the 22 VMs:
| Scenario | Total time | AD/DNS interruption |
|---|---|---|
| DRP MCO (switchover) | ~7 min | 0 seconds |
| MCO Failback (22 VMs) | ~2 h* | 0 seconds |
| FULL DRP (crash) | ~3 min | N/A |
*The full failback of the 22 VMs took about two hours (full disk digests, without Quick Rollback) — significantly longer than the old “fast” version. This is expected: when returning to production, data integrity takes precedence over speed. In fact, it was precisely by prioritizing speed at all costs that I broke the failback.
The failover pipeline, on the other hand, saved time and reduced downtime per service, without compromising security in any way. Key takeaway for scheduling windows: going live takes minutes; returning to production takes hours. This asymmetry is expected.
What this story taught me
A Veeam command that “works” doesn’t necessarily do what its name suggests. Stop-...Failback doesn’t cancel; it… well, actually, it does cancel. The commit is Start-...Failback -Complete. The naming is misleading, and the official documentation is incomplete regarding side effects. The Veeam R&D forums (thanks to Andreas Neufert) were more reliable than the documentation for distinguishing between commit and undo.
A failback test without sample data tests nothing. If nothing has changed between the outbound and return processes, a failed failback is indistinguishable from a successful one. The timestamped file is the only judge.
Silence is the enemy, not the crash. Thisis the common thread among the four bugs. Undo failover, the fake commit, Quick Rollback, and even the unelevated Get-VM: none of them produce anerror. All return a success or a misleading null. A script that crashes alerts you; a script that lies lets you believe everything is fine until the day you look for data that is no longer there. The whole point of this rewrite, ultimately, is to turn these silences into messages.
Safeguards are better than trust. The line-by-line review before testing, the safeguard that refuses to leave the infrastructure without a DC, the summary of failures at the end of the script: this is what turns a into a visible one. In production, that’s the difference between data loss and an alert.
Publishing a script means taking responsibility. Someone could have retrieved my April scripts and deployed them to production. That’s also why this republished version explicitly replaces previous versions, rather than quietly fixing the code. If you were using the old scripts: please download these, and test the sample file before entrusting your data to them.
Why I’m republishing instead of quietly fixing it
I could have quietly edited the April article, replaced three lines of code, and acted as if nothing had happened. No one would have noticed.
But that wouldn’t have been honest, and above all, it wouldn’t have been in the spirit of ApertureZone. The very first article on this site laid out the ground rules: here, we talk about what really works. About what really crashes. About solutions found at 3 a.m. when everything goes haywire. We’ve never claimed to be infallible, nor to sell best practices straight out of a sterile lab.
A DRP you think is perfect but that eats up your data at the worst possible moment is exactly the kind of gap between the datasheet and the real world that this site exists to document. The error isn’t the problem—hiding it would be. The key is to acknowledge it, dissect it, and come out with something more solid. This article, in its corrected version, is better than the original precisely because it bears the scar.
Welcome to the Zone.
The Scripts
Both scripts have been running on HYPERV2 (the DRP server). They include a versioned changelog, a -WhatIf mode, timestamped logs in C:\Scripts\DRP\Logs\, and safeguards at every critical step. These versions supersede all previous ones.
> VM, domain, and server names have been anonymized. The actual infrastructure differs from what is described here. The timings and order are tailored to my personal infrastructure—adjust them to your own, and test on an isolated VM before going live. (shit happens, Murphy’s Law, etc….)
Start-DRP.ps1 (switch from prod to DRP)
# Usage
.\Start-DRP.ps1 # Interactive menu
.\Start-DRP.ps1 -Mode MCO # Direct MCO
.\Start-DRP.ps1 -Mode CRASH # Direct crash
.\Start-DRP.ps1 -Mode MCO -SkipReplication # MCO without replication
Architecture v3.3: individual failover Start-VBRHvReplicaFailover -RunAsync, shutdown/boot pipeline for non-critical components, groups with wait + safeguards for DCs/DNS, tolerance for an unreachable production environment in CRASH mode, final verification of replicas, and mandatory elevation check at startup.
# =============================================================================
# Start-DRP.ps1 - v3.3
# Failover from prod (HYPERV1) to DRP (HYPERV2). To be run from HYPERV2.
#
# CRASH MODE: prod down, failover everything. Tolerates HYPERV1 being unreachable.
# MCO MODE: scheduled maintenance, service continuity guaranteed.
# Shutdown->failover pipeline for non-critical services, then
# switchover of DCs/DNS in groups with wait time + safeguards.
#
# ---------------------------------------------------------------------------
# NOTE v3.3 - WHAT HAS CHANGED COMPARED TO THE VERSIONS RELEASED IN APRIL
# ---------------------------------------------------------------------------
# * Discontinuation of Veeam Failover Plans ("FULL-DRP"/"MCO-DRP") in favor of
# individual failover Start-VBRHvReplicaFailover -RunAsync. Enables the
# shutdown/boot pipeline and eliminates a synchronization point to maintain.
# * MCO: blocking safeguard - Group 2 is NOT shut down if Group 1
# is not confirmed as Running. Previously, the script continued despite the failure
# -> risk of ending up with NO DCs anywhere.
# * CRASH: tolerates an unreachable production environment (previously: exit 1 on the WinRM check,
# which made the script unusable in a real crash).
# * Get-VBRBackupSession instead of Get-VBRSession (which opened an interactive prompt
# and froze the script despite -ErrorAction SilentlyContinue).
# * Added a mandatory elevation check at the beginning (v3.3): without
# elevated admin rights, Get-VM silently returns NotFound (incident 12/06).
# =============================================================================
param(
[ValidateSet("CRASH","MCO")]
[string]$Mode = "",
[switch]$SkipReplication
)
# --- ELEVATION CHECK (mandatory) -----------------------------------------
# Without ELEVATED admin rights, Get-VM returns an empty collection WITHOUT an error:
# VMs appear as "NotFound" silently (see incident on 12/06).
$isAdmin = ([Security.Principal.WindowsPrincipal]`
[Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole(`
[Security.Principal.WindowsBuiltInRole]::Administrator)
if (-not $isAdmin) {
Write-Host "ERROR: Non-elevated window. Run as administrator." -ForegroundColor Red
exit 1
}
# --- INTERACTIVE MENU ---------------------------------------------------------
if ($Mode -eq "") {
Write-Host ""
Write-Host "============================================================" -ForegroundColor Cyan
Write-Host " DRP PROCEDURE - Select failover mode" -ForegroundColor Cyan
Write-Host "============================================================" -ForegroundColor Cyan
Write-Host ""
Write-Host " [1] CRASH " -ForegroundColor Red -NoNewline
Write-Host "- Production down, immediate startup on DRP"
Write-Host " Individual failover, FULL DRP order (DC first)"
Write-Host ""
Write-Host " [2] MCO " -ForegroundColor Yellow -NoNewline
Write-Host "- Scheduled maintenance, service continuity guaranteed"
Write-Host " Pipeline shutdown/failover VM by VM"
Write-Host ""
Write-Host " [3] MCO + Skip " -ForegroundColor Yellow -NoNewline
Write-Host "- MCO without replication (replicas already up to date)"
Write-Host ""
Write-Host "============================================================" -ForegroundColor Cyan
Write-Host ""
$choice = Read-Host "Your choice (1/2/3)"
switch ($choice) {
"1" { $Mode = "CRASH"; Write-Host "`nCRASH mode selected." -ForegroundColor Red }
"2" { $Mode = "MCO"; Write-Host "`nMCO mode selected." -ForegroundColor Yellow }
"3" { $Mode = "MCO"; $SkipReplication = $true
Write-Host "`nMCO + SkipReplication mode selected." -ForegroundColor Yellow }
default { Write-Host "`nInvalid choice. Exit." -ForegroundColor Red; exit 1 }
}
Write-Host ""
Write-Host " Mode : $Mode" -ForegroundColor White
Write-Host " Failover : Single failover (pipeline)" -ForegroundColor White
Write-Host " Replication : $(if ($SkipReplication) { 'IGNORED' } else { 'YES' })" -ForegroundColor White
Write-Host ""
$confirm = Read-Host "Confirm launch? (Y/N)"
if ($confirm -notmatch "^[Y]$") { Write-Host "Canceling." -ForegroundColor Yellow; exit 0 }
}
# --- CONFIGURATION -----------------------------------------------------------
$ScriptVersion = "3.3"
$VeeamReplicaJobName = "ReplicaVM-HYPERV1_Daily"
$ProdHost = "HYPERV1"
$LogFile = "C:\Scripts\DRP\Logs\DRP_${Mode}_$(Get-Date -Format 'yyyyMMdd_HHmmss').log"
$ShutdownTimeout = 300
$ReplicationTimeout = 7200
$VMReadyTimeout = 600
$VeeamModule = "C:\Program Files\Veeam\Backup and Replication\Console\Veeam.Backup.PowerShell.dll"
$DRPFlagFile = "C:\Scripts\DRP\DRP_MODE.flag"
# MCO PIPELINE: Non-critical VMs (waves 5->3). For each:
# blocking shutdown on HYPERV1 followed by asynchronous failover -> the boot process overlaps
# with the shutdown of the next VM. Production DCs/DNS remain up.
$PipelineMCO = @(
"WS-03", "WS-02", "WS-01", "SRV-DNSINT", "SRV-PXE",
"SRV-PKI", "SRV-PRINT", "SRV-WSUS", "SRV-MONITORING", "SRV-SIEM",
"SRV-PASSBOLT", "SRV-SMTP", "SRV-DEV", "SRV-PROXY", "SRV-RADIUS"
)
# Secondary DCs + Primary DNS (peers remain up on HYPERV1)
$MCOGroup1 = @("SRV-DC1", "SRV-DC2", "SRV-DNS1")
# PDC + DNS failover
$MCOGroup2 = @("SRV-PDC1", "SRV-PDC2", "SRV-DNS2")
# FULL DRP (CRASH) order: DC/DNS first, delay between asynchronous launches
$FullDRPOrder = @(
@{ Name = "SRV-PDC1"; Delay = 120 }, @{ Name = "SRV-PDC2"; Delay = 90 }, @{ Name = "SRV-DNS1"; Delay = 60 },
@{ Name = "SRV-DC1"; Delay = 60 }, @{ Name = "SRV-DC2"; Delay = 60 }, @{ Name = "SRV-DNS2"; Delay = 30 },
@{ Name = "SRV-RADIUS"; Delay = 45 }, @{ Name = "SRV-PROXY"; Delay = 30 }, @{ Name = "SRV-DEV"; Delay = 20 },
@{ Name = "SRV-DNSINT"; Delay = 20 }, @{ Name = "SRV-SMTP"; Delay = 30 }, @{ Name = "SRV-PASSBOLT"; Delay = 30 },
@{ Name = "SRV-SIEM"; Delay = 45 }, @{ Name = "SRV-MONITORING"; Delay = 45 }, @{ Name = "SRV-WSUS"; Delay = 30 },
@{ Name = "SRV-PRINT"; Delay = 30 }, @{ Name = "SRV-PKI"; Delay = 20 },
@{ Name = "SRV-PXE"; Delay = 20 }, @{ Name = "WS-01"; Delay = 20 }, @{ Name = "WS-02"; Delay = 20 },
@{ Name = "WS-03"; Delay = 20 }
)
$FailedFailovers = @()
# --- FUNCTIONS ---------------------------------------------------------------
function Write-Log {
param([string]$Message, [string]$Level = "INFO")
$line = "[$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')] [$Level] $Message"
Write-Host $line -ForegroundColor $(switch ($Level) {
"OK" {"Green"} "WARN" {"Yellow"} "ERROR" {"Red"} default {"Cyan"} })
Add-Content -Path $LogFile -Value $line
}
function Wait-VMOff {
param([string]$VMName, [int]$TimeoutSec = $ShutdownTimeout)
$elapsed = 0
while ($elapsed -lt $TimeoutSec) {
$state = Invoke-Command -ComputerName $ProdHost -ScriptBlock {
param($n) (Get-VM -Name $n -ErrorAction SilentlyContinue).State.ToString()
} -ArgumentList $VMName
if ($state -eq "Off") { return $true }
Start-Sleep -Seconds 5; $elapsed += 5
}
return $false
}
function Stop-VMProprement {
param([string]$VMName)
if (-not $script:ProdReachable) {
Write-Log "VM '$VMName': $ProdHost unreachable, shutdown ignored" "WARN"; return
}
$vmState = Invoke-Command -ComputerName $ProdHost -ScriptBlock {
param($n) $v = Get-VM -Name $n -ErrorAction SilentlyContinue
if ($v) { $v.State.ToString() } else { "NotFound" }
} -ArgumentList $VMName
if ($vmState -eq "NotFound") { Write-Log "VM '$VMName' not found, skipped" "WARN"; return }
if ($vmState -eq "Off") { Write-Log "VM '$VMName' already powered off, skipped" "OK"; return }
Write-Log "Shutting down '$VMName' (state: $vmState)..."
Invoke-Command -ComputerName $ProdHost -ScriptBlock {
param($n) Stop-VM -Name $n -Force -ErrorAction SilentlyContinue
} -ArgumentList $VMName
if (Wait-VMOff -VMName $VMName) {
Write-Log "VM '$VMName' shut down properly" "OK"
} else {
Write-Log "VM '$VMName' not responding, forcing power off..." "WARN"
Invoke-Command -ComputerName $ProdHost -ScriptBlock {
param($n) Stop-VM -Name $n -TurnOff -Force -ErrorAction SilentlyContinue
} -ArgumentList $VMName
Start-Sleep -Seconds 10
}
}
function Start-FailoverAsync {
# Individual failover via the most recent RP. -RunAsync = the script
# does NOT wait for boot -> pipeline. The source VM MUST be Off beforehand
# (otherwise name/IP conflict) - guaranteed by Stop-VMProperly upstream.
param([string]$VMName)
$rp = Get-VBRRestorePoint |
Where-Object { $_.IsReplica() -and $_.VmName -eq $VMName -and $_.State.ToString() -ne "Failover" } |
Sort-Object CreationTime -Descending | Select-Object -First 1
if (-not $rp) {
Write-Log "[$VMName] No restore points available!" "ERROR"
$script:FailedFailovers += $VMName; return $false
}
try {
Start-VBRHvReplicaFailover -RestorePoint $rp -RunAsync -ErrorAction Stop | Out-Null
Write-Log "[$VMName] Failover starting (async, RP from $($rp.CreationTime))" "OK"; return $true
} catch {
Write-Log "[$VMName] Failover error: $_" "ERROR"
$script:FailedFailovers += $VMName; return $false
}
}
function Wait-VMRunningLocal {
# Verification via local Hyper-V on the DRP (name _VeeamReplica), not via
# the network -> avoids false positives if the same IP responds from HYPERV1.
param([string]$VMName, [int]$TimeoutSec = $VMReadyTimeout)
$replicaName = "${VMName}_VeeamReplica"; $elapsed = 0
Write-Log "Waiting for '$replicaName' to be running on the DRP (local Hyper-V)..." "WARN"
while ($elapsed -lt $TimeoutSec) {
$vm = Get-VM -ComputerName localhost -Name $replicaName -ErrorAction SilentlyContinue
$state = if ($vm) { $vm.State.ToString() } else { "NotFound" }
if ($state -eq "Running") { Write-Log "'$VMName' confirmed as Running on the DRP" "OK"; return $true }
Write-Log "'$VMName' not yet Running (state: $state)" "WARN"
Start-Sleep -Seconds 15; $elapsed += 15
}
Write-Log "'$VMName' not Running after $($TimeoutSec/60) min" "ERROR"; return $false
}
function Wait-GroupRunning {
param([string[]]$VMNames)
$allReady = $true
foreach ($vmName in $VMNames) { if (-not (Wait-VMRunningLocal -VMName $vmName)) { $allReady = $false } }
return $allReady
}
# --- INITIALIZATION ----------------------------------------------------------
$logDir = Split-Path $LogFile
if (-not (Test-Path $logDir)) { New-Item -ItemType Directory -Path $logDir -Force | Out-Null }
Write-Log "============================================================"
Write-Log "STARTING DRP PROCEDURE - Version $ScriptVersion - Mode $Mode"
Write-Log "============================================================"
try { Import-Module Hyper-V -ErrorAction Stop -WarningAction SilentlyContinue; Write-Log "Hyper-V module loading" "OK" }
catch { Write-Log "Hyper-V module failed: $_" "ERROR"; exit 1 }
try { Import-Module $VeeamModule -ErrorAction Stop -WarningAction SilentlyContinue; Write-Log "Veeam module loaded" "OK" }
catch { Write-Log "Veeam module failed: $_" "ERROR"; exit 1 }
# Check WinRM - TOLERANT in CRASH mode (previously: systematic exit 1)
$ProdReachable = $false
try {
Invoke-Command -ComputerName $ProdHost -ScriptBlock { $env:COMPUTERNAME } -ErrorAction Stop | Out-Null
$ProdReachable = $true; Write-Log "WinRM connection to $ProdHost OK" "OK"
} catch {
if ($Mode -eq "CRASH") {
Write-Log "$ProdHost UNREACHABLE - consistent with a crash" "WARN"
Write-Log "WARNING: Physically verify that $ProdHost is indeed down" "WARN"
$c = Read-Host "Is $ProdHost really down? (Y/N)"
if ($c -notmatch "^[Y/N]$") { Write-Log "Abort - check the status of $ProdHost" "ERROR"; exit 1 }
} else {
Write-Log "$ProdHost unreachable - MCO mode requires a reachable production server" "ERROR"; exit 1
}
}
# --- STEP 1: REPLICATION ---------------------------------------------------
if ($SkipReplication -or (-not $ProdReachable)) {
Write-Log "STEP 1: Replication skipped" "WARN"
} else {
Write-Log "STEP 1: Replication '$VeeamReplicaJobName'"
Invoke-Command -ComputerName $ProdHost -ScriptBlock {
param($f) New-Item -Path $f -ItemType File -Force | Out-Null
} -ArgumentList $DRPFlagFile
$job = Get-VBRJob -Name $VeeamReplicaJobName -ErrorAction Stop
if (-not $job.IsRunning) { Start-VBRJob -Job $job | Out-Null; Write-Log "Job starting" "OK" }
Start-Sleep -Seconds 20
$elapsed = 0; $success = $false
while ($elapsed -lt $ReplicationTimeout) {
$job = Get-VBRJob -Name $VeeamReplicaJobName
if (-not $job.IsRunning) {
# FIX: Get-VBRBackupSession, NOT Get-VBRSession (interactive prompt)
$s = Get-VBRBackupSession -ErrorAction SilentlyContinue |
Where-Object { $_.JobName -eq $VeeamReplicaJobName } |
Sort-Object CreationTime -Descending | Select-Object -First 1
if ($s -and ($s.Result -eq "Success" -or $s.Result -eq "Warning")) {
Write-Log "Replication OK (Result: $($s.Result))" "OK"; $success = $true; break
} elseif ($s -and $s.Result -ne "" -and $s.Result -ne "None") {
Write-Log "Replication ERROR (Result: $($s.Result))" "ERROR"; break
}
}
Start-Sleep -Seconds 30; $elapsed += 30
Write-Log "Replication in progress... ($([math]::Round($elapsed/60,1)) min)"
}
if (-not $success) { Write-Log "Replication failed/timeout. Shutting down." "ERROR"; exit 1 }
}
# --- FAILOVER -----------------------------------------------------------------
if ($Mode -eq "CRASH") {
if ($ProdReachable) {
Write-Log "STEP 2: Shutdown all VMs on $ProdHost"
foreach ($vmName in $PipelineMCO) { Stop-VMProprement -VMName $vmName }
foreach ($vmName in ($MCOGroup2 + $MCOGroup1)) { Stop-VMProprement -VMName $vmName }
} else {
Write-Log "STEP 2: Shutdowns ignored ($ProdHost unreachable)" "WARN"
}
Write-Log "STEP 3: Individual failover - FULL DRP order (DC/DNS first)"
foreach ($vm in $FullDRPOrder) {
Start-FailoverAsync -VMName $vm.Name | Out-Null
Start-Sleep -Seconds $vm.Delay
}
} else {
Write-Log "STEP 2: PIPELINE waves 5/4/3 - shutdown -> async failover"
Write-Log "Prod DCs/DNS remain up: AD/DNS continuity guaranteed" "OK"
foreach ($vmName in $PipelineMCO) {
Write-Log "============ $vmName ============"
Stop-VMProprement -VMName $vmName
Start-FailoverAsync -VMName $vmName | Out-Null
# No wait: the async boot overlaps the shutdown of the next VM
}
Write-Log "STEP 3: Group 1 - $($MCOGroupe1 -join ' + ')"
foreach ($vmName in $MCOGroup1) {
Stop-VMProprement -VMName $vmName
Start-FailoverAsync -VMName $vmName | Out-Null
}
Write-Log "STEP 4: Waiting for Group 1 to run on the DRP"
$group1Ready = Wait-GroupRunning -VMNames $MCOGroup1
# CRITICAL SAFEGUARD (new in v3.2): if Group 1 is not
# fully Running, shutting down Group 2 would leave both
# domains WITHOUT ANY DC. We stop here in a degraded but stable state.
if (-not $group1Ready) {
Write-Log "SAFETY STOP: Group 1 not fully Running." "ERROR"
Write-Log "Group 2 WILL NOT be shut down. PDC still up in production." "ERROR"
Write-Log "Diagnose Group 1 then restart or manually failover." "ERROR"
exit 1
}
Write-Log "STEP 5: Group 2 - $($MCOGroup2 -join ' + ')"
Write-Log "Group 1 confirmed up on the DRP -> AD/guaranteed" "OK"
foreach ($vmName in $MCOGroup2) {
Stop-VMProperly -VMName $vmName
Start-FailoverAsync -VMName $vmName | Out-Null
}
Write-Log "STEP 6: Waiting for Group 2 to be Running on the DRP"
Wait-GroupeRunning -VMNames $MCOGroupe2 | Out-Null
}
# --- FINAL VERIFICATION -----------------------------------------------------
Write-Log "VERIFICATION: All replicas running on the DRP"
$AllVMNames = if ($Mode -eq "CRASH") { $FullDRPOrder | ForEach-Object { $_.Name } }
else { $PipelineMCO + $MCOGroup1 + $MCOGroup2 }
$notRunning = @()
foreach ($vmName in $AllVMNames) {
$vm = Get-VM -ComputerName localhost -Name "${vmName}_VeeamReplica" -ErrorAction SilentlyContinue
$state = if ($vm) { $vm.State.ToString() } else { "NotFound" }
if ($state -eq "Running") { Write-Log "Replica '$vmName': Running" "OK" }
else { Write-Log "Replica '$vmName': $state" "ERROR"; $notRunning += $vmName }
}
# Second pass after 60s for async failovers still booting
if ($notRunning.Count -gt 0) {
Write-Log "$($notRunning.Count) replica(s) not Running - second pass in 60s..." "WARN"
Start-Sleep -Seconds 60
$still = @()
foreach ($vmName in $notRunning) {
$vm = Get-VM -ComputerName localhost -Name "${vmName}_VeeamReplica" -ErrorAction SilentlyContinue
if ($vm -and $vm.State.ToString() -eq "Running") { Write-Log "Replica '$vmName': Running (2nd pass)" "OK" }
else { Write-Log "Replica '$vmName': still down" "ERROR"; $still += $vmName }
}
$notRunning = $still
}
$totalIssues = ($FailedFailovers + $notRunning) | Select-Object -Unique
Write-Log "============================================================"
if ($totalIssues.Count -eq 0) {
Write-Log "DRP PROCEDURE $Mode COMPLETED - FAILOVER TO DRP COMPLETE" "OK"
} else {
Write-Log "DRP $Mode COMPLETED WITH ERRORS: $($totalIssues -join ', ')" "ERROR"
}
Write-Log "Full log: $LogFile"
if ($totalIssues.Count -gt 0) { exit 1 }
Start-FailbackToProd.ps1 (failback from DRP to prod)
# Usage
.\Start-FailbackToProd.ps1 # Interactive menu
.\Start-FailbackToProd.ps1 -WhatIf # Dry run
Architecture v2.4: full failback without Quick Rollback, without prior undo, true commit via -Complete, verification of the new RP before commit, VM not started if failback fails, summary of failures at the end of the procedure, and mandatory elevation check at startup.
# =============================================================================
# Start-FailbackToProd.ps1 - v2.4
# DRP failback (HYPERV2) -> prod (HYPERV1). To be run from HYPERV2.
#
# VMs are processed one by one, in PAIRS to ensure
# that one DC per domain and one DNS are always up (when a PDC comes back online,
# its secondary DC is still in failover on the DRP, and vice versa).
#
# =============================================================================
# WARNING THREE COMMANDS NEVER TO BE RUN AGAIN
# =============================================================================
# These three "solutions" were included in the versions released in April.
# Each one destroys data WITHOUT generating an error. Cause of the
# data loss in June 2026. Never re-enter them.
#
# 1. NEVER use Stop-VBRReplicaFailover before failback.
# = UNDO FAILOVER. Veeam Doc: "All changes that were made to the
# replicas during failover are discarded." Discards all work
# done during the DRP window, even BEFORE resync. Also useless:
# Start-VBRHvReplicaFailback shuts down the replica itself.
#
# 2. NEVER use -QuickRollback on Start-VBRHvReplicaFailback.
# Relies on the source’s CBT to transfer only a delta.
# After a hardware incident / restore / VM recreation (= the cases
# that trigger a DRP), the CBT is invalid -> blocks are silently
# skipped -> corruption. Full failback (digests) is
# slower but reliable. Only allowed mode.
#
# 3. NEVER use Stop-VBRHvReplicaFailback to "commit".
# = UNDO FAILBACK. Veeam Doc: "Undoes Hyper-V replica failback."
# Cancels the resync we just performed. The actual commit is:
# Start-VBRHvReplicaFailback -RestorePoint <rp index="" 1=""> -Complete
# Always index 1 (pre-failback RP), never index 0 (-> LockedItem).
# =============================================================================
param(
[ValidateSet("MCO","CRASH")]
[string]$Mode = "",
[switch]$WhatIf
)
# --- ELEVATION CHECK (required) -----------------------------------------
# Without ELEVATED admin rights, Get-VM returns an empty collection WITHOUT an error:
# VMs appear as "NotFound" silently (see incident on 06/12).
$isAdmin = ([Security.Principal.WindowsPrincipal]`
[Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole(`
[Security.Principal.WindowsBuiltInRole]::Administrator)
if (-not $isAdmin) {
Write-Host "ERROR: Window not elevated. Restart as administrator." -ForegroundColor Red
exit 1
}
# --- INTERACTIVE MENU ---------------------------------------------------------
if ($Mode -eq "") {
Write-Host ""
Write-Host "============================================================" -ForegroundColor Cyan
Write-Host " FAILBACK TO PRODUCTION PROCEDURE" -ForegroundColor Cyan
Write-Host "============================================================" -ForegroundColor Cyan
Write--Host " [1] MCO FAILBACK - Return after scheduled maintenance"
Write-Host " [2] CRASH FAILBACK - Return after disaster"
Write-Host "============================================================" -ForegroundColor Cyan
$choice = Read-Host "Your choice (1/2)"
switch ($choice) {
"1" { $Mode = "MCO"; Write-Host "`nMode FAILBACK MCO." -ForegroundColor Yellow }
"2" { $Mode = "CRASH"; Write-Host "`nMode FAILBACK CRASH." -ForegroundColor Red }
default { Write-Host "`nInvalid choice. Stopping." -ForegroundColor Red; exit 1 }
}
Write-Host "`n Resync: COMPLETE (digests, no Quick Rollback)" -ForegroundColor White
$confirm = Read-Host "Confirm launch? (Y/N)"
if ($confirm -notmatch "^[Y]$") { Write-Host "Canceling." -ForegroundColor Yellow; exit 0 }
}
# --- CONFIGURATION -----------------------------------------------------------
$ScriptVersion = "2.4"
$ProdHost = "HYPERV1"
$LogFile = "C:\Scripts\DRP\Logs\FAILBACK_${Mode}_$(Get-Date -Format 'yyyyMMdd_HHmmss').log"
$VeeamModule = "C:\Program Files\Veeam\Backup and Replication\Console\Veeam.Backup.PowerShell.dll"
$DRPFlagFile = "C:\Scripts\DRP\DRP_MODE.flag"
# Order by pairs (DC/DNS continuity)
$VMStartOrder = @(
@{ Name = "SRV-PDC1"; Delay = 120 }, @{ Name = "SRV-DC1"; Delay = 60 },
@{ Name = "SRV-PDC2"; Delay = 90 }, @{ Name = "SRV-DC2"; Delay = 60 },
@{ Name = "SRV-DNS1"; Delay = 60 }, @{ Name = "SRV-DNS2"; Delay = 30 },
@{ Name = "SRV-RADIUS"; Delay = 45 }, @{ Name = "SRV-PROXY"; Delay = 30 },
@{ Name = "SRV-DEV"; Delay = 20 }, @{ Name = "SRV-DNSINT"; Delay = 20 },
@{ Name = "SRV-SMTP"; Delay = 30 }, @{ Name = "SRV-PASSBOLT"; Delay = 30 },
@{ Name = "SRV-SIEM"; Delay = 45 }, @{ Name = "SRV-MONITORING"; Delay = 45 },
@{ Name = "SRV-WSUS"; Delay = 30 }, @{ Name = "SRV-PRINT"; Delay = 30 },
@{ Name = "SRV-PKI"; Delay = 20 }, @{ Name = "SRV-PXE"; Delay = 20 },
@{ Name = "WS-01"; Delay = 20 }, @{ Name = "WS-02"; Delay = 20 }, @{ Name = "WS-03"; Delay = 20 }
)
$FailedVMs = @(); $SkippedVMs = @()
# --- FUNCTIONS ---------------------------------------------------------------
function Write-Log {
param([string]$Message, [string]$Level = "INFO")
$prefix = if ($WhatIf) { "[WHATIF] " } else { "" }
$line = "[$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')] [$Level] $prefix$Message"
Write-Host $line -ForegroundColor $(switch ($Level) {
"OK" {"Green"} "WARN" {"Yellow"} "ERROR" {"Red"} default {"Cyan"} })
Add-Content -Path $LogFile -Value $line
}
function Get-FailoverRestorePoint {
param([string]$VmName)
Get-VBRRestorePoint |
Where-Object { $_.IsReplica() -and $_.VmName -eq $VmName -and $_.State.ToString() -eq "Failover" } |
Sort-Object CreationTime -Descending | Select-Object -First 1
}
function Get-NewestRestorePoint {
param([string]$VmName)
Get-VBRRestorePoint |
Where-Object { $_.IsReplica() -and $_.VmName -eq $VmName } |
Sort-Object CreationTime -Descending | Select-Object -First 1
}
function Get-CommitRestorePoint {
# Index 1 = pre-failback RP. NEVER index 0 (-> LockedItem).
param([string]$VmName)
Get-VBRRestorePoint |
Where-Object { $_.IsReplica() -and $_.VmName -eq $VmName } |
Sort-Object CreationTime -Descending | Select-Object -Skip 1 -First 1
}
# --- INITIALIZATION ----------------------------------------------------------
$logDir = Split-Path $LogFile
if (-not (Test-Path $logDir)) { New-Item -ItemType Directory -Path $logDir -Force | Out-Null }
Write-Log "============================================================"
Write-Log "FAILBACK TO PRODUCTION - Version $ScriptVersion - Mode $Mode"
Write-Log "Resync COMPLETE (digests, Quick Rollback disabled)"
if ($WhatIf) { Write-Log "DRY RUN MODE - NO ACTUAL ACTION" "WARN" }
Write-Log "============================================================"
try { Import-Module $VeeamModule -ErrorAction Stop -WarningAction SilentlyContinue; Write-Log "Loading Veeam module" "OK" }
catch { Write-Log "Veeam module failed: $_" "ERROR"; exit 1 }
try {
Invoke-Command -ComputerName $ProdHost -ScriptBlock { $env:COMPUTERNAME } -ErrorAction Stop | Out-Null
Write-Log "WinRM connection to $ProdHost OK" "OK"
} catch { Write-Log "$ProdHost unreachable: $_" "ERROR"; exit 1 }
# --- PRE-VOL -----------------------------------------------------------------
Write-Log "VERIFICATION: restore points in Failover state"
$missingVMs = @()
foreach ($vm in $VMStartOrder) {
$rp = Get-FailoverRestorePoint -VmName $vm.Name
if (-not $rp) { Write-Log "No Failover RP for '$($vm.Name)'" "WARN"; $missingVMs += $vm.Name }
else { Write-Log "OK: '$($vm.Name)' -> RP from $($rp.CreationTime)" "OK" }
}
if ($missingVMs.Count -gt 0) {
Write-Log "$($missingVMs.Count) VM(s) without Failover RP: $($missingVMs -join ', ')" "WARN"
}
# --- DRY RUN -----------------------------------------------------------------
if ($WhatIf) {
foreach ($vm in $VMStartOrder) {
$rp = Get-FailoverRestorePoint -VmName $vm.Name
if ($rp) {
Write-Log " -> '$($vm.Name)' failback COMPLETE (digests + resync)" "WARN"
Write-Log " -> Check new RP, then COMMIT -Complete (index 1)" "WARN"
} else { Write-Log " -> '$($vm.Name)' not in Failover - direct startup" "WARN" }
Write-Log " -> Starting '$($vm.Name)' - delay $($vm.Delay)s" "WARN"
}
Write-Log "DRY RUN COMPLETE - no action" "WARN"; exit 0
}
# --- PROCESSING VM BY VM ----------------------------------------------------
Write-Log "STARTING VM-BY-VM FAILBACK (pair-wise order)"
foreach ($vm in $VMStartOrder) {
$vmName = $vm.Name; $delay = $vm.Delay
$failbackOK = $false; $startVM = $true
Write-Log "============ $vmName ============"
$rp = Get-FailoverRestorePoint -VmName $vmName
if ($rp) {
# STEP A - COMPLETE failback directly to the Failover RP.
# NO prior Stop-VBRReplicaFailover (= destructive undo).
# NO -QuickRollback (unreliable CBT delta after incident).
Write-Log "[$vmName] COMPLETE failback (blocking - digests + resync)..."
try {
Start-VBRHvReplicaFailback -RestorePoint $rp -PowerOn:$false -ErrorAction Stop | Out-Null
Write-Log "[$vmName] Failback complete" "OK"; $failbackOK = $true
} catch { Write-Log "[$vmName] Failback error: $_" "ERROR" }
# STEP B - Verify that a new RP has been created (prevent blind commit)
if ($failbackOK) {
$rpNew = Get-NewestRestorePoint -VmName $vmName
if ($rpNew -and $rpNew.CreationTime -gt $rp.CreationTime) {
Write-Log "[$vmName] New RP confirmed ($($rpNew.CreationTime))" "OK"
} else {
Write-Log "[$vmName] NO new RP - commit canceled for security" "ERROR"
$failbackOK = $false
}
}
# STEP C - THE ACTUAL COMMIT: Start-VBRHvReplicaFailback -Complete on index 1
if ($failbackOK) {
Write-Log "[$vmName] Commit (Start-VBRHvReplicaFailback -Complete, index 1)..."
$rpCommit = Get-CommitRestorePoint -VmName $vmName
if ($rpCommit) {
try {
Start-VBRHvReplicaFailback -RestorePoint $rpCommit -Complete -ErrorAction Stop | Out-Null
Write-Log "[$vmName] Commit OK (RP from $($rpCommit.CreationTime))" "OK"
} catch {
Write-Log "[$vmName] Commit error: $_ - manual commit required" "WARN"
}
} else { Write-Log "[$vmName] No RP index 1 - manual commit required" "WARN" }
Write-Log "[$vmName] Waiting 15s for VHDX release..."
Start-Sleep -Seconds 15
} else {
# Failback failure: we DO NOT start the VM (out-of-date state = silent loss)
Write-Log "[$vmName] FAILBACK FAILED - VM not started. Replica intact on the DRP." "ERROR"
$FailedVMs += $vmName; $startVM = $false
}
} else {
Write-Log "[$vmName] No Failover RP - VM skipped" "WARN"; $SkippedVMs += $vmName
}
# STEP E - Start on production (unless failback fails)
if ($startVM) {
$vmState = Invoke-Command -ComputerName $ProdHost -ScriptBlock {
param($n) $v = Get-VM -Name $n -ErrorAction SilentlyContinue
if ($v) { $v.State.ToString() } else { "NotFound" }
} -ArgumentList $vmName
if ($vmState -eq "NotFound") { Write-Log "[$vmName] VM not found on $ProdHost" "WARN" }
elseif ($vmState -eq "Running") { Write-Log "[$vmName] VM already running" "OK" }
else {
try {
Invoke-Command -ComputerName $ProdHost -ScriptBlock {
param($n) Start-VM -Name $n -ErrorAction Stop
} -ArgumentList $vmName
Write-Log "[$vmName] VM started on $ProdHost" "OK"
} catch { Write-Log "[$vmName] Startup error: $_" "ERROR"; $FailedVMs += $vmName }
}
Write-Log "[$vmName] Waiting $delay seconds before the next VM..."
Start-Sleep -Seconds $delay
}
}
# --- CLEANUP FLAG + RECAP --------------------------------------------------
try {
Invoke-Command -ComputerName $ProdHost -ScriptBlock {
param($f) if (Test-Path $f) { Remove-Item $f -Force }
} -ArgumentList $DRPFlagFile -ErrorAction Stop
Write-Log "DRP flag removed" "OK"
} catch { Write-Log "Unable to remove DRP flag: $_" "WARN" }
Write-Log "============================================================"
if ($FailedVMs.Count -eq 0) {
Write-Log "FAILBACK COMPLETED - PRODUCTION RESTORED ON $ProdHost" "OK"
} else {
Write-Log "FAILBACK COMPLETED WITH ERRORS - ACTION REQUIRED" "ERROR"
Write-Log "Failed VM(s): $($FailedVMs -join ', ')" "ERROR"
Write-Log "Their replicas/data remain on the DRP. Handle before performing any undo." "ERROR"
}
if ($SkippedVMs.Count -gt 0) { Write-Log "Skipped VM(s): $($SkippedVMs -join ', ')" "WARN" }
Write-Log "Re-enable autostart: Get-VM | Set-VM -AutomaticStartAction StartIfRunning"
Write-Log "Full log: $LogFile"
if ($FailedVMs.Count -gt 0) { exit 1 }
Each deleted destructive command is documented in the DANGER block at the top of the file, with a citation from the Veeam documentation, to prevent accidental reintroduction.
Tags: #veeam #hyper-v #drp #powershell #active-directory