Frequently Asked Disaster Recovery Questions (FAQs)
You must be using Veeam v10 or later to backup your VMware servers and be sending a copy of your backups to S3 compatible object storage.
Our SaaS automation platform for DR builds a dedicated on-demand bare-metal environment, complete with 25Gbps NFS storage, firewall, networking, VPN and routing. The automation then performs full restores of your Veeam backups from your object storage bucket(s) to your dedicated physical host(s) in our Bare Metal Servers
Cloud IBR provides recovery of VMware, Hyper-V and agent-based physical server backups that were backed up with Veeam and stored in an S3 object storage bucket, either backed up via Direct to Object Storage with v12, or Scale-Out-Backup-Repository (SOBR) offloads with v10 or later. For Hyper-V and agent-based backups, we perform a fully automated Instant Recovery with Veeam. These servers will require you to assign an IP address to them via their console, as Veeam Instant Recovery will not recover them with an IP address.
Most organization’s recoveries are completed in less than a few hours, but can take longer, depending on how large your backups are.
Cloud IBR is designed to use Read-Only keys to perform restores from your object storage. Read-Only keys work with object storage buckets used with “Direct to object storage backups” in Veeam v12 as well as with Scale-Out Backup Repositories (SOBR) created by Veeam v10 and later, that do not have Encryption enabled at the SOBR level. For SOBR’s that have Encryption enabled, Veeam requires Read-Write access to the bucket when adding it to the Cloud IBR Veeam server, which we chose not to support in the initial launch. You can confirm if you have an encryption key assigned to your SOBR by opening the properties of your SOBR and checking your configuration as shown here. To work around this limitation, simply make sure you have enabled encryption on each of your backup jobs by editing each job, click the Storage section, click Advanced button, select the Storage tab and check “Enable backup file encryption”, as shown here. You can then safely disable encryption for the SOBR, as having encryption enabled in both locations does not provide any additional benefit, since the data offloaded to object storage will already be encrypted once you enable encryption on each backup job.
You can begin using the service for recovery, immediately upon signing up.
Cloud IBR recovers all of your backups to VMware ESXi hosts with up to 4TB of RAM and up to 25TB of 25Gbps NFS storage. You can request to raise the storage amount if you require more. At the start of each recovery test, we automatically choose the highest performing servers available from PhoenixNAP’s inventory. To confirm Cloud IBR can handle recovering your environment, sign up for the free 30-day trial using the veeam30 promo code and run as many recovery tests as you want within the 30-day period to see if the service meets your requirements!
All customers will receive a 30-day free trial account, ensuring that Cloud IBR is the appropriate recovery platform for your business. Both Monthly and Annual Subscriptions can be cancelled at any time which will prevent future billing.
If the server have been recovered, but data is missing, this typically means your data has not been backed up properly. If any servers were not restored at all, look under the section “Skipped Servers / Encrypted Backups” in your Cloud IBR recovery, to see if any backups were found but were skipped due to being outdated or due to a missing or incorrect Veeam encryption key provided during your onboarding.
There are several methods of remotely accessing your restored environment:
- Use the OpenVPN client to connect individual users to your recovery environment.
- Connect to your Veeam recovery server via RDP
- Connect to your ESXi host(s) console via https.
- Connect to any of your servers via the port mappings you created in the Cloud IBR dashboard.
- Connect your headquarters and branch offices to your recovery environment via IPSEC tunnels with the Gold subscription and higher.
There is no time limit on how long you can keep your environment online after performing a recovery. Keep in mind that your credit card will be charged for hourly usage of the recovery infrastructure, so if you intend on keeping the environment running long-term after an actual disaster, contact us for long-term pricing options.
Your subscription to Cloud IBR covers unlimited usage of the Instant Backup Recovery platform. PhoenixNAP is an infrastructure provider that we use as the destination for restoring all your servers. When you sign up for Cloud IBR, we create an account for you at PhoenixNAP using your email address and credit card. Each time you run a recovery test, we provision servers on-demand at PhoenixNAP, on your behalf, who charges you hourly for the physical servers you utilize during the recovery test. You can see their hourly pricing on this page, which ranges from ten cents to a few dollars per hour, depending on the server type assigned during a recovery test. During your onboarding, you will be asked how much memory (RAM) your recovery environment requires, which can be used as the base metric for estimating your usage fees.
This screenshot shows the monthly usage fees for an account that ran two disaster recovery tests, one in Ashburn, VA with 128GB of RAM and one in Phoenix, AZ with 512GB of RAM, for a total of $10.58 for the two tests, which ran for two hours each.
Backblaze is a leading S3 Object Storage provider, with over two billion gigabytes of data storage under management. The company currently works with more than 500,000 customers in over 175 countries. Backblaze is a Veeam Ready Partner, certified by Veeam for their Object with Immutability integration with Veeam Backup & Replication software.
Wasabi provides simple and affordable hot cloud storage for businesses all over the world. It enables organizations to store and instantly access an unlimited amount of data with no complex tiers or egress or API fees, delivering predictable costs that save money and industry leading security and performance businesses can count on. Trusted by customers worldwide, Wasabi has been recognized as one of technology’s fastest-growing and most visionary companies. Created by Carbonite co-founders and cloud storage pioneers David Friend and Jeff Flowers, Wasabi is a privately held company based in Boston. Wasabi is a Proud Partner of the Boston Red Sox, and the Official Cloud Storage Partner of Liverpool Football Club and the Boston Bruins. Wasabi is a Veeam Ready Partner, certified by Veeam for their Object with Immutability integration with Veeam Backup & Replication software.
Please see the Wasabi Knowledge Base article, for the simple instructions on how to configure Cloud IBR to recover the Veeam backups of your physical and VMware/Hyper-V virtual servers that you store in Wasabi’s immutable S3 compatible object storage.
Absolutely. Every recovery test you run with Cloud IBR provides a full report of your disaster recovery test, which is perfect for showing proof of recovery to your auditor, insurance carrier, customer, etc. By enabling the fully automated scheduled DR test, whether it be daily, weekly, or monthly, you’ll always have an up-to-date report showing that you are recoverable, or letting you know when a change made to your production environment prevents a 100% successful recovery. You can review an example DR Recovery Report here.
Once you’ve recovered your servers in our infrastructure, you can RDP to the Veeam server that we built for you, to create and manage the backups of your newly built environment, just like you’ve always done in your production environment. For our Platinum customers, this service is automatically performed by our staff at no charge. When performing this on your own, make sure to add a new read-write key to the S3 repository in your Veeam console, so that the VBR has permissions to send new backups to your bucket.
Once you are running your production environment in our cloud infrastructure and you’ve configured backups as discussed above, you can schedule a switchback to your production environment at any time that is convenient for you. The process is simply the reverse of how we recovered you to our infrastructure. For our Platinum customers, the switchback is performed by our staff at no charge. When performing this on your own, and once you have your original production environment rebuilt, you will schedule a maintenance window to shutdown your recovery servers, perform a final backup to your Backblaze or Wasabi object storage, and then restore them to your newly built production environment using the Veeam server you’ve built in your environment. There are more costly options for replicating the data from our cloud to your hardware and performing a switchback in a shorter window, but the standard method described here is sufficient for most small businesses, as the Recovery Time Objective (RTO) will be similar to the one you experienced when you recovered using Cloud IBR. Alternatively, if you’d like to leave your production systems running in our cloud long-term, please contact us for long-term pricing options.
Our team provides support to all customers at no fee, for any issues or bugs directly related to our SaaS offering. Recovery assistance is available at a rate of $250.00USD/hour. As we grow our network of MSP’s, you will also have the option of selecting a local partner to work with.
No. This is because Synology Cloud Sync sends the backups to Object Storage in a format that is only readable by the Synology platform and not by Veeam. To recover servers with Cloud IBR, you can either create a Direct to Object Storage backup job in Veeam v12, or add the Synology to a Scale-Out-Backup-Repository (SOBR) that offloads backups to Backblaze, with Veeam v10 or later.
Both options require a Veeam Enterprise license. If you currently use the Veeam Community Edition, you can take advantage of our 30-day trial license promotion by signing up for Cloud IBR using promo code veeam30 to get a 30-day Veeam Enterprise edition trial license. We provide two pricing options for purchasing Veeam licensing:
Veeam Universal License (VUL): Click here to calculate your annual subscription fees.
Monthly Rental through Veeam Cloud & Service Provider (VCSP) program.
Please note that the VUL annual subscription allows you to open tickets directly with Veeam as per their Support Policy. For monthly rental licensing, our Platinum customers can open Veeam support tickets with us at no cost and Silver subscribers are billed $250.00USD/hour for Veeam support tickets.
By design, Cloud IBR restores every server in the object storage bucket(s) that you specify in the Cloud IBR configuration, excluding any servers that haven’t been backed up within the last X days. The default is 7 days and you can change that to anything between 1 and 30 days. If you want to permanently exclude a server from your recovery environment, you can move that server to a backup job that backs that server up to any other object storage bucket that you haven’t added to the Cloud IBR config.
The Cloud IBR automated recovery process provisions a /29 block of IP addresses. All 6 usable IP addresses are automatically assigned during the recovery to provide remote access into your infrastructure. One of the IP’s is assigned to a router, two are assigned to the management port on physical hosts and the remaining three are assigned to the firewall, which is used for your server port mappings that you specify in the Cloud IBR console during onboarding. If you require additional IP’s, you can purchase additional /28 or /29 blocks in the PhoenixNAP BMC console, after you perform your recovery.
This error is generally caused by the read-only key not having the “list all bucket names” permission. Try creating a new read-only key for your Object Storage bucket and make sure to give the key the ‘Allow listing all bucket names’ permission and then adding the new key to Cloud IBR and running another recovery.
Error Adding Object Storage Repository BucketName – Failed to load S3 configuration: Invalid credentials for AmazonS3Compatible. See log file Satellite_PowerShell on server ibr-veeam-xxxx for details.
If you ran a Manual Shutdown recovery test and did not check the “Internet Access Allowed” option, you can manually enable outbound internet by opening an ssh connection to your Cloud IBR Firewall and performing the following steps.
1- Edit /etc/ufw/before.rules and uncomment the following line in by removing only the pound sign:
#-A POSTROUTING -o ens160 -j MASQUERADE
2- Save the file and reboot the firewall: sudo reboot now
The Cloud IBR Firewall is an Ubuntu based firewall running UFW to manage inbound access rules from the Internet and IPTABLES /etc/ufw/before.rules to manage NAT mappings.
If you’d like to customize your firewall rules during a Manual Shutdown recovery test or during an actual Disaster, please refer to the following Ubuntu documentation.
There are also countless websites publicly available with examples of how to accomplish most any goal you might be looking to accomplish with UFW and IPTABLES.
In the Session Summary section and next to each server in the Recovery Overview report, you’ll see status discs of different colors and statuses. Following is a list of each status and their meaning.
Session Summary:
- Success status disc indicates that the server successfully completed all steps of the recovery and is ready for you to access.
- Skipped status disc indicates that your Veeam Repository contains backups of these servers that are older than the number of days you selected to restore from (default 7 days), or due to incompatibility with the current version of Veeam. You can open a ticket with Veeam to ask for this functionality to be added to their software and upvote the request in the Veeam forums.
- Failed status disc indicates that a portion of the recovery for these servers has failed. See troubleshooting below.
Status discs next to each server name:
For recovery of VMware backups:
- Green Stopped status disc indicates all recovery steps for this server finished successfully. This will show as a success in the session summary.
- Red Stopped status disc indicates the server at least one of the recovery steps. This will show as failed in the session summary. See troubleshooting below.
For Instant Recovery of Agent-Based, Hyper-V and Proxmox backups:
- Green Migrated status disc indicates that the Instant Recovery for this server was successful, the server booted up and the Veeam Migration of the server from your backup repository to ESXi was complete. This will show as a success in the Session Summary.
- Red Mounted status disc indicates the Instant Recovery for the server was successful and the server booted up, but it failed the Veeam Migrate step of migrating the server from your backup repository to the ESXi host. Cloud IBR automatically retries each failure one additional time. This status will show as failed in the Session Summary. See troubleshooting below.
- Gray Migrating status disc indicates the Instant Recovery for the server was successful, the server booted up and the Veeam Migrate step of migrating the server from your backup repository to the ESXi host is currently in progress.
Troubleshooting:
There are several FAQs that provide possible resolutions for individual servers failing to recover. If the FAQs do not address your issue, Cloud IBR automatically exfiltrates the Veeam logs for every test you run so that you can download the logs from the Recovery Information section of the Recovery Overview within Cloud IBR and open a ticket with Veeam to troubleshoot with their support team.
The most common reason why individual server restores fail while all others are successful is due to a corrupt backup of those servers in your Veeam Repository. One step to troubleshot this is to confirm that you have Health Checks enabled on the Veeam Backup Job. If not, enable the Health Check within advanced settings of the job and schedule when you would like it to run. Once the Health Check completes, check the logs to see if there were any issues. Corrupted backups typically show the below errors for each of the individual servers that failed to restore.
12/10/2024 12:53:09 PM Failed Verifying disks (100% done) 0:34:47
12/10/2024 1:27:56 PM Failed S3 error: We encountered an internal error. Please retry the operation again later. Code: InternalError Agent failed to process method {Signature.FullRecheckBackup}.
12/10/2024 1:27:56 PM Failed Processing finished with errors at 12/10/2024 1:27:56 PM
Running an Active Full backup or Active Full copy job is one action Veeam recommends that will usually fix the corrupted backup. If running an Active Full does not fix the issue, Cloud IBR automatically exfiltrates the Veeam logs for every test you run so that you can download the logs from the Recovery Information section of the Recovery Overview within Cloud IBR and open a ticket with Veeam to troubleshoot with their support team.
Option 1: If you’re running VMware in your production environment, you can use a Veeam Replication job and Failover Plan to easily switch back to your production environment with minimal down time.
- On your production VBR, add the Cloud IBR ESXi hosts to the Veeam Virtual Infrastructure Inventory.
- Create a Veeam Replication job to send hourly snapshots of your Cloud IBR recovered servers back to your production VMware hosts over the IPSEC VPN tunnel that is included in Gold and Platinum subscriptions.
- Continue running workloads in the BMC without interrupting your business while the Replication job syncs your servers back to your production environment.
- Create a Veeam Failover plan that will automatically power down your recovered servers in the Cloud IBR BMC, send a final snapshot and power on the servers in your production environment.
- Schedule a failback window for anytime you wish and run the Failover plan. This Veeam process typically takes less than 15 to 30 minutes depending on the size of your environment.
Option 2: If you don’t run VMware in production:
- Shut down all of your servers in the Bare Metal Cloud (BMC).
- Run a final backup to your offsite target.
- Rescan your Veeam repository from your production VBR.
- Restore your servers to your production environment and automatically power them on.
All of the above steps can be performed by your internal IT staff, your Managed Service Provider or you can upgrade to our Annual Platinum Plus service plan which provides full management and support of the above steps from the Cloud IBR Disaster Recovery experts.
The default maximum SAN storage for a recovery with Cloud IBR is 25TB.
Follow these steps if your environment requires more than 25TB of recovery storage:
- Increase your PhoenixNAP BMC quota by logging in to the BMC portal with the email address you used to register for Cloud IBR.
- Choose Forgot Password if you’ve never logged in before.
- Click on your initials at the top right and click Request Quota Increase button from the drop down menu.
- To prevent Quota email warnings from PhoenixNAP, request to increase your quota to 2x more than you require.
- Once you receive confirmation from PhoenixNAP that your BMC storage quota was increased, open a support ticket with Cloud IBR to have your Storage Requirement increased in the portal to half of what you increased your quota to.
How Does Cloud Instant Backup Recovery Work?
One-click Deploys Bare Metal Cloud Servers.
Automatically Imports Veeam Backups Stored in Object Storage or with a VCSP.
Immediate Failover Recovery to Bare Metal Cloud.
Offsite your Veeam backups to Object Storage or with a VCSP and in one-click Cloud IBR provisions on-demand BMC server + storage infrastructure in the event of a disaster and provides automated recovery test reporting for cybersecurity compliance regulations.
Ready to Learn About
Automated Disaster Recovery?
Book a demo today! See the impact and ease of automation…
In less than 30 minutes, you’ll see how easy Cloud IBR makes recovering and testing your Veeam Backups.