Backup Preparations¶

On this page

Backup Sizing Recommendation
Snapshot Frequency and Retention Policy
Backup Considerations

Before backing up your cluster or replica set, decide how to back up the data and what data to back up. This page describes items you must consider before starting a backup.

Backup Sizing Recommendation¶

The backup recommendation depends on the Feature Compatibility Version of the database that you want to back up.

FCV of 4.0 or earlier
FCV of 4.2 or later

When sizing the backup of your data, keep the replica set size to 2 TB or less of uncompressed data. If your database increases beyond 2 TB of uncompressed data:

Shard the database
Keep each shard to 2 TB or less of uncompressed data

These size recommendations are a best practice. They are not a limitation of the MongoDB database or Cloud Manager.

Backup and restore can use large amounts of CPU, memory, storage, and network bandwidth.

Example

Your stated network throughput, such as 10 Gbps or 100 Gbps, is a theoretical maximum. That value doesn’t account for sharing or throttling of network traffic.

Consider the following scenario:

You want to back up a 2 TB database.
Your hosts support a 10 Gbps TCP connection from Cloud Manager to its backup storage.
The network connection has very low packet loss and a low round trip delay time.

A full backup of your data would take more than 30 hours to complete. [*]

This doesn’t account for disk read and write speeds, which can be, at most, 3 Gbps reads and 1 Gbps writes for a single or mirrored NVMe storage device.

The time required to complete each successive incremental backup depends on write load.

If you shard this database into 4 shards, each shard runs its backup separately. This results in a backup that takes less than 8 hours to complete.

[*]	These throughput figures were calculated using the Network Throughput Calculator and assume no additional network compression.

When sizing the backup of your data, keep the replica set size to 2 TB or less of compressed data. If your database increases beyond 2 TB of compressed data:

Shard the database
Keep each shard to 2 TB or less of compressed data

These size recommendations are a best practice. They are not a limitation of the MongoDB database or Cloud Manager.

Backup and restore can use large amounts of CPU, memory, storage, and network bandwidth.

Example

Your stated network throughput, such as 10 Gbps or 100 Gbps, is a theoretical maximum. That value doesn’t account for sharing or throttling of network traffic.

Consider the following scenario:

You want to back up a 2 TB database.
Your hosts support a 10 Gbps TCP connection from Cloud Manager to its backup storage.
The network connection has very low packet loss and a low round trip delay time.

A full backup of your data would take more than 30 hours to complete. [†]

This doesn’t account for disk read and write speeds, which can be, at most, 3 Gbps reads and 1 Gbps writes for a single or mirrored NVMe storage device.

The time required to complete each successive incremental backup depends on write load.

If you shard this database into 4 shards, each shard runs its backup separately. This results in a backup that takes less than 8 hours to complete.

[†]	These throughput figures were calculated using the Network Throughput Calculator and assume no additional network compression.

Snapshot Frequency and Retention Policy¶

By default, Cloud Manager takes a base snapshot of your data every 6 hours.

If desired, administrators can change the frequency of base snapshots to 6, 8, 12, or 24 hours. Cloud Manager creates snapshots automatically on a schedule; you cannot take snapshots on demand.

Cloud Manager retains snapshots for the time periods listed in the following table.

If you terminate a deployment’s backup, Cloud Manager immediately deletes the snapshots that are within the dates of the current retention policy.

If you stop a deployment’s backup, Cloud Manager stops taking new snapshots but retains existing snapshots until their listed expiration date.

Snapshot	Default Retention Policy	Maximum Retention Policy
Base snapshot	2 days	5 days (30 days if frequency is 24 hours)
Daily snapshot	7 days	1 year
Weekly snapshot	4 weeks	1 year
Monthly snapshot	13 months	7 years

Changes to the snapshot schedule affect your snapshot storage costs.

You can change a backed-up deployment’s schedule through its Edit Snapshot Schedule menu option, available through the Backup page. Administrators can change snapshot frequency and retention through the snapshotSchedule resource in the API.

You can’t change the reference time for a snapshot in Cloud Manager.

Limits¶

Cloud Manager does not backup deployments where the total number of collections on the deployment meets or exceeds 100,000.
Cloud Manager does not replicate index collection options.

Encryption¶

You can’t store backups using WiredTiger encryption. You can store backups using the WiredTiger storage engine if you don’t enable encryption. If you restore from a backup, you restore unencrypted files.

Backup Considerations¶

Databases Running FCV 4.2 and 4.4¶

Backup support for the following MongoDB versions is growing but currently limited:

4.2 with "featureCompatibilityVersion" : 4.2
4.4 with "featureCompatibilityVersion" : 4.4

Support will be extended in future releases of Cloud Manager.

Backup Features Supported at Present¶

Feature	MongoDB 4.4 with FCV : 4.4	MongoDB 4.4 with FCV : 4.2	MongoDB 4.2 with FCV : 4.2	MongoDB 4.2 with FCV : 4.0	MongoDB 4.0 or earlier
Backs up Data using WiredTiger Snapshots	check circle icon	check circle icon	check circle icon
Backs up Replica Sets	check circle icon	check circle icon	check circle icon	check circle icon	check circle icon
Backs up Sharded Clusters	check circle icon	check circle icon	check circle icon	check circle icon	check circle icon
Can Filter using Namespaces				check circle icon	check circle icon
Can Specify Sync Source Database				check circle icon	check circle icon
Can Restore Data to Specific Point in Time	check circle icon	check circle icon	check circle icon	check circle icon	check circle icon
Can Perform Incremental Backups [1]	check circle icon	check circle icon	check circle icon	check circle icon	check circle icon
Supports Databases running MongoDB Enterprise	check circle icon	check circle icon	check circle icon	check circle icon	check circle icon
Supports Databases running MongoDB Community [2]				check circle icon	check circle icon
Requires a MongoDB Agent with backup enabled on every `mongod` cluster node	check circle icon	check circle icon	check circle icon

[1]

Cloud Manager requires a full backup for your first backup, after a snapshot has been deleted, and if the blockstore block size has been changed. Incremental backups reduce network transfer and storage costs.

This feature works with:

MongoDB 4.0 and earlier.
MongoDB 4.2.6 or later if running FCV 4.2 or later.

[2]	Cloud Manager grants a special license to use MongoDB Enterprise to MongoDB Community users for backup only.

Requirements and Limitations¶

To run backups and restores if you are running MongoDB 4.2 or later with FCV 4.2 or later, you:

Must run MongoDB Enterprise. MongoDB, Inc. grants a special license to use MongoDB Enterprise for Cloud Manager backups.

Can’t use namespace filter lists to define the namespaces included in a backup. Snapshots using FCV 4.2 or later always include all namespaces.
Don’t need a sync source database. When taking a Snapshot, Cloud Manager selects the replica set member with the least performance impact and greatest storage-level duplication of Snapshot data.
Must deploy a MongoDB Agent with every mongod node in the cluster.

Note

If Cloud Manager doesn’t manage your cluster:

Grant the backup and clusterAdmin roles to the MongoDB user that runs backups.
Ensure that the operating system user that runs the MongoDB Agent has read permission for all data files (including journal files) of the deployment.

Databases Running FCV 4.0 and Earlier¶

Important

Only sharded clusters or replica sets can be backed up. To back up a standalone mongod process, you must convert it to a single-member replica set.

The following considerations apply when your databases run any version of MongoDB 4.0 or earlier or when they run MongoDB 4.2 with "featureCompatibilityVersion" : 4.0

Garbage Collection of Expired Snapshots¶

Cloud Manager manages expired snapshots using groom jobs. These groom jobs act differently depending upon which snapshot store contains the snapshots:

Snapshot Store How Groom Jobs Work

MongoDB blockstore Uses additional disk space up to the amount of living blocks for each job.

Filesystem snapshot stores Deletes expired snapshots

S3 snapshot stores

For FCV 4.0 or earlier, may use additional disk space if Cloud Manager creates a snapshot while the groom job is running.

For FCV 4.2 or later, Cloud Manager can’t create snapshots while a groom job is running.

Namespaces Filter¶

The namespaces filter lets you specify which databases and collections to back up. You create either a Blacklist of those to exclude or a Whitelist of those to include. You make your selections when starting a backup and can later edit them as needed. If you change the filter in a way that adds data to your backup, a resync is required.

Use the blacklist to prevent backup of collections that contain logging data, caches, or other ephemeral data. Excluding these kinds of databases and collections will allow you to reduce backup time and costs. Using a blacklist is often preferable to using a whitelist as a whitelist requires you to intentionally opt in to every namespace you want backed up.

Note

MongoDB deployments with "featureCompatibilityVersion" : 4.2 do not support namespaces filters.

Storage Engine¶

To back up MongoDB clusters, use the WiredTiger storage engine storage engine.

If your current backing databases use MMAPv1, upgrade to WiredTiger:

With WiredTiger, Cloud Manager limits backups to deployments with fewer than 100,000 files. Files includes collections and indexes.

MongoDB 4.2 removes MMAPv1 storage. To learn more about storage engines, see Storage in the MongoDB manual.

Resyncing Production Deployments¶

For production deployments, it is recommended that as a best practice you periodically (annually) resync all backed-up replica sets. When you resync, data is read from a secondary in each replica set. During resync, no new snapshots are generated.

You may also want to resync your backup after:

A reduction in data size, such that the size on disk of Cloud Manager’s copy of the data is also reduced. This scenario also includes if you:
- Have a TTL index in place, which periodically deletes documents.
- Drop a collection (MMAPv1 only).
- Run a sharded cluster, and there have been a lot of chunks moved off a particular shard.
A switch in storage engines, if you want Cloud Manager to provide snapshots in the new storage engine format.
A manual build of an index on a replica set in a rolling fashion (as per Build Indexes on Replica Sets in the MongoDB manual).

Checkpoints¶

Important

You may use checkpoints for clusters that run MongoDB with Feature Compatibility Version of 4.0 or earlier. Checkpoints were removed from MongoDB instances with FCV of 4.2 or later.

For sharded clusters, checkpoints provide additional restore points between snapshots. With checkpoints enabled, Cloud Manager creates restoration points at configurable intervals of every 15, 30 or 60 minutes between snapshots. To enable checkpoints, see enable checkpoints.

To create a checkpoint, Cloud Manager stops the balancer and inserts a token into the oplog of each shard and config server in the cluster. These checkpoint tokens are lightweight and do not have a consequential impact on performance or disk use.

Backup does not require checkpoints, and they are disabled by default.

Restoring from a checkpoint requires Cloud Manager to apply the oplog of each shard and config server to the last snapshot captured before the checkpoint. Restoration from a checkpoint takes longer than restoration from a snapshot.

Snapshots when Agent Cannot Stop Balancer¶

For sharded clusters running with FCV 4.0 or earlier, Cloud Manager disables the balancer before taking a cluster snapshot. In certain situations, such as a long migration or no running mongos, Cloud Manager tries to disable the balancer but cannot. In such cases, Cloud Manager will continue to take cluster snapshots but will flag the snapshots with a warning that data may be incomplete and/or inconsistent. Cluster snapshots taken during an active balancing operation run the risk of data loss or orphaned data.

Snapshots when Agent Cannot Contact a `mongod`¶

For sharded clusters, if the Backup cannot reach a mongod process, whether a shard or config server, then the agent cannot insert a synchronization oplog token. If this happens, Cloud Manager does not create the snapshot and displays a warning message.

← Back up MongoDB Deployments Back up a Deployment →

Cloud Manager will no longer support Automation for MongoDB 3.4 clusters after February 10, 2022. Please upgrade your MongoDB deployment or migrate to Atlas.