Navigation
You were redirected from a different version of the documentation. Click here to go back.

Automation

On this page

Cloud Manager Automation allows you to deploy, configure, and manage MongoDB deployments with the Cloud Manager UI. Cloud Manager Automation relies on an MongoDB Agent, which must be installed on every server in the deployment. The MongoDB Agents periodically poll the Cloud Manager service to determine the current goal, and continually report their status to Cloud Manager.

Automation Runs Only on 64-bit Architectures

Cloud Manager provides only 64-bit downloads of the MongoDB Agent.

Using Own Hardware

  • If you deploy Automation manually, ensure that you have one MongoDB Agent on every server.

  • If you deploy the agent manually, you must create MongoDB’s dbpath and the directory for the MongoDB binaries and ensure that the user running the agent owns these directories.

    If you install using the rpm package, the agent runs as the mongod user; if using the deb package, the agent runs as the mongodb user. If you install using the tar.gz archive file, you can run the agent as any user.

Networking

Connectivity to MongoDB Ports

All hosts must be able to allow communication between MongoDB ports. The default is 27017, but you can configure alternate port ranges in the Cloud Manager interface.

The MongoDB Agent must be able to connect to cloud.mongodb.com on port 443 (HTTPS). For more information on access to ports and IP addresses, see Security Overview.

Intra-cluster Connectivity Issue

When performing a rolling update, the MongoDB Agent tries to avoid downtime. It needs to collect state from other members of the cluster. A connectivity issue (between mongods and mongoses), such as hostname resolution or misconfigured firewall, may prevent the MongoDB Agent from determining the remote processes state and completing the change.

To ensure all members of the cluster can communicate with each other:

  1. For a non-sharded cluster:
    1. Log into each mongod.
    2. From that mongod host, log into all the other members of the replica set.
  2. For a sharded cluster:
    1. Log into each mongod.
    2. From that mongod host, log into all the other members of the shard.
    3. Log into each mongos.
    4. From that the mongos host:
      1. Log into the other mongos hosts.
      2. Log into all the other members of each shard.

Frequent Automation Connections

MongoDB Agent gathers the state from each member of the cluster every 10 seconds to ensure that the environment is in the expected state. As part of this assessment, MongoDB Agent creates a connection, checks certain files to determine state, and then closes the connection. These frequent, short-lived connections are part of MongoDB Agent’s routine activity and should not impact performance.

Automation Configuration

After completing the automation configuration, ensure that the deployment plan satisfies the needs of your deployment. Check hostnames and ports before confirming the deployment.

Sizing

  • Ensure that you provision hosts with enough space to run MongoDB and support the requirements of your data set.
  • Ensure that you provision a sufficient number of hosts to run your deployment. Each mongod should run on its own host.

Frequent Connection Timeouts

The MongoDB Agent may frequently time out of connections for one or more of the following reasons:

  • Connection timeouts
  • High network latency
  • High server load
  • Large SSL keys
  • Insufficient CPU speed

By default, connections time out after 40 seconds. MongoDB recommends gradually increasing the value of the dialTimeoutSeconds MongoDB Agent configuration setting to prevent frequent premature connection timeouts. However, increasing this value also increases the time required to deploy future configuration changes. Experiment with small, incremental increases until you determine the optimum value for your deployment.

To learn more, see dialTimeoutSeconds in MongoDB Agent Connection Settings.

Deployments

A banner that displays We have detected a potential problem while deploying… appears when certain conditions apply. These are some examples.

Deployment Change Unable to Complete / Not Proceeding

If you have added or restarted a deployment and the deployment remains in the In Progress state for several minutes, the banner displays.

At this point, you have four options:

  1. Click View Diagnostics.

    The View Diagnostics modal displays any errors that may have happened during deployment.

  2. Click View Status.

    The Automation Status modal displays the deployment processes, when they last reported their deployment status, what task they are trying to complete, and the deployment status.

    To learn more about the status of any of the individual processes, you can click the ellipsis icon and select either View Details or View Agent Logs.

    • View Details shows what the deployment plan for the process is and which stage of that plan the MongoDB Agent currently is.
    • View Agent Logs opens a new browser window with the Deployment > Agents > Agent Logs screen shown and the contents of the MongoDB Agent log displayed by default. Click the View menu to select a different agent log.

  3. Click View Agent Logs.

    A new browser window opens with the Deployment > Agents > Agent Logs screen shown and the contents of the MongoDB Agent log displayed by default. Click the View menu to select a different agent log.

  4. Click Allow Override & Edit Configuration.

    If you diagnose an error and need to correct the deployment configuration, follow the procedure to edit the deployment.

If you shut down the deployment and still cannot find a solution, remove the deployment from Cloud Manager.

MongoDB Agents Are Down or Not Communicating

If Automation Module of MongoDB Agent can’t communicate with the Cloud Manager API endpoint or the MongoDB Server processes, Cloud Manager displays a warning banner in the Project. You can resolve this in one of two ways depending upon whether or not you expect the MongoDB Agents to be communicating:

MongoDB Agents need to communicate

If the MongoDB Agent(s) should be communicating with the Cloud Manager host or MongoDB instances, confirm the following for each MongoDB Agent:

  1. The Agent is up and running on each host.
  2. The Agent and the Cloud Manager API endpoint can communicate.

MongoDB Agents Do Not Need to Communicate

If Automation Module of MongoDB Agent(s) should be communicating with the Cloud Manager API endpoint or MongoDB Server processes, confirm the following for each automated MongoDB Server deployment:

  1. Click the Allow Editing & Override Current Configuration link in the warning banner.
  2. Remove all processes (mongod and mongos) running on the hosts that serve the unneeded MongoDB Agents.

MongoDB Agent Permissions

A permissions issue may prevent automation from completing a change. If View Status or View Diagnostics report an permissions-related error (such as open /data/db/mongod.lock: permission denied), ensure that the MongoDB Agent user owns and has read and write permissions to the dbpath and logpath files.

Remove Problematic MongoDB Hosts

You can use the console or the API to remove stale, broken, or problematic hosts from automation. This may include the circumstance when the MongoDB Agent can’t be reached.

To remove a problematic host using the console:

  1. Navigate to your project.

  2. Click Servers.

  3. Find your problematic host.

  4. Click ellipsis icon , then Remove Server.

    Cloud Manager displays the Are you sure you want to remove this server> modal.

  5. Enter the provided hostname into the Enter the host name field an click Remove if you want to remove this server.

    Warning

    Cloud Manager removes all monitoring data for this host when you click Remove. Cloud Manager provides no confirmation or cancellation for this action.

    If you don’t want to remove the server, click Cancel.

  6. Click Review & Deploy to review your changes.

    Cloud Manager displays your proposed changes.

    • If you are satisfied, click Confirm & Deploy.

      Cloud Manager removes all processes and agents at this time.

    • If you want to make further configuration changes, click Cancel.

To remove a problematic host using the API:

  1. Get the current automation config.
  2. Edit the automation configuration JSON file.
  3. Remove the stale node from processes and replica sets.
  4. Update the automation config file.
  5. Wait for a few minutes.
  6. Check the Agents view.
  7. Confirm the host no longer appears on the list of Agents.

Ensure TLS Certificates Contain a Subject Alternative Name

Warning

The MongoDB Agent from version 10.29.0.6856 requires TLS certificates to include a value in the Subject Alternative Name field. Before upgrading to MongoDB Agent 10.29.0.6856-1 or later, make sure that all TLS certificates used in your MongoDB deployment contain a SAN. [1]

To temporarily use TLS certificates without a SAN, set the $GODEBUG environment variable. This workaround will not be supported in upcoming releases of the MongoDB Agent.

To temporarily use TLS certificates without a SAN, perform one of the following actions:

  1. Upgrade your MongoDB Agent packages, which automatically sets the $GODEBUG environment variable on your host, or
  2. Manually set the $GODEBUG environment variable for the Agent process to x509ignoreCN=0 on the host on which you will install the RPM, DEB, or Windows packages.

In Go 1.17, the GODEBUG=x509ignoreCN=0 flag won’t be supported.

[1]

The MongoDB Agent is written in the Go language. Go 1.15 changed the default behavior for using the X.509 CommonName field in Go 1.15.

When clients validate TLS certificates, the client checks the hostname or hostnames to which the certs apply from the values in the cert’s SAN or Subject Distinguished Name (DN) fields. When creating TLS certificates, some people would use the Subject Common Name (CN) field to store the hostname. CNs have limitations that make them a poor choice to store hostnames. These limits include a 64-character maximum length and no support for Name Constraints. RFC 2818 deprecated using CN to store hostnames in May 2000. This RFC required clients to fall back to the CN if the certificate had no values in the SAN field. RFC 6125 removed the requirement in 2011.

Go 1.15 disables adherence to RFC 2818 and uses the RFC 6125 practice of making CN optional. In practice, this change requires you to either add SAN values or enable the use of CNs.

To learn more, see Fraser Tweedale’s blog post on this topic

Store Configuration Files in Memory for Existing Clusters

If you use Cloud Manager version 4.2 or versions 4.4.0 - 4.4.6, you may encounter errors when setting enableLocalConfigurationServer to true and restarting your MongoDB Agent.

This issue only affects existing clusters where enableLocalConfigurationServer is set to true after the cluster is created. Setting this value before creating the cluster does not trigger this issue.

To safely change this setting for existing clusters:

  1. At the end of your MongoDB Agent configuration file, add:

    copy 
    enableLocalConfigurationServer=true

  2. Shut down each process managed by the MongoDB Agent.

  3. Restart the MongoDB Agent by running the following command:

    copy 
    service mongodb-mms-automation-agent restart

  4. Restart the MongoDB processes that you shut down.

  5. Verify that the automation-mongod.conf file has the __rest expansion directive.

For more information on storing configuration files in memory, see Configure How the MongoDB Agent Manages Config Files and Passwords.

←   Authentication Monitoring  →

© MongoDB, Inc 2008-present. MongoDB, Mongo, and the leaf logo are registered trademarks of MongoDB, Inc.