MongoSync Setup and Operations

This guide provides setup, configuration, initial sync, operations, monitoring, and troubleshooting guidance for AppViewX MongoSync based DR replication.

Overview

This section provides step-by-step instructions to set up, configure, and operate MongoSync for near real-time MongoDB replication between AppViewX clusters. It covers architecture, prerequisites, installation, initial synchronization, and Day-2 operations such as pause, resume, failover, and password rotation. It also includes monitoring, command reference, and troubleshooting guidance for a stable disaster recovery posture.

Audience:

  • AppViewX customers operating on-prem DR with MongoSync.
  • AppViewX Customer Support.

Architecture overview

MongoSync provides continuous replication from a source AppViewX cluster, where MongoSync runs, to a destination cluster that acts as DR.

  • Replication is typically one-way: source is active and destination receives the copy.
  • Destination MongoDB is exposed through NodePorts across clusters.
  • The sync engine connects source MongoDB as cluster0 and destination MongoDB as cluster1.
  • A reverse sync path is used when roles are swapped after failover.
SOURCE CLUSTER (mongodb rs0 + mongosync) --sync--> DESTINATION CLUSTER (mongodb rs0 over NodePorts 30096+N)

Prerequisites

  • Network

    Allow MongoDB NodePort traffic bidirectionally between source and destination clusters. Ports are assigned sequentially starting at 30096, one per MongoDB StatefulSet member.

    MongoDB pod NodePort
    mongodb-0 30096
    mongodb-1 30097
    mongodb-2 30098
    mongodb-N 30096+N

    Example: two MongoDB pods require 30096 and 30097 open both ways; three pods require 30096, 30097, and 30098.

  • Vault (OpenBao)

    Encrypted MongoDB passwords must decrypt with the same Vault transit key material on the cluster that runs MongoSync (source).

    • Recommended: restore Vault/OpenBao backup from source to destination before enabling sync, so both sites share key material.
    • Encrypt passwords with ./appviewx.sh --password-encrypt on the cluster that will decrypt them.
  • MongoDB server version

    MongoSync 1.21.x supports MongoDB major versions such as 8.0. It does not support rapid/minor server lines such as 8.1, 8.2, or 8.3.

    Ensure source and destination MongoDB are on a supported 8.0.x patch before enabling sync.

  • Passwords

    Special characters are handled by product scripts, but alphanumeric passwords are recommended to reduce edge cases in connection strings.

Extract artifacts

Perform on both source and destination clusters.

  1. Extract the package:
cd <INSTALLER_DIR>
tar -xzf appviewx_mongosync.tar.gz
cd appviewx_mongosync/
  1. Move required files into place:
mv mongo_sync ../yaml/
cp -rf mongosync_scripts/* ../scripts/

This places assets under yaml/mongo_sync/ and utilities such as mongosync_utility.sh under scripts/.

Configuration

Edit scripts/appviewx.conf on each cluster. Plan host and port pairs from destination pod placement and fixed port sequence 30096, 30097, and so on.

Discover destination MongoDB placement on destination cluster:

kubectl get pods -n avx -o wide | grep mongodb
  1. Source cluster

    Update appviewx.conf with:

    DR_SYNC_ENABLED=TRUE
MONGO_DR_USERNAME=admin
MONGO_SYNC_DR_CLUSTER=<host:port>[,<host:port>...]
DR_MONGO_ENCRYPTED_PASSWORD="vault:v1:<encrypted_string>"

    Mapping rules:

    • List destination MongoDB members in order: first entry is mongodb-0, second is mongodb-1, and so on.
    • Port mapping: mongodb-0 to 30096, mongodb-1 to 30097, mongodb-N to 30096+N.
    • Host is FQDN or resolvable hostname of the node where each mongodb-N pod runs.

    Encrypt destination MongoDB password on source:

    ./appviewx.sh --password-encrypt
  2. Destination cluster

    Set in appviewx.conf:

    DR_SYNC_ENABLED=TRUE
MONGO_DR_USERNAME=admin
MONGO_SYNC_DR_CLUSTER=<dest-node>:<port>[,...]

    DR_MONGO_ENCRYPTED_PASSWORD is not required on destination for normal DR sync setup.

Initial sync setup

Run commands from <INSTALLER_PATH>/scripts on the relevant cluster.

  • Destination prepare DR MongoDB

    This exposes MongoDB via NodePort, prepares destination databases for DR flow, scales down application tiers, and reconfigures replica set external hostnames.

    ./mongosync_utility.sh stop --destination
  • Source prepare DR MongoDB

    Source start MongoSync

    ./mongosync_utility.sh start --source

    This deploys MongoSync Helm workflow and starts replication.

  • Verify on source
    ./mongosync_utility.sh progress --source

    Expected:

    • state is RUNNING
    • info reaches change event application after initial copy
    • lagTimeSeconds stabilizes
  • AppViewX v2024.2.0.0 only

    After start --source, verify chart:

    helm ls -A | grep avx-mongo-sync

    If missing, run from scripts:

    helm install avx-mongo-sync ../yaml/mongo_sync/mongo_sync/chart \
--set common.appviewx_path=/home/appviewx/appviewx/ \
--set common.user_id=$(id -u) \
--namespace avx --create-namespace

Day-to-day operations

  • Pause sync

    Before maintenance, pause from both clusters.

    On source:

    ./mongosync_utility.sh pause --source

    On destination:

    ./mongosync_utility.sh pause --destination
  • Resume sync

    After maintenance on both sites:

    On destination:

    ./mongosync_utility.sh resume --destination

    On source:

    ./mongosync_utility.sh resume --source
  • Progress (source only)
    ./mongosync_utility.sh progress --source
  • Unhealthy commit (destination)

    Use only when source is unavailable and destination must be forced to commit.

    ./mongosync_utility.sh unhealthy-commit --destination

Reverse sync (failover)

  1. Pause on current source: ./mongosync_utility.sh pause --source
  2. Pause on current destination: ./mongosync_utility.sh pause --destination
  3. Update appviewx.conf for new source and destination.
  4. On new destination: ./mongosync_utility.sh stop --destination
  5. On new source: ./mongosync_utility.sh start --source

Database password rotation

  • Change password on a cluster
    ./appviewx.sh --change-db-password

    If target user is admin, the tool updates MONGO_ENCRYPTED_PASSWORD in avx-common-config across namespaces and restarts related workloads.

  • Change destination DB password from source
    ./appviewx.sh --change-db-password --destination

    When target matches MONGO_DR_USERNAME, tool scales MongoSync down, updates password, updates DR_MONGO_ENCRYPTED_PASSWORD on source, then restores MongoSync.

  • Destination ConfigMaps after DR password change

    Option A: re-encrypt locally on destination:

    ./appviewx.sh --change-db-password

    Option B: copy ciphertext from source and patch destination ConfigMaps:

    kubectl get cm avx-common-config -n avx -o jsonpath='{.data.DR_MONGO_ENCRYPTED_PASSWORD}'

kubectl patch cm avx-common-config -n avx --type merge -p '{"data":{"MONGO_ENCRYPTED_PASSWORD":"<value>"}}'
kubectl patch cm avx-common-config -n avx-jobs --type merge -p '{"data":{"MONGO_ENCRYPTED_PASSWORD":"<value>"}}'
kubectl patch cm avx-common-config -n <dc-namespace> --type merge -p '{"data":{"MONGO_ENCRYPTED_PASSWORD":"<value>"}}'

kubectl rollout restart deployment -n avx
kubectl rollout restart deployment -n avx-jobs
kubectl rollout restart deployment -n <dc-namespace>

Monitoring and health

  • Progress URLs via ingress
    URL Purpose
    https://<ingress>:31443/mongosync/progress/status In-sync or not-in-sync style status
    https://<ingress>:31443/mongosync/progress/details Full JSON progress output
  • Pod checks
    kubectl get pods -l app=mongosync -n avx
kubectl logs -l app=mongosync -n avx --tail=100
  • Log files
    <INSTALLATION_PATH>/logs/mongosync_logs/

Command quick reference

  • Source commands
    ./mongosync_utility.sh start --source
./mongosync_utility.sh pause --source
./mongosync_utility.sh resume --source
./mongosync_utility.sh progress --source
  • Destination commands
    ./mongosync_utility.sh stop --destination
./mongosync_utility.sh pause --destination
./mongosync_utility.sh resume --destination
./mongosync_utility.sh unhealthy-commit --destination
  • Password and Vault helper commands
    ./appviewx.sh --change-db-password
./appviewx.sh --change-db-password --destination
./appviewx.sh --password-encrypt

Troubleshooting

Symptom What to check
MongoSync pod not healthy or not Ready Vault connectivity; DR_MONGO_ENCRYPTED_PASSWORD and MONGO_ENCRYPTED_PASSWORD in avx-common-config; firewall for required NodePorts; logs under <INSTALLATION_PATH>/logs/mongosync_logs/
AuthenticationFailed errors Wrong or empty decrypted password; ciphertext encrypted under different key; re-encrypt with --password-encrypt on decrypting cluster and update ConfigMap
Auth failures after DR password change Re-run --change-db-password on destination with same old/new password, or patch ConfigMaps as documented in section 8.3
Version compatibility errors MongoDB must be on supported major line 8.0.x; coordinate with AppViewX before server version changes
Lag increasing continuously Network path to destination NodePorts; destination disk space and MongoDB writable state; mongosync logs
Connection issues Verify MONGO_SYNC_DR_CLUSTER FQDNs and ports; confirm pod placement still matches appviewx.conf after reschedule
Progress command fails Confirm MongoSync pod is running on source; progress is source-only

Removing Helm release

Not for normal operations. Use only for cleanup:

helm uninstall avx-mongo-sync

Prefer mongosync_utility.sh for day-to-day lifecycle.