Files
AzerothCore-RealmMaster/docs/SCRIPTS.md
uprightbass360 5c9f1d7389 feat: comprehensive module system and database management improvements
This commit introduces major enhancements to the module installation system,
database management, and configuration handling for AzerothCore deployments.

## Module System Improvements

### Module SQL Staging & Installation
- Refactor module SQL staging to properly handle AzerothCore's sql/ directory structure
- Fix SQL staging path to use correct AzerothCore format (sql/custom/db_*/*)
- Implement conditional module database importing based on enabled modules
- Add support for both cpp-modules and lua-scripts module types
- Handle rsync exit code 23 (permission warnings) gracefully during deployment

### Module Manifest & Automation
- Add automated module manifest generation via GitHub Actions workflow
- Implement Python-based module manifest updater with comprehensive validation
- Add module dependency tracking and SQL file discovery
- Support for blocked modules and module metadata management

## Database Management Enhancements

### Database Import System
- Add db-guard container for continuous database health monitoring and verification
- Implement conditional database import that skips when databases are current
- Add backup restoration and SQL staging coordination
- Support for Playerbots database (4th database) in all import operations
- Add comprehensive database health checking and status reporting

### Database Configuration
- Implement 10 new dbimport.conf settings from environment variables:
  - Database.Reconnect.Seconds/Attempts for connection reliability
  - Updates.AllowedModules for module auto-update control
  - Updates.Redundancy for data integrity checks
  - Worker/Synch thread settings for all three core databases
- Auto-apply dbimport.conf settings via auto-post-install.sh
- Add environment variable injection for db-import and db-guard containers

### Backup & Recovery
- Fix backup scheduler to prevent immediate execution on container startup
- Add backup status monitoring script with detailed reporting
- Implement backup import/export utilities
- Add database verification scripts for SQL update tracking

## User Import Directory

- Add new import/ directory for user-provided database files and configurations
- Support for custom SQL files, configuration overrides, and example templates
- Automatic import of user-provided databases and configs during initialization
- Documentation and examples for custom database imports

## Configuration & Environment

- Eliminate CLIENT_DATA_VERSION warning by adding default value syntax
- Improve CLIENT_DATA_VERSION documentation in .env.template
- Add comprehensive database import settings to .env and .env.template
- Update setup.sh to handle new configuration variables with proper defaults

## Monitoring & Debugging

- Add status dashboard with Go-based terminal UI (statusdash.go)
- Implement JSON status output (statusjson.sh) for programmatic access
- Add comprehensive database health check script
- Add repair-storage-permissions.sh utility for permission issues

## Testing & Documentation

- Add Phase 1 integration test suite for module installation verification
- Add comprehensive documentation for:
  - Database management (DATABASE_MANAGEMENT.md)
  - Module SQL analysis (AZEROTHCORE_MODULE_SQL_ANALYSIS.md)
  - Implementation mapping (IMPLEMENTATION_MAP.md)
  - SQL staging comparison and path coverage
  - Module assets and DBC file requirements
- Update SCRIPTS.md, ADVANCED.md, and troubleshooting documentation
- Update references from database-import/ to import/ directory

## Breaking Changes

- Renamed database-import/ directory to import/ for clarity
- Module SQL files now staged to AzerothCore-compatible paths
- db-guard container now required for proper database lifecycle management

## Bug Fixes

- Fix module SQL staging directory structure for AzerothCore compatibility
- Handle rsync exit code 23 gracefully during deployments
- Prevent backup from running immediately on container startup
- Correct SQL staging paths for proper module installation
2025-11-20 18:26:00 -05:00

14 KiB

AzerothCore RealmMaster - Script Reference

This document provides comprehensive documentation for all scripts included in the AzerothCore RealmMaster project. These scripts automate deployment, module management, backup operations, and system administration tasks.

Overview

The AzerothCore RealmMaster includes a comprehensive set of scripts organized into several categories:

  • Core Deployment Scripts - Essential scripts for setup, build, and deployment
  • Container Lifecycle Management - Scripts for managing Docker containers and services
  • Database & Backup Management - Tools for backup, restore, and data management
  • Module Management Scripts - Automated module staging, configuration, and integration
  • Post-Deployment Automation - Scripts that run after deployment for configuration
  • Advanced Deployment Tools - Specialized tools for remote deployment and validation
  • Backup System Scripts - Automated backup scheduling and management

Script Reference

Core Deployment Scripts

setup.sh - Interactive Environment Configuration

Interactive .env generator with module selection, server configuration, and deployment profiles.

./setup.sh                                    # Interactive configuration
./setup.sh --module-config sam                # Use predefined module profile, check profiles directory
./setup.sh --playerbot-max-bots 3000          # Set playerbot limits

Features:

  • Server address and port configuration
  • Module selection with presets
  • Storage path configuration (NFS/local)
  • Playerbot configuration
  • Backup retention settings
  • User/group permission settings

build.sh - Custom Image Builder

Compiles AzerothCore with enabled C++ modules and creates deployment-ready Docker images.

./build.sh                                    # Interactive build
./build.sh --yes                              # Auto-confirm all prompts
./build.sh --force                            # Force rebuild regardless of state
./build.sh --source-path /custom/path         # Use custom source path
./build.sh --skip-source-setup                # Skip source repo setup

What it does:

  • Clones/updates AzerothCore source repository
  • Stages enabled modules into source tree
  • Compiles server binaries with modules
  • Builds and tags Docker images (<project>:authserver-modules-latest, etc.)
  • Updates build state markers

deploy.sh - Deployment Orchestrator

Module-aware deployment with automatic profile selection and optional remote deployment.

./deploy.sh                                    # Interactive deployment
./deploy.sh --yes                              # Auto-confirm deployment
./deploy.sh --profile standard                 # Force standard AzerothCore
./deploy.sh --profile playerbots               # Force playerbots branch
./deploy.sh --profile modules                  # Force custom modules build
./deploy.sh --no-watch                         # Don't tail worldserver logs
./deploy.sh --keep-running                     # Deploy and exit immediately

# Remote deployment
./deploy.sh --remote-host server.com \
            --remote-user username \
            --remote-project-dir /path/to/project

Automated workflow:

  1. Loads environment configuration
  2. Detects required profile based on enabled modules
  3. Triggers build if C++ modules or playerbots enabled
  4. Launches Docker Compose with appropriate profiles
  5. Optionally migrates stack to remote host

cleanup.sh - Project Cleanup Utility

Comprehensive cleanup with multiple destruction levels and safety checks.

./cleanup.sh                                  # Interactive cleanup
./cleanup.sh --soft                           # Stop containers only
./cleanup.sh --hard                           # Remove containers, networks, volumes
./cleanup.sh --nuclear                        # Full cleanup including images
./cleanup.sh --preserve-backups               # Retain backup data during cleanup
./cleanup.sh --dry-run                        # Preview cleanup actions

Container Lifecycle Management

scripts/bash/start-containers.sh - Service Startup

Starts all configured containers using appropriate profiles.

scripts/bash/stop-containers.sh - Graceful Shutdown

Stops all containers with proper cleanup and data protection.

status.sh - Service Health Monitoring

./status.sh                                    # Single status check with summary
./status.sh --watch                           # Continuous monitoring mode
./status.sh --once                            # Script-friendly single check

Database & Backup Management

scripts/bash/backup-export.sh - User Data Export

Exports user accounts and character data for migration or backup purposes.

./scripts/bash/backup-export.sh                            # Export to ExportBackup_<timestamp>/
./scripts/bash/backup-export.sh /path/to/backup/dir       # Export to specific directory

Output Structure:

ExportBackup_YYYYMMDD_HHMMSS/
├── acore_auth.sql.gz         # User accounts
├── acore_characters.sql.gz   # Character data
└── manifest.json             # Backup metadata

scripts/bash/backup-import.sh - User Data Import

Restores user accounts and characters from backup while preserving world data.

./scripts/bash/backup-import.sh --backup-dir storage/backups/ExportBackup_20241029_120000 --password azerothcore123

# Restore directly from an ExportBackup archive you just unpacked
./scripts/bash/backup-import.sh --backup-dir ExportBackup_20241029_120000 --password azerothcore123 --all

The importer always requires --backup-dir. A common workflow is to extract an ExportBackup_* archive into storage/backups/ (so automated jobs can see it) and pass that directory to the script, but you can point to any folder that contains the SQL dumps.

Required Files:

  • acore_auth.sql[.gz] - User accounts (required)
  • acore_characters.sql[.gz] - Character data (required)
  • acore_world.sql[.gz] - World data (optional)

Module Management Scripts

scripts/bash/stage-modules.sh - Module Staging

Downloads and stages enabled modules for source integration.

./scripts/bash/stage-modules.sh                    # Stage all enabled modules

Called automatically by build.sh. Downloads enabled modules from GitHub and prepares them for compilation.

scripts/bash/setup-source.sh - Source Repository Setup

Initializes or updates AzerothCore source repositories for compilation.

./scripts/bash/setup-source.sh                     # Setup source for current configuration

Automatically clones the appropriate AzerothCore fork (main or playerbot) based on configuration.

scripts/bash/manage-modules.sh - Module Management Container

Internal script that runs inside the ac-modules container to handle module lifecycle:

  • Downloads module source code
  • Executes module SQL scripts
  • Manages module configuration files
  • Tracks installation state

config/module-manifest.json & scripts/python/modules.py

Central module registry and management system:

  • config/module-manifest.json - Declarative manifest defining all 30+ supported modules with metadata:
    • Repository URLs
    • Module type (cpp, data, lua)
    • Build requirements
    • SQL scripts and config files
    • Dependencies
  • scripts/python/modules.py - Python helper that reads the manifest and .env to:
    • Generate modules.env with enabled module lists
    • Determine if rebuild is required
    • Provide module metadata to shell scripts

This centralized approach eliminates duplicate module definitions across scripts.

scripts/python/update_module_manifest.py - GitHub Topic Sync

Automates manifest population directly from the official AzerothCore GitHub topics.

# Preview new modules across all default topics
python3 scripts/python/update_module_manifest.py --dry-run --log

# Update config/module-manifest.json with latest repos (requires GITHUB_TOKEN)
GITHUB_TOKEN=ghp_yourtoken python3 scripts/python/update_module_manifest.py --refresh-existing
  • Queries azerothcore-module, azerothcore-lua, azerothcore-sql, azerothcore-tools, and azerothcore-module+ac-premium
  • Merges new repositories without touching existing customizations
  • Optional --refresh-existing flag rehydrates names/descriptions from GitHub
  • Designed for both local execution and the accompanying GitHub Action workflow

scripts/bash/manage-modules-sql.sh - Module Database Integration

Executes module-specific SQL scripts for database schema updates.

scripts/bash/copy-module-configs.sh - Configuration File Management

Creates module .conf files from .dist.conf templates for active modules.

./scripts/bash/copy-module-configs.sh              # Create missing module configs

Post-Deployment Automation

scripts/bash/auto-post-install.sh - Post-Installation Configuration

Automated post-deployment tasks including module configuration, service verification, and initial setup.

./scripts/bash/auto-post-install.sh                # Run post-install tasks

Automated Tasks:

  1. Module configuration file creation
  2. Service health verification
  3. Database connectivity testing
  4. Initial realm configuration

scripts/bash/manual-backup.sh - On-Demand Backup Helper

Runs the ac-backup container's dump logic immediately and stores results under /backups/<label>_<timestamp>.

./scripts/bash/manual-backup.sh                     # Manual backup with default label
./scripts/bash/manual-backup.sh --label hotfix      # Custom label for the backup directory

Advanced Deployment Tools

scripts/bash/migrate-stack.sh - Remote Deployment Migration

Exports and transfers locally built images to remote hosts via SSH.

./scripts/bash/migrate-stack.sh \
  --host docker-server \
  --user sam \
  --project-dir /home/sam/AzerothCore-RealmMaster

./scripts/bash/migrate-stack.sh \
  --host remote.example.com \
  --user deploy \
  --port 2222 \
  --identity ~/.ssh/deploy_key \
  --skip-storage

What it does:

  1. Exports module images to local-storage/images/acore-modules-images.tar
  2. Syncs project files (.env, docker-compose.yml, scripts) via rsync/scp
  3. Syncs storage directory (unless --skip-storage)
  4. Imports images on remote host

Note: Typically called via ./deploy.sh --remote-host rather than directly.

scripts/bash/deploy-tools.sh - Management Tools Deployment

Deploys web-based management tools (phpMyAdmin, Keira3) independently.

./scripts/bash/deploy-tools.sh                     # Deploy management tools only

scripts/bash/verify-deployment.sh - Deployment Validation

Comprehensive deployment verification with health checks and service validation.

./scripts/bash/verify-deployment.sh                        # Full deployment verification
./scripts/bash/verify-deployment.sh --skip-deploy         # Verify existing deployment
./scripts/bash/verify-deployment.sh --quick               # Quick health check only

Backup System Scripts

scripts/bash/backup-scheduler.sh - Automated Backup Service

Runs inside the backup container to provide scheduled database backups.

Features:

  • Hourly backups (retained for 6 hours)
  • Daily backups (retained for 3 days)
  • Automatic cleanup based on retention policies
  • Database detection (includes playerbots if present)

Script Usage Patterns

Common Workflows

Initial Setup and Deployment

# 1. Configure environment and select modules
./setup.sh

# 2. Build custom images (if using C++ modules or playerbots)
./build.sh --yes

# 3. Deploy services
./deploy.sh

Module Configuration Changes

# 1. Reconfigure modules
./setup.sh

# 2. Rebuild if you changed C++ modules
./build.sh --yes

# 3. Redeploy with new configuration
./deploy.sh

Remote Deployment

# 1. Configure and build locally
./setup.sh
./build.sh --yes

# 2. Deploy to remote host
./deploy.sh --remote-host server.com --remote-user username

Service Management

# Check service status
./status.sh

# Monitor continuously
./status.sh --watch

# Stop all services
./scripts/bash/stop-containers.sh

# Start services
./scripts/bash/start-containers.sh

Backup and Restore Operations

# Export user data for migration
./scripts/bash/backup-export.sh

# Import user data from backup
./scripts/bash/backup-import.sh /path/to/backup

# Verify deployment health
./scripts/bash/verify-deployment.sh --quick

Project Cleanup

# Soft cleanup (stop containers)
./cleanup.sh --soft

# Full cleanup with backup preservation
./cleanup.sh --nuclear --preserve-backups

Script Dependencies and Requirements

System Requirements

  • Docker and Docker Compose
  • Bash 4.0+ (most scripts)
  • Python 3.6+ (module management)
  • rsync and ssh (remote deployment)
  • Standard Unix utilities (grep, awk, sed, etc.)

Environment Variables

Most scripts rely on environment variables defined in .env. Key variables include:

  • PROJECT_NAME - Docker image and container naming
  • STORAGE_PATH - Primary storage location
  • STORAGE_PATH_LOCAL - Local build storage
  • Module toggles (MODULE_* variables)
  • Docker Compose profile settings
  • Backup retention configurations

File Dependencies

  • .env - Environment configuration (created by setup.sh)
  • docker-compose.yml - Container orchestration
  • config/module-manifest.json - Module definitions
  • config/module-profiles/*.json - Module presets

Troubleshooting Scripts

When scripts encounter issues, use these debugging approaches:

General Debugging

# Run scripts with debug output
bash -x ./script-name.sh

# Check script exit codes
echo $? # after running a script
# Check Docker daemon
systemctl status docker

# Verify Docker Compose
docker compose version

# Check container logs
docker logs <container-name>

Environment Issues

# Verify .env file
cat .env | grep -v '^#'

# Check module configuration
./scripts/python/modules.py --list-enabled

Permission Issues

# Check file ownership
ls -la storage/

# Fix permissions (if needed)
sudo chown -R $USER:$USER storage/

For additional help and troubleshooting information, refer to the main README.md file or check the project repository issues.