# Implementation Map: Database & Module Management Improvements **Created:** 2025-01-14 **Status:** Planning Phase **Total Improvements:** 19 across 6 categories --- ## TOUCHPOINT AUDIT ### Core Files by Size and Impact | File | Lines | Category | Impact Level | |------|-------|----------|--------------| | `scripts/bash/backup-merge.sh` | 1041 | Backup | Medium | | `scripts/bash/manage-modules.sh` | 616 | Module Mgmt | **HIGH** | | `scripts/python/modules.py` | 546 | Module Mgmt | **HIGH** | | `scripts/bash/rebuild-with-modules.sh` | 524 | Build | Low | | `scripts/bash/backup-import.sh` | 473 | Backup | Medium | | `scripts/bash/migrate-stack.sh` | 416 | Deployment | Low | | `scripts/bash/manage-modules-sql.sh` | 381 | **Module SQL** | **CRITICAL** | | `scripts/bash/stage-modules.sh` | 375 | Module Mgmt | Medium | | `scripts/bash/db-import-conditional.sh` | 340 | **DB Import** | **CRITICAL** | | `scripts/python/apply-config.py` | 322 | Config | Medium | | `scripts/bash/backup-export.sh` | 272 | Backup | Low | | `scripts/bash/fix-item-import.sh` | 256 | Backup | Low | | `scripts/bash/backup-scheduler.sh` | 225 | Backup | Medium | | `scripts/bash/download-client-data.sh` | 202 | Setup | Low | | `scripts/bash/verify-deployment.sh` | 196 | Deployment | Low | | `scripts/bash/auto-post-install.sh` | 190 | **Config** | **HIGH** | | `scripts/bash/configure-server.sh` | 163 | Config | Medium | | `scripts/bash/setup-source.sh` | 154 | Setup | Low | **CRITICAL FILES** (Will be modified in Phase 1): 1. `scripts/bash/manage-modules-sql.sh` (381 lines) - Complete refactor 2. `scripts/bash/db-import-conditional.sh` (340 lines) - Add verification 3. `scripts/bash/auto-post-install.sh` (190 lines) - Playerbots DB integration **HIGH IMPACT FILES** (Will be modified in Phase 2-3): 1. `scripts/bash/manage-modules.sh` (616 lines) - SQL staging changes 2. `scripts/python/modules.py` (546 lines) - Minor updates --- ## DETAILED TOUCHPOINT ANALYSIS ### Category A: Module SQL Management #### A1: Refactor Module SQL to Use AzerothCore's System **Files to Modify:** 1. **`scripts/bash/manage-modules-sql.sh`** (381 lines) - **Current Function:** Manually executes SQL files via `mysql_exec` - **Changes Required:** - Remove `run_custom_sql_group()` function - Remove `mysql_exec()` wrapper - Remove `render_sql_file_for_execution()` (playerbots template) - Remove `playerbots_table_exists()` check - Add SQL staging logic to copy files to AzerothCore structure - Add verification via `updates` table query - **Lines to Remove:** ~250 lines (execution logic) - **Lines to Add:** ~50 lines (staging + verification) - **Net Change:** -200 lines 2. **`scripts/bash/manage-modules.sh`** (616 lines) - **Current Function:** Calls `manage-modules-sql.sh` for SQL execution - **Changes Required:** - Update SQL helper invocation (lines 472-606) - Add SQL file staging to proper AzerothCore directory structure - Add timestamp-based filename generation - Add SQL validation before staging - **Lines to Change:** ~50 lines - **Lines to Add:** ~80 lines (staging logic) - **Net Change:** +30 lines 3. **`scripts/python/modules.py`** (546 lines) - **Current Function:** Module manifest management - **Changes Required:** - Add SQL file discovery in module repos - Add SQL file metadata to module state - Generate SQL staging manifest - **Lines to Add:** ~40 lines - **Net Change:** +40 lines **New Files to Create:** 4. **`scripts/bash/stage-module-sql.sh`** (NEW) - **Purpose:** Stage module SQL files to AzerothCore structure - **Functions:** - `copy_sql_to_acore_structure()` - Copy SQL with proper naming - `validate_sql_file()` - Basic SQL syntax check - `generate_sql_timestamp()` - Create YYYYMMDD_HH filename - **Estimated Lines:** ~150 lines 5. **`scripts/bash/verify-sql-updates.sh`** (NEW) - **Purpose:** Verify SQL updates in `updates` table - **Functions:** - `check_update_applied()` - Query updates table - `list_module_updates()` - Show module SQL status - `verify_sql_hash()` - Check hash matches - **Estimated Lines:** ~100 lines **Docker/Config Files:** 6. **`docker-compose.yml`** or relevant compose file - Add volume mount for module SQL staging directory - Ensure `/azerothcore/modules/` is accessible **SQL Directory Structure to Create:** ``` local-storage/source/azerothcore-playerbots/modules/ ├── mod-aoe-loot/ │ └── data/ │ └── sql/ │ ├── base/ │ │ └── db_world/ │ └── updates/ │ └── db_world/ │ └── 20250114_01_aoe_loot_init.sql ├── mod-learn-spells/ │ └── data/ │ └── sql/... └── [other modules...] ``` **Total Impact:** - Files Modified: 3 - Files Created: 2 - Net Code Change: -130 lines (significant reduction!) - Complexity: Medium-High --- #### A2: Add Module SQL Verification **Files to Modify:** 1. **`scripts/bash/verify-sql-updates.sh`** (created in A1) - Already includes verification logic 2. **`scripts/bash/manage-modules.sh`** - Add post-installation verification call - Lines to add: ~20 lines **Total Impact:** - Files Modified: 1 - Code Change: +20 lines - Complexity: Low (builds on A1) --- #### A3: Support Module SQL Rollback **New Files to Create:** 1. **`scripts/bash/rollback-module-sql.sh`** (NEW) - **Purpose:** Rollback module SQL changes - **Functions:** - `create_rollback_sql()` - Generate reverse SQL - `apply_rollback()` - Execute rollback - `track_rollback()` - Update rollback state - **Estimated Lines:** ~200 lines **Module Directory Structure:** ``` modules/mod-example/ └── data/ └── sql/ ├── updates/ │ └── db_world/ │ └── 20250114_01_feature.sql └── rollback/ └── db_world/ └── 20250114_01_feature_rollback.sql ``` **Total Impact:** - Files Created: 1 - Code Change: +200 lines - Complexity: Medium --- ### Category B: Database Restoration & Verification #### B1: Add Post-Restore Verification **Files to Modify:** 1. **`scripts/bash/db-import-conditional.sh`** (340 lines) - **CRITICAL** - **Current Function:** Restores backups or runs dbimport - **Changes Required:** - Add verification step after restore (line ~283-290) - Call dbimport with --dry-run to check state - Apply missing updates if found - Log verification results - **Location:** After `restore_backup` function - **Lines to Add:** ~60 lines **Code Insertion Point:** ```bash # Current code (line ~283): if restore_backup "$backup_path"; then echo "$(date): Backup successfully restored from $backup_path" > "$RESTORE_SUCCESS_MARKER" echo "🎉 Backup restoration completed successfully!" exit 0 fi # ADD HERE: Verification step verify_and_update_databases() { # New function to add } ``` **New Functions to Add:** ```bash verify_and_update_databases() { echo "🔍 Verifying restored database integrity..." cd /azerothcore/env/dist/bin # Check what would be applied local dry_run_output dry_run_output=$(./dbimport --dry-run 2>&1) || true # Parse output to see if updates are needed if echo "$dry_run_output" | grep -q "would be applied"; then warn "Missing updates detected, applying now..." ./dbimport || { err "Update verification failed"; return 1; } else ok "All updates are current" fi # Verify critical tables exist verify_core_tables } verify_core_tables() { # Check that core tables are present local tables=("account" "characters" "creature") # ... verification logic } ``` **Total Impact:** - Files Modified: 1 - Code Change: +60 lines - Complexity: Medium --- #### B2: Use updates Table for State Tracking **Files to Modify:** 1. **`scripts/bash/db-import-conditional.sh`** (340 lines) - **Changes:** Replace marker file checks with SQL queries - **Lines to Change:** ~40 lines - **Lines to Add:** ~30 lines (helper functions) **New Helper Functions:** ```bash is_database_initialized() { local db_name="$1" mysql -h ${CONTAINER_MYSQL} -u${MYSQL_USER} -p${MYSQL_ROOT_PASSWORD} -N -e \ "SELECT COUNT(*) FROM ${db_name}.updates WHERE state='RELEASED'" 2>/dev/null || echo 0 } get_last_update_timestamp() { local db_name="$1" mysql -h ${CONTAINER_MYSQL} -u${MYSQL_USER} -p${MYSQL_ROOT_PASSWORD} -N -e \ "SELECT MAX(timestamp) FROM ${db_name}.updates" 2>/dev/null || echo "" } count_module_updates() { local db_name="$1" mysql -h ${CONTAINER_MYSQL} -u${MYSQL_USER} -p${MYSQL_ROOT_PASSWORD} -N -e \ "SELECT COUNT(*) FROM ${db_name}.updates WHERE state='MODULE'" 2>/dev/null || echo 0 } ``` **Replacement Examples:** ```bash # OLD: if [ -f "$RESTORE_SUCCESS_MARKER" ]; then echo "✅ Backup restoration completed successfully" exit 0 fi # NEW: if is_database_initialized "acore_world"; then local last_update last_update=$(get_last_update_timestamp "acore_world") echo "✅ Database initialized (last update: $last_update)" exit 0 fi ``` **Total Impact:** - Files Modified: 1 - Code Change: +30 lines, -10 lines (marker logic) - Complexity: Low-Medium --- #### B3: Add Database Schema Version Checking **New Files to Create:** 1. **`scripts/bash/check-schema-version.sh`** (NEW) - **Purpose:** Check and report database schema version - **Functions:** - `get_schema_version()` - Query version from DB - `compare_versions()` - Version comparison logic - `warn_version_mismatch()` - Alert on incompatibility - **Estimated Lines:** ~120 lines **Files to Modify:** 2. **`scripts/bash/db-import-conditional.sh`** - Add version check before restore - Lines to add: ~15 lines **Total Impact:** - Files Created: 1 - Files Modified: 1 - Code Change: +135 lines - Complexity: Medium --- #### B4: Implement Database Health Check Script **New Files to Create:** 1. **`scripts/bash/db-health-check.sh`** (NEW) - **Quick Win!** - **Purpose:** Comprehensive database health reporting - **Functions:** - `check_auth_db()` - Auth database status - `check_world_db()` - World database status - `check_characters_db()` - Characters database status - `check_module_updates()` - Module SQL status - `show_database_sizes()` - Storage usage - `list_pending_updates()` - Show pending SQL - `generate_health_report()` - Formatted output - **Estimated Lines:** ~250 lines **Example Output:** ``` 🏥 AZEROTHCORE DATABASE HEALTH CHECK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 📊 Database Status ✅ Auth DB (acore_auth) - Updates: 45 applied - Last update: 2025-01-26 14:30:22 - Size: 12.3 MB ✅ World DB (acore_world) - Updates: 1,234 applied (15 module) - Last update: 2025-01-26 14:32:15 - Size: 2.1 GB ✅ Characters DB (acore_characters) - Updates: 89 applied - Last update: 2025-01-26 14:31:05 - Characters: 145 (5 active today) - Size: 180.5 MB 📦 Module Updates ✅ mod-aoe-loot: 2 updates applied ✅ mod-learn-spells: 1 update applied ✅ mod-playerbots: 12 updates applied ⚠️ Pending Updates - db_world/2025_01_27_00.sql (waiting) - db_world/2025_01_27_01.sql (waiting) 💾 Total Storage: 2.29 GB 🔄 Last backup: 2 hours ago ``` **Total Impact:** - Files Created: 1 - Code Change: +250 lines - Complexity: Low-Medium - **User Value: HIGH** (immediate utility) --- ### Category C: Playerbots Database Integration #### C1: Integrate Playerbots into dbimport **Files to Modify:** 1. **`scripts/bash/db-import-conditional.sh`** (340 lines) - **Changes:** Update dbimport.conf generation (lines 310-327) - **Current:** Only has Login, World, Character DBs - **Add:** PlayerbotsDatabaseInfo line - **Update:** `Updates.EnableDatabases = 15` (was 7) **Code Change:** ```bash # OLD (line 310-318): cat > /azerothcore/env/dist/etc/dbimport.conf < /azerothcore/env/dist/etc/dbimport.conf < "$conf_dir/database.conf" <