This commit introduces major enhancements to the module installation system, database management, and configuration handling for AzerothCore deployments. ## Module System Improvements ### Module SQL Staging & Installation - Refactor module SQL staging to properly handle AzerothCore's sql/ directory structure - Fix SQL staging path to use correct AzerothCore format (sql/custom/db_*/*) - Implement conditional module database importing based on enabled modules - Add support for both cpp-modules and lua-scripts module types - Handle rsync exit code 23 (permission warnings) gracefully during deployment ### Module Manifest & Automation - Add automated module manifest generation via GitHub Actions workflow - Implement Python-based module manifest updater with comprehensive validation - Add module dependency tracking and SQL file discovery - Support for blocked modules and module metadata management ## Database Management Enhancements ### Database Import System - Add db-guard container for continuous database health monitoring and verification - Implement conditional database import that skips when databases are current - Add backup restoration and SQL staging coordination - Support for Playerbots database (4th database) in all import operations - Add comprehensive database health checking and status reporting ### Database Configuration - Implement 10 new dbimport.conf settings from environment variables: - Database.Reconnect.Seconds/Attempts for connection reliability - Updates.AllowedModules for module auto-update control - Updates.Redundancy for data integrity checks - Worker/Synch thread settings for all three core databases - Auto-apply dbimport.conf settings via auto-post-install.sh - Add environment variable injection for db-import and db-guard containers ### Backup & Recovery - Fix backup scheduler to prevent immediate execution on container startup - Add backup status monitoring script with detailed reporting - Implement backup import/export utilities - Add database verification scripts for SQL update tracking ## User Import Directory - Add new import/ directory for user-provided database files and configurations - Support for custom SQL files, configuration overrides, and example templates - Automatic import of user-provided databases and configs during initialization - Documentation and examples for custom database imports ## Configuration & Environment - Eliminate CLIENT_DATA_VERSION warning by adding default value syntax - Improve CLIENT_DATA_VERSION documentation in .env.template - Add comprehensive database import settings to .env and .env.template - Update setup.sh to handle new configuration variables with proper defaults ## Monitoring & Debugging - Add status dashboard with Go-based terminal UI (statusdash.go) - Implement JSON status output (statusjson.sh) for programmatic access - Add comprehensive database health check script - Add repair-storage-permissions.sh utility for permission issues ## Testing & Documentation - Add Phase 1 integration test suite for module installation verification - Add comprehensive documentation for: - Database management (DATABASE_MANAGEMENT.md) - Module SQL analysis (AZEROTHCORE_MODULE_SQL_ANALYSIS.md) - Implementation mapping (IMPLEMENTATION_MAP.md) - SQL staging comparison and path coverage - Module assets and DBC file requirements - Update SCRIPTS.md, ADVANCED.md, and troubleshooting documentation - Update references from database-import/ to import/ directory ## Breaking Changes - Renamed database-import/ directory to import/ for clarity - Module SQL files now staged to AzerothCore-compatible paths - db-guard container now required for proper database lifecycle management ## Bug Fixes - Fix module SQL staging directory structure for AzerothCore compatibility - Handle rsync exit code 23 gracefully during deployments - Prevent backup from running immediately on container startup - Correct SQL staging paths for proper module installation
10 KiB
Dead Code Analysis - Module SQL Staging
Date: 2025-11-16 Context: Phase 1 SQL Staging Implementation Status: 🔍 Analysis Complete
Executive Summary
After implementing runtime SQL staging in stage-modules.sh, we discovered that the original build-time SQL staging system is no longer functional and creates dead code. The build-time system stages SQL to module directories that AzerothCore's DBUpdater never scans.
Key Finding: AzerothCore's DBUpdater ONLY scans /azerothcore/data/sql/updates/ (core directory), NOT /azerothcore/modules/*/data/sql/updates/ (module directories).
Dead Code Identified
1. Build-Time SQL Staging in manage-modules.sh
File: scripts/bash/manage-modules.sh
Lines: 480-557
Functions:
stage_module_sql_files()(lines 480-551)execute_module_sql()(lines 553-557)
What it does:
- Called during
build.sh(image build process) - Stages SQL to
/azerothcore/modules/*/data/sql/updates/db_world/ - Creates properly named SQL files with timestamps
- Intended to let AzerothCore's native updater find them
Why it's dead:
- AzerothCore's DBUpdater does NOT scan module directories
- Files created here are NEVER read or executed
- Confirmed by checking worldserver logs - no module SQL from this location
Evidence:
$ docker exec ac-worldserver ls /azerothcore/modules/mod-npc-beastmaster/data/sql/updates/db_world/
2025_11_15_00_npc_beastmaster.sql # ❌ NEVER PROCESSED
2025_11_15_01_beastmaster_tames.sql # ❌ NEVER PROCESSED
2025_11_15_02_beastmaster_tames_inserts.sql # ❌ NEVER PROCESSED
$ docker logs ac-worldserver 2>&1 | grep "2025_11_15_00_npc_beastmaster"
# NO RESULTS - File was never found by DBUpdater
2. Stand-alone stage-module-sql.sh Script
File: scripts/bash/stage-module-sql.sh
Lines: 297 lines total
Purpose: Called by manage-modules.sh to stage individual module SQL
What it does:
- Takes module path and target path as arguments
- Discovers SQL files in module
- Copies them with proper naming to target directory
- Validates SQL files (security checks)
Why it's potentially dead:
- Only called by
manage-modules.sh:527(which is dead code) - NOT called by the working runtime staging in
stage-modules.sh - The runtime staging does direct docker exec copying instead
Current usage:
- ✅ Called by
manage-modules.sh(build-time - DEAD) - ❌ NOT called by
stage-modules.sh(runtime - ACTIVE) - ✅ Referenced by
test-phase1-integration.sh(test script)
Status: Potentially useful - Could be refactored for runtime use, but currently not in active code path
3. SQL Manifest System
Files:
scripts/python/modules.py- Generates.sql-manifest.jsonlocal-storage/modules/.sql-manifest.json- Generated manifest file
What it does:
- Scans all modules during state generation
- Creates JSON manifest of all module SQL files
- Includes metadata: file paths, database types, checksums
- Used by
manage-modules.shto know which SQL to stage
Why it's potentially dead:
- Created during build process
- Consumed by
manage-modules.sh:stage_module_sql_files()(dead code) - NOT used by runtime staging in
stage-modules.sh
Current usage:
- ✅ Generated by
modules.py generatecommand - ✅ Read by
manage-modules.sh(build-time - DEAD) - ❌ NOT used by
stage-modules.sh(runtime - ACTIVE) - ✅ Checked by
test-phase1-integration.sh(test script)
Status: Potentially useful - Contains valuable metadata but not in active deployment path
4. Test Files in /tmp
Files:
/tmp/test-discover.sh- Testing SQL discovery logic/tmp/test-sql-staging.log- Deployment test output
Status: Temporary test files - Should be cleaned up
Working System (NOT Dead Code)
Runtime SQL Staging in stage-modules.sh
File: scripts/bash/stage-modules.sh
Lines: 372-450
Function: stage_module_sql_to_core()
What it does:
- Starts containers (including worldserver)
- Waits for worldserver to be running
- Uses
docker execto scan/azerothcore/modules/*/data/sql/db-world/(source files) - Copies SQL to
/azerothcore/data/sql/updates/db_world/(core directory) - Renames with timestamp prefix:
YYYY_MM_DD_HHMMSS_{counter}_MODULE_{module_name}_{original}.sql - AzerothCore's DBUpdater automatically processes them on startup
Evidence of success:
$ docker logs ac-worldserver 2>&1 | grep "Applying update" | grep MODULE
>> Applying update "2025_11_16_010945_0_MODULE_data_arac.sql" '025553C'...
>> Applying update "2025_11_16_010945_6_MODULE_data_beastmaster_tames.sql" '8C65AB2'...
# ✅ 46 MODULE SQL files successfully applied
Status: ✅ ACTIVE AND WORKING
Architecture Comparison
Build-Time Staging (DEAD)
build.sh
└─> manage-modules.sh
└─> stage_module_sql_files()
└─> stage-module-sql.sh
└─> Copies SQL to: /azerothcore/modules/*/data/sql/updates/db_world/
└─> ❌ DBUpdater never scans this location
Runtime Staging (ACTIVE)
deploy.sh
└─> stage-modules.sh
└─> stage_module_sql_to_core()
└─> Direct docker exec copying
└─> Copies SQL to: /azerothcore/data/sql/updates/db_world/
└─> ✅ DBUpdater scans and processes this location
Recommended Actions
Option 1: Complete Removal (Aggressive)
Remove:
stage_module_sql_files()function frommanage-modules.sh(lines 480-551)execute_module_sql()function frommanage-modules.sh(lines 553-557)scripts/bash/stage-module-sql.sh(entire file - 297 lines)- SQL manifest generation from
modules.py - Test files:
/tmp/test-discover.sh,/tmp/test-sql-staging.log
Update:
test-phase1-integration.sh- Remove SQL manifest checksbuild.sh- Remove call to SQL staging (if present)
Pros:
- Removes ~400 lines of dead code
- Simplifies architecture to single SQL staging approach
- Eliminates confusion about which system is active
Cons:
- Loses standalone
stage-module-sql.shtool (could be useful for manual operations) - Loses SQL manifest metadata (though not currently used)
Option 2: Refactor and Reuse (Conservative)
Keep but refactor:
- Keep
stage-module-sql.shas a standalone tool for manual SQL staging - Update it to stage to core directory (
/azerothcore/data/sql/updates/) instead of module directory - Document that it's a manual tool, not part of automated deployment
- Keep SQL manifest as optional metadata for debugging
Remove:
stage_module_sql_files()andexecute_module_sql()frommanage-modules.sh- Automated call to staging during build process
- Test files in
/tmp
Update:
- Document
stage-module-sql.shas manual/utility tool - Update its target directory logic to match runtime approach
- Add clear comments explaining the architecture
Pros:
- Preserves utility scripts for manual operations
- Maintains SQL discovery/validation logic
- More flexible for future use cases
Cons:
- Still carries some dead weight
- More complex to maintain
Option 3: Hybrid Approach (Recommended)
Phase 1 - Immediate Cleanup:
- Remove
stage_module_sql_files()andexecute_module_sql()frommanage-modules.sh - Remove automated SQL staging from build process
- Delete test files from
/tmp - Update
test-phase1-integration.shto test runtime staging instead
Phase 2 - Refactor for Future:
- Keep
stage-module-sql.shbut mark it clearly as "UTILITY - Not in deployment path" - Update it to stage to core directory for manual use cases
- Keep SQL manifest generation but make it optional
- Document the runtime staging approach as the canonical implementation
Pros:
- Immediate removal of dead code from active paths
- Preserves potentially useful utilities for future
- Clear documentation of what's active vs. utility
- Flexibility for future enhancements
Cons:
- Still maintains some unused code
- Requires clear documentation to prevent confusion
Impact Analysis
If We Remove All Dead Code
Build Process:
- ✅ No impact - build doesn't need SQL staging
- ✅ Modules still built correctly with C++ code
- ✅ Source SQL files still included in module directories
Deployment Process:
- ✅ No impact - runtime staging handles everything
- ✅ All 46 module SQL files still applied correctly
- ✅ AzerothCore's autoupdater still works
Testing:
- ⚠️ Need to update
test-phase1-integration.sh - ⚠️ Remove SQL manifest checks
- ✅ Can add runtime staging verification instead
Future Development:
- ⚠️ Loses SQL discovery logic (but it's reimplemented in runtime staging)
- ⚠️ Loses SQL validation logic (security checks for shell commands)
- ✅ Simpler architecture is easier to maintain
Decision Required
Question for User: Which cleanup approach should we take?
- Aggressive - Remove all dead code completely
- Conservative - Refactor and keep as utilities
- Hybrid - Remove from active paths, keep utilities documented
Recommendation: Hybrid approach - Remove dead code from active deployment/build paths while preserving utility scripts for future manual operations.
Files Summary
| File | Lines | Status | Recommendation |
|---|---|---|---|
manage-modules.sh:480-557 |
78 | Dead Code | Remove functions |
stage-module-sql.sh |
297 | Not in active path | Refactor as utility |
modules.py (SQL manifest) |
~50 | Generated but unused | Keep as optional |
/tmp/test-discover.sh |
~30 | Test file | Delete |
/tmp/test-sql-staging.log |
N/A | Test output | Delete |
test-phase1-integration.sh |
N/A | Needs update | Update to test runtime staging |
stage-modules.sh:372-450 |
78 | ✅ ACTIVE | Keep (working code) |
Total Dead Code: ~450 lines across multiple files
Next Step: Await user decision on cleanup approach, then proceed with selected option.