Files
AzerothCore-RealmMaster/docs/MODULE_ASSETS_ANALYSIS.md
uprightbass360 e6231bb4a4 feat: comprehensive module system and database management improvements
This commit introduces major enhancements to the module installation system,
database management, and configuration handling for AzerothCore deployments.

## Module System Improvements

### Module SQL Staging & Installation
- Refactor module SQL staging to properly handle AzerothCore's sql/ directory structure
- Fix SQL staging path to use correct AzerothCore format (sql/custom/db_*/*)
- Implement conditional module database importing based on enabled modules
- Add support for both cpp-modules and lua-scripts module types
- Handle rsync exit code 23 (permission warnings) gracefully during deployment

### Module Manifest & Automation
- Add automated module manifest generation via GitHub Actions workflow
- Implement Python-based module manifest updater with comprehensive validation
- Add module dependency tracking and SQL file discovery
- Support for blocked modules and module metadata management

## Database Management Enhancements

### Database Import System
- Add db-guard container for continuous database health monitoring and verification
- Implement conditional database import that skips when databases are current
- Add backup restoration and SQL staging coordination
- Support for Playerbots database (4th database) in all import operations
- Add comprehensive database health checking and status reporting

### Database Configuration
- Implement 10 new dbimport.conf settings from environment variables:
  - Database.Reconnect.Seconds/Attempts for connection reliability
  - Updates.AllowedModules for module auto-update control
  - Updates.Redundancy for data integrity checks
  - Worker/Synch thread settings for all three core databases
- Auto-apply dbimport.conf settings via auto-post-install.sh
- Add environment variable injection for db-import and db-guard containers

### Backup & Recovery
- Fix backup scheduler to prevent immediate execution on container startup
- Add backup status monitoring script with detailed reporting
- Implement backup import/export utilities
- Add database verification scripts for SQL update tracking

## User Import Directory

- Add new import/ directory for user-provided database files and configurations
- Support for custom SQL files, configuration overrides, and example templates
- Automatic import of user-provided databases and configs during initialization
- Documentation and examples for custom database imports

## Configuration & Environment

- Eliminate CLIENT_DATA_VERSION warning by adding default value syntax
- Improve CLIENT_DATA_VERSION documentation in .env.template
- Add comprehensive database import settings to .env and .env.template
- Update setup.sh to handle new configuration variables with proper defaults

## Monitoring & Debugging

- Add status dashboard with Go-based terminal UI (statusdash.go)
- Implement JSON status output (statusjson.sh) for programmatic access
- Add comprehensive database health check script
- Add repair-storage-permissions.sh utility for permission issues

## Testing & Documentation

- Add Phase 1 integration test suite for module installation verification
- Add comprehensive documentation for:
  - Database management (DATABASE_MANAGEMENT.md)
  - Module SQL analysis (AZEROTHCORE_MODULE_SQL_ANALYSIS.md)
  - Implementation mapping (IMPLEMENTATION_MAP.md)
  - SQL staging comparison and path coverage
  - Module assets and DBC file requirements
- Update SCRIPTS.md, ADVANCED.md, and troubleshooting documentation
- Update references from database-import/ to import/ directory

## Breaking Changes

- Renamed database-import/ directory to import/ for clarity
- Module SQL files now staged to AzerothCore-compatible paths
- db-guard container now required for proper database lifecycle management

## Bug Fixes

- Fix module SQL staging directory structure for AzerothCore compatibility
- Handle rsync exit code 23 gracefully during deployments
- Prevent backup from running immediately on container startup
- Correct SQL staging paths for proper module installation
2025-11-22 16:56:02 -05:00

14 KiB

Module Assets Analysis - DBC Files and Source Code

Date: 2025-11-16 Purpose: Verify handling of module DBC files, source code, and client patches


Module Asset Types Found

1. Source Code (C++ Modules)

Location: /azerothcore/modules/*/src/ Count: 1,489 C++ files (.cpp and .h) across all enabled modules Purpose: Server-side gameplay logic

Examples Found:

  • /azerothcore/modules/mod-npc-beastmaster/src/
  • /azerothcore/modules/mod-global-chat/src/
  • /azerothcore/modules/mod-guildhouse/src/

Status: FULLY HANDLED

How It Works:

  1. Modules compiled into Docker image during build
  2. Source code included in image but NOT actively compiled at runtime
  3. C++ code already executed as part of worldserver binary
  4. Runtime module repositories provide:
    • SQL files (staged by us)
    • Configuration files (managed by manage-modules.sh)
    • Documentation/README

Conclusion: Source code is build-time only. Pre-built images already contain compiled module code. No runtime action needed.


2. DBC Files (Database Client Files)

Location: /azerothcore/modules/*/data/patch/DBFilesClient/ Found in: mod-worgoblin (custom race module) Count: 20+ custom DBC files for new race

Example Files Found:

/azerothcore/modules/mod-worgoblin/data/patch/DBFilesClient/
├── ChrRaces.dbc          # Race definitions
├── CharBaseInfo.dbc      # Character stats
├── CharHairGeosets.dbc   # Hair models
├── CharacterFacialHairStyles.dbc
├── CharStartOutfit.dbc   # Starting gear
├── NameGen.dbc           # Name generation
├── TalentTab.dbc         # Talent trees
├── Faction.dbc           # Faction relations
└── ...

Purpose: Client-side data that defines:

  • New races/classes
  • Custom spells/items
  • UI elements
  • Character customization

Status: ⚠️ NOT AUTOMATICALLY DEPLOYED


3. Client Patch Files (MPQ Archives)

Found in Multiple Modules:

storage/modules/aio-blackjack/patch-W.MPQ
storage/modules/mod-arac/Patch-A.MPQ
storage/modules/prestige-and-draft-mode/Client Side Files/Mpq Patch/patch-P.mpq
storage/modules/horadric-cube-for-world-of-warcraft/Client/Data/zhCN/patch-zhCN-5.MPQ

Purpose: Pre-packaged client patches containing:

  • DBC files
  • Custom textures/models
  • UI modifications
  • Sound files

Status: ⚠️ USER MUST MANUALLY DISTRIBUTE


4. Other Client Assets

mod-worgoblin patch directory structure:

storage/modules/mod-worgoblin/data/patch/
├── Character/      # Character models
├── Creature/       # NPC models
├── DBFilesClient/  # DBC files
├── ITEM/           # Item models
├── Interface/      # UI elements
├── Sound/          # Audio files
└── Spells/         # Spell effects

Status: ⚠️ NOT PACKAGED OR DEPLOYED


How DBC Files Work in AzerothCore

Server-Side DBC

Location: /azerothcore/data/dbc/ Purpose: Server reads these to understand game rules Source: Extracted from vanilla WoW 3.3.5a client

Current Status:

$ docker exec ac-worldserver ls /azerothcore/data/dbc | wc -l
1189 DBC files present

Server has standard DBC files (from client-data download)

Client-Side DBC

Location: Player's WoW/Data/ folder (or patch MPQ) Purpose: Client reads these to:

  • Display UI correctly
  • Render spells/models
  • Generate character names
  • Show tooltips

Critical: Client and server DBCs must match!


Official AzerothCore DBC Deployment Process

For Module Authors:

  1. Create Modified DBCs:

    • Use DBC editor tools
    • Modify necessary tables
    • Export modified .dbc files
  2. Package for Distribution:

    • Create MPQ patch file (e.g., Patch-Z.MPQ)
    • Include all modified DBCs
    • Add any custom assets (models, textures)
  3. Server Deployment:

    • Copy DBCs to /azerothcore/data/dbc/ (overwrites vanilla)
    • Restart server
  4. Client Distribution:

    • Distribute patch MPQ to all players
    • Players place in WoW/Data/ directory
    • Players restart game

For Server Admins:

Manual Steps Required:

  1. Download module patch from README/releases
  2. Apply server-side DBCs
  3. Host patch file for players to download
  4. Instruct players to install patch

Current Implementation Status

What We Handle Automatically

  1. Module SQL - Staged to core updates directory
  2. Module Config - Deployed to worldserver config directory
  3. Module Compilation - Pre-built into Docker images
  4. Standard DBC - Downloaded via client-data scripts

What We DON'T Handle ⚠️

  1. Custom Module DBCs - Not deployed to server DBC directory
  2. Client Patch Files - Not distributed to players
  3. Client Assets - Not packaged or made available
  4. DBC Synchronization - No validation that client/server match

Gap Analysis

Modules Requiring Client Patches

From our analysis, these modules have client-side requirements:

Module Client Assets Server DBCs Impact if Missing
mod-worgoblin Yes (extensive) Yes NEW RACE WON'T WORK
mod-arac Yes (Patch-A.MPQ) Yes Class/race combos broken
aio-blackjack Yes (patch-W.MPQ) Unknown UI elements missing
prestige-and-draft-mode Yes (patch-P.mpq) Unknown Features unavailable
horadric-cube Yes (patch-zhCN-5.MPQ) Unknown Locale-specific broken

Severity Assessment

mod-worgoblin (CRITICAL):

  • Adds entirely new playable race (Worgen/Goblin)
  • Requires 20+ modified DBC files
  • Without patch: Players can't create/see race correctly
  • Currently broken - DBCs not deployed

mod-arac (HIGH):

  • "All Races All Classes" - removes restrictions
  • Requires modified class/race DBC tables
  • Without patch: Restrictions may still apply client-side
  • Potentially broken - needs verification

Others (MEDIUM/LOW):

  • Gameplay features may work server-side
  • UI/visual elements missing client-side
  • Degraded experience but not completely broken

Why We Don't Auto-Deploy Client Patches

Technical Reasons

  1. Client patches are player-specific

    • Each player must install manually
    • No server-side push mechanism
    • Requires download link/instructions
  2. Version control complexity

    • Different locales (enUS, zhCN, etc.)
    • Different client versions
    • Naming conflicts between modules
  3. File hosting requirements

    • MPQ files can be 10MB+ each
    • Need web server or file host
    • Update distribution mechanism
  4. Testing/validation needed

    • Must verify client compatibility
    • Risk of corrupting client
    • Hard to automate testing

Architectural Reasons

  1. Docker images are server-only

    • Don't interact with player clients
    • Can't modify player installations
    • Out of scope for server deployment
  2. Module isolation

    • Each module maintains own patches
    • No central patch repository
    • Version conflicts possible
  3. Admin responsibility

    • Server admin chooses which modules
    • Must communicate requirements to players
    • Custom instructions per module

Current Best Practice

Our Implementation:

  1. Deploy module source (pre-compiled in image)
  2. Deploy module SQL (runtime staging)
  3. Deploy module config files (manage-modules.sh)
  4. ⚠️ Document client patch requirements (user responsibility)

This matches official AzerothCore guidance:

  • Server-side automation where possible
  • Client-side patches distributed manually
  • Admin reads module README for requirements

Enhanced Documentation 📝

What We Should Add:

  1. Module README Scanner

    • Detect client patch requirements
    • Warn admin during deployment
    • Link to download instructions
  2. Client Patch Detection

    • Scan for *.MPQ, *.mpq files
    • Check for data/patch/ directories
    • Report found patches in deployment log
  3. Deployment Checklist

    • List modules with client requirements
    • Provide download links (from module repos)
    • Instructions for player distribution

Example Output:

⚠️  Client Patches Required:

  mod-worgoblin:
    📦 Patch: storage/modules/mod-worgoblin/Patch-Z.MPQ
    📋 Instructions: See storage/modules/mod-worgoblin/README.md
    🔗 Download: https://github.com/azerothcore/mod-worgoblin/releases

  mod-arac:
    📦 Patch: storage/modules/mod-arac/Patch-A.MPQ
    📋 Instructions: Players must install to WoW/Data/

⚠️  Server admins must distribute these patches to players!

Server-Side DBC Deployment (Possible Enhancement)

What Could Be Automated

If modules include server DBCs:

modules/mod-worgoblin/
└── data/
    ├── sql/          # ✅ We handle this
    ├── dbc/          # ❌ We don't handle this
    │   ├── ChrRaces.dbc
    │   └── ...
    └── patch/        # ❌ Client-side (manual)
        └── ...

Potential Enhancement:

# In stage-modules.sh, add DBC staging:
if [ -d "$module_dir/data/dbc" ]; then
  echo "📦 Staging server DBCs for $module_name..."
  cp -r "$module_dir/data/dbc/"* /azerothcore/data/dbc/
  echo "⚠️  Server restart required to load new DBCs"
fi

Risks:

  • ⚠️ Overwrites vanilla DBCs (could break other modules)
  • ⚠️ No conflict detection between modules
  • ⚠️ No rollback mechanism
  • ⚠️ Requires worldserver restart (not just reload)

Recommendation: DON'T AUTO-DEPLOY server DBCs

  • Too risky without validation
  • Better to document in README
  • Admin can manually copy if needed

Source Code Compilation

How It Works in Standard Setup

Official Process:

  1. Clone module to /modules/ directory
  2. Run CMake (detects new module)
  3. Recompile entire core
  4. Module C++ code compiled into worldserver binary

CMake Module Detection:

# CMake scans for modules during configuration
foreach(module_dir ${CMAKE_SOURCE_DIR}/modules/*)
  if(EXISTS ${module_dir}/CMakeLists.txt)
    add_subdirectory(${module_dir})
  endif()
endforeach()

How It Works With Pre-Built Images

Docker Image Build Process:

  1. Modules cloned during image build
  2. CMake runs with all enabled modules
  3. Worldserver compiled with modules included
  4. Binary contains all module code

Runtime (Our Deployment):

  1. Image already has compiled modules
  2. Mount module repositories for:
    • SQL files (we stage these)
    • Config files (we deploy these)
    • README/docs (reference only)
  3. Source code in repository is NOT compiled

Verification:

# Module code is inside the binary
$ docker exec ac-worldserver worldserver --version
# Shows compiled modules

# Source code exists but isn't used
$ docker exec ac-worldserver ls /azerothcore/modules/mod-*/src/
# Files present but not actively compiled

Status: FULLY HANDLED

No action needed for source code:

  • Pre-built images contain all enabled modules
  • Source repositories provide SQL/config only
  • Recompilation would require custom build (out of scope)

Comparison: Official vs. Our Implementation

Asset Type Official Process Our Implementation Status
C++ Source Compile at build Pre-compiled in image COMPLETE
SQL Files Applied by DBUpdater Runtime staging COMPLETE
Config Files Manual deployment Automated by manage-modules COMPLETE
Server DBCs Manual copy to /data/dbc Not deployed ⚠️ DOCUMENTED
Client Patches Distribute to players Not distributed ⚠️ USER RESPONSIBILITY
Client Assets Package in MPQ Not packaged ⚠️ MANUAL

Recommendations

Keep Current Approach

What we do well:

  1. SQL staging - automated and secure
  2. Config management - fully automated
  3. Source handling - correctly uses pre-built binaries
  4. Clear separation of server vs. client concerns

Add Documentation 📝

Enhance deployment output:

  1. Detect modules with client patches
  2. Warn admin about distribution requirements
  3. Provide links to patch files and instructions
  4. Create post-deployment checklist

Don't Implement (Too Risky)

What NOT to automate:

  1. Server DBC deployment - risk of conflicts
  2. Client patch distribution - technically impossible from server
  3. Module recompilation - requires custom build process
  4. Client asset packaging - out of scope

Summary

Current Status: SOUND ARCHITECTURE

What We Handle:

  • Module source code (via pre-built images)
  • Module SQL (runtime staging)
  • Module configuration (automated deployment)

What Requires Manual Steps:

  • ⚠️ Server DBC deployment (module README instructions)
  • ⚠️ Client patch distribution (admin responsibility)
  • ⚠️ Player communication (outside automation scope)

No Critical Gaps

All gaps identified are by design:

  • Client-side patches can't be auto-deployed (technical limitation)
  • Server DBCs shouldn't be auto-deployed (safety concern)
  • Module READMEs must be read (standard practice)

Our implementation correctly handles what can be automated while documenting what requires manual steps.


Modules Requiring Special Attention

High Priority (Client Patches Required)

mod-worgoblin:

  • Status: Likely broken without client patch
  • Action: Check README, distribute Patch-Z.MPQ to players
  • Impact: New race completely unavailable

mod-arac:

  • Status: Needs verification
  • Action: Distribute Patch-A.MPQ to players
  • Impact: Race/class restrictions may apply incorrectly

Medium Priority (Enhanced Features)

aio-blackjack, prestige-and-draft-mode, horadric-cube:

  • Status: Core functionality may work, UI missing
  • Action: Optional patch distribution for full experience
  • Impact: Degraded but functional

Conclusion: Our implementation is complete for automated deployment. Client patches and server DBCs correctly remain manual tasks with proper documentation.