feat: comprehensive module system and database management improvements

This commit introduces major enhancements to the module installation system,
database management, and configuration handling for AzerothCore deployments.

## Module System Improvements

### Module SQL Staging & Installation
- Refactor module SQL staging to properly handle AzerothCore's sql/ directory structure
- Fix SQL staging path to use correct AzerothCore format (sql/custom/db_*/*)
- Implement conditional module database importing based on enabled modules
- Add support for both cpp-modules and lua-scripts module types
- Handle rsync exit code 23 (permission warnings) gracefully during deployment

### Module Manifest & Automation
- Add automated module manifest generation via GitHub Actions workflow
- Implement Python-based module manifest updater with comprehensive validation
- Add module dependency tracking and SQL file discovery
- Support for blocked modules and module metadata management

## Database Management Enhancements

### Database Import System
- Add db-guard container for continuous database health monitoring and verification
- Implement conditional database import that skips when databases are current
- Add backup restoration and SQL staging coordination
- Support for Playerbots database (4th database) in all import operations
- Add comprehensive database health checking and status reporting

### Database Configuration
- Implement 10 new dbimport.conf settings from environment variables:
  - Database.Reconnect.Seconds/Attempts for connection reliability
  - Updates.AllowedModules for module auto-update control
  - Updates.Redundancy for data integrity checks
  - Worker/Synch thread settings for all three core databases
- Auto-apply dbimport.conf settings via auto-post-install.sh
- Add environment variable injection for db-import and db-guard containers

### Backup & Recovery
- Fix backup scheduler to prevent immediate execution on container startup
- Add backup status monitoring script with detailed reporting
- Implement backup import/export utilities
- Add database verification scripts for SQL update tracking

## User Import Directory

- Add new import/ directory for user-provided database files and configurations
- Support for custom SQL files, configuration overrides, and example templates
- Automatic import of user-provided databases and configs during initialization
- Documentation and examples for custom database imports

## Configuration & Environment

- Eliminate CLIENT_DATA_VERSION warning by adding default value syntax
- Improve CLIENT_DATA_VERSION documentation in .env.template
- Add comprehensive database import settings to .env and .env.template
- Update setup.sh to handle new configuration variables with proper defaults

## Monitoring & Debugging

- Add status dashboard with Go-based terminal UI (statusdash.go)
- Implement JSON status output (statusjson.sh) for programmatic access
- Add comprehensive database health check script
- Add repair-storage-permissions.sh utility for permission issues

## Testing & Documentation

- Add Phase 1 integration test suite for module installation verification
- Add comprehensive documentation for:
  - Database management (DATABASE_MANAGEMENT.md)
  - Module SQL analysis (AZEROTHCORE_MODULE_SQL_ANALYSIS.md)
  - Implementation mapping (IMPLEMENTATION_MAP.md)
  - SQL staging comparison and path coverage
  - Module assets and DBC file requirements
- Update SCRIPTS.md, ADVANCED.md, and troubleshooting documentation
- Update references from database-import/ to import/ directory

## Breaking Changes

- Renamed database-import/ directory to import/ for clarity
- Module SQL files now staged to AzerothCore-compatible paths
- db-guard container now required for proper database lifecycle management

## Bug Fixes

- Fix module SQL staging directory structure for AzerothCore compatibility
- Handle rsync exit code 23 gracefully during deployments
- Prevent backup from running immediately on container startup
- Correct SQL staging paths for proper module installation
This commit is contained in:
uprightbass360
2025-11-20 18:26:00 -05:00
committed by Deckard
parent 0d83f01995
commit e6231bb4a4
56 changed files with 11298 additions and 487 deletions

11
import/.gitignore vendored Normal file
View File

@@ -0,0 +1,11 @@
# Ignore all files in import directories by default
db/*
conf/*
# But keep the directory structure and examples
!db/examples/
!db/examples/**
!conf/examples/
!conf/examples/**
!.gitignore
!README.md

123
import/README.md Normal file
View File

@@ -0,0 +1,123 @@
# Import Directory
This directory allows you to easily import custom database files and configuration overrides into your AzerothCore server.
## 📁 Directory Structure
```
import/
├── db/ # Database SQL files to import
└── conf/ # Configuration file overrides
```
## 🗄️ Database Import (`import/db/`)
Place your custom SQL files here to import them into the database on server startup or deployment.
### Supported Files
- `auth.sql` - Authentication database updates
- `characters.sql` - Character database updates
- `world.sql` - World database updates
- `*.sql` - Any other SQL files will be imported automatically
### Usage
1. Place your SQL files in `import/db/`:
```bash
cp my_custom_npcs.sql import/db/world.sql
cp my_accounts.sql import/db/auth.sql
```
2. Deploy or restart your server:
```bash
./scripts/bash/import-database-files.sh
```
### Example Files
See `import/db/examples/` for sample SQL files.
## ⚙️ Configuration Import (`import/conf/`)
Place module configuration files here to override default settings.
### Supported Files
Any `.conf` file placed here will be copied to the server's config directory, overriding the default settings.
### Common Configuration Files
- `worldserver.conf` - Core world server settings
- `authserver.conf` - Authentication server settings
- `playerbots.conf` - Playerbot module settings
- `AutoBalance.conf` - AutoBalance module settings
- Any other module `.conf` file
### Usage
1. Create or copy a configuration file:
```bash
cp storage/config/playerbots.conf.dist import/conf/playerbots.conf
```
2. Edit the file with your custom settings:
```ini
AiPlayerbot.MinRandomBots = 100
AiPlayerbot.MaxRandomBots = 200
```
3. Apply the configuration:
```bash
./scripts/bash/configure-server.sh
```
Or use the Python config tool for advanced merging:
```bash
python3 scripts/python/apply-config.py
```
### Configuration Presets
Instead of manual configuration, you can use presets from `config/server-overrides.conf`:
```ini
[worldserver.conf]
Rate.XP.Kill = 2.0
Rate.XP.Quest = 2.0
[playerbots.conf]
AiPlayerbot.MinRandomBots = 100
AiPlayerbot.MaxRandomBots = 200
```
See `config/CONFIG_MANAGEMENT.md` for detailed preset documentation.
## 🔄 Automated Import
Both database and configuration imports are automatically handled during:
- **Initial Setup**: `./setup.sh`
- **Deployment**: `./deploy.sh`
- **Module Staging**: `./scripts/bash/stage-modules.sh`
## 📝 Notes
- Files in `import/` are preserved across deployments
- SQL files are only imported once (tracked by filename hash)
- Configuration files override defaults but don't replace them
- Use `.gitignore` to keep sensitive files out of version control
## 🚨 Best Practices
1. **Backup First**: Always backup your database before importing SQL
2. **Test Locally**: Test imports on a dev server first
3. **Document Changes**: Add comments to your SQL files explaining what they do
4. **Use Transactions**: Wrap large imports in transactions for safety
5. **Version Control**: Keep track of what you've imported
## 📚 Related Documentation
- [Database Management](../docs/DATABASE_MANAGEMENT.md)
- [Configuration Management](../config/CONFIG_MANAGEMENT.md)
- [Module Management](../docs/ADVANCED.md#module-management)

View File

@@ -0,0 +1,24 @@
# Example Playerbots Configuration Override
# Copy this file to import/conf/playerbots.conf and customize
[worldserver]
###################################################################################################
# PLAYERBOTS SETTINGS
###################################################################################################
# Number of random bots
AiPlayerbot.MinRandomBots = 100
AiPlayerbot.MaxRandomBots = 200
# Bot movement speed modifier (1.0 = normal speed)
AiPlayerbot.BotActiveAlone = 1
# Allow bots to form groups with players
AiPlayerbot.AllowPlayerBots = 1
# Bot gear update frequency (in seconds)
AiPlayerbot.BotGearScoreUpdateTime = 600
# Enable random bot login
AiPlayerbot.EnableRandomBots = 1

View File

@@ -0,0 +1,18 @@
-- Example: Add a custom NPC vendor
-- This file demonstrates how to add a custom NPC to your world database
-- Add the NPC template
INSERT INTO `creature_template` (`entry`, `name`, `subname`, `minlevel`, `maxlevel`, `faction`, `npcflag`, `scale`, `unit_class`, `unit_flags`, `type`, `type_flags`, `InhabitType`, `RegenHealth`, `flags_extra`, `ScriptName`)
VALUES
(900000, 'Custom Vendor', 'Example NPC', 80, 80, 35, 128, 1, 1, 0, 7, 0, 3, 1, 2, '');
-- Add the NPC spawn location (Stormwind Trade District)
INSERT INTO `creature` (`guid`, `id1`, `map`, `position_x`, `position_y`, `position_z`, `orientation`, `spawntimesecs`, `MovementType`)
VALUES
(900000, 900000, 0, -8833.38, 628.628, 94.0066, 0.715585, 300, 0);
-- Add some items to sell (optional)
-- INSERT INTO `npc_vendor` (`entry`, `item`, `maxcount`, `incrtime`, `ExtendedCost`)
-- VALUES
-- (900000, 2901, 0, 0, 0), -- Mining Pick
-- (900000, 5956, 0, 0, 0); -- Blacksmith Hammer