module setup process

This commit is contained in:
uprightbass360
2025-11-17 02:23:53 -05:00
parent ea908dbbcf
commit d3484a3aea
30 changed files with 7685 additions and 430 deletions

773
docs/DATABASE_MANAGEMENT.md Normal file
View File

@@ -0,0 +1,773 @@
# AzerothCore Database Management Guide
**Version:** 1.0
**Last Updated:** 2025-01-14
This guide covers all aspects of database management in your AzerothCore deployment, including backups, restores, migrations, and troubleshooting.
---
## Table of Contents
- [Overview](#overview)
- [Database Structure](#database-structure)
- [Backup System](#backup-system)
- [Restore Procedures](#restore-procedures)
- [Health Monitoring](#health-monitoring)
- [Module SQL Management](#module-sql-management)
- [Migration & Upgrades](#migration--upgrades)
- [Troubleshooting](#troubleshooting)
- [Best Practices](#best-practices)
---
## Overview
### Databases in AzerothCore
Your server uses four primary databases:
| Database | Purpose | Size (typical) |
|----------|---------|----------------|
| **acore_auth** | Account authentication, realm list | Small (< 50MB) |
| **acore_world** | Game world data (creatures, quests, items) | Large (1-3GB) |
| **acore_characters** | Player character data | Medium (100MB-1GB) |
| **acore_playerbots** | Playerbot AI data (if enabled) | Small (< 100MB) |
### Update System
AzerothCore uses a built-in update system that:
- Automatically detects and applies SQL updates on server startup
- Tracks applied updates in the `updates` table (in each database)
- Uses SHA1 hashes to prevent duplicate execution
- Supports module-specific updates
---
## Database Structure
### Core Tables by Database
**Auth Database (acore_auth)**
- `account` - User accounts
- `account_access` - GM permissions
- `realmlist` - Server realm configuration
- `updates` - Applied SQL updates
**World Database (acore_world)**
- `creature` - NPC spawns
- `gameobject` - Object spawns
- `quest_template` - Quest definitions
- `item_template` - Item definitions
- `updates` - Applied SQL updates
**Characters Database (acore_characters)**
- `characters` - Player characters
- `item_instance` - Player items
- `character_spell` - Character spells
- `character_inventory` - Equipped/bagged items
- `updates` - Applied SQL updates
### Updates Table Structure
Every database has an `updates` table:
```sql
CREATE TABLE `updates` (
`name` varchar(200) NOT NULL, -- Filename (e.g., 2025_01_14_00.sql)
`hash` char(40) DEFAULT '', -- SHA1 hash of file
`state` enum('RELEASED','CUSTOM','MODULE','ARCHIVED','PENDING'),
`timestamp` timestamp DEFAULT CURRENT_TIMESTAMP,
`speed` int unsigned DEFAULT '0', -- Execution time (ms)
PRIMARY KEY (`name`)
);
```
**Update States:**
- `RELEASED` - Official AzerothCore updates
- `MODULE` - Module-specific updates
- `CUSTOM` - Your custom SQL changes
- `ARCHIVED` - Historical updates (consolidated)
- `PENDING` - Queued for application
---
## Backup System
### Automated Backups
The system automatically creates backups on two schedules:
**Hourly Backups**
- Frequency: Every N minutes (default: 60)
- Retention: Last N hours (default: 6)
- Location: `storage/backups/hourly/YYYYMMDD_HHMMSS/`
**Daily Backups**
- Frequency: Once per day at configured hour (default: 09:00)
- Retention: Last N days (default: 3)
- Location: `storage/backups/daily/YYYYMMDD_HHMMSS/`
### Configuration
Edit `.env` to configure backup settings:
```bash
# Backup intervals
BACKUP_INTERVAL_MINUTES=60 # Hourly backup frequency
BACKUP_RETENTION_HOURS=6 # How many hourly backups to keep
BACKUP_RETENTION_DAYS=3 # How many daily backups to keep
BACKUP_DAILY_TIME=09 # Daily backup hour (00-23)
# Additional databases
BACKUP_EXTRA_DATABASES="" # Comma-separated list
```
### Manual Backups
Create an on-demand backup:
```bash
./scripts/bash/manual-backup.sh --label my-backup-name
```
Options:
- `--label NAME` - Custom backup name
- `--container NAME` - Backup container name (default: ac-backup)
Output location: `manual-backups/LABEL_YYYYMMDD_HHMMSS/`
### Export Backups
Create a portable backup for migration:
```bash
./scripts/bash/backup-export.sh \
--password YOUR_MYSQL_PASSWORD \
--auth-db acore_auth \
--characters-db acore_characters \
--world-db acore_world \
--db auth,characters,world \
-o ./export-location
```
This creates: `ExportBackup_YYYYMMDD_HHMMSS/` with:
- Compressed SQL files (.sql.gz)
- manifest.json (metadata)
---
## Restore Procedures
### Automatic Restore on Startup
The system automatically detects and restores backups on first startup:
1. Searches for backups in priority order:
- `/backups/daily/` (latest)
- `/backups/hourly/` (latest)
- `storage/backups/ExportBackup_*/`
- `manual-backups/`
2. If backup found:
- Restores all databases
- Marks restoration complete
- Skips schema import
3. If no backup:
- Creates fresh databases
- Runs `dbimport` to populate schemas
- Applies all pending updates
### Restore Safety Checks & Sentinels
Because MySQL stores its hot data in a tmpfs (`/var/lib/mysql-runtime`) while persisting only backups and status markers under `local-storage/mysql-data`, it is possible for the runtime data to be wiped (for example, after a host reboot) while the sentinel `.restore-completed` file still claims the databases are ready. To prevent the worldserver and authserver from entering restart loops, the `ac-db-import` workflow now performs an explicit sanity check before trusting those markers:
- The import script queries MySQL for the combined table count across `acore_auth`, `acore_world`, and `acore_characters`.
- If **any tables exist**, the script logs `Backup restoration completed successfully` and skips the expensive restore just as before.
- If **no tables are found or the query fails**, the script logs `Restoration marker found, but databases are empty - forcing re-import`, automatically clears the stale marker, and reruns the backup restore + `dbimport` pipeline so services always start with real data.
Manual intervention is only required if you intentionally want to force a fresh import despite having data. In that scenario:
1. Stop the stack: `docker compose down`
2. Delete the sentinel: `rm -f local-storage/mysql-data/.restore-completed`
3. Run `docker compose run --rm ac-db-import`
See [docs/ADVANCED.md#database-hardening](ADVANCED.md#database-hardening) for more background on the tmpfs/persistent split and why the sentinel exists, and review [docs/TROUBLESHOOTING.md](TROUBLESHOOTING.md#database-connection-issues) for quick steps when the automation logs the warning above.
### Manual Restore
**Restore from backup directory:**
```bash
./scripts/bash/backup-import.sh \
--backup-dir ./storage/backups/ExportBackup_20250114_120000 \
--password YOUR_MYSQL_PASSWORD \
--auth-db acore_auth \
--characters-db acore_characters \
--world-db acore_world \
--all
```
**Selective restore (only specific databases):**
```bash
./scripts/bash/backup-import.sh \
--backup-dir ./path/to/backup \
--password YOUR_PASSWORD \
--db characters \
--characters-db acore_characters
```
**Skip specific databases:**
```bash
./scripts/bash/backup-import.sh \
--backup-dir ./path/to/backup \
--password YOUR_PASSWORD \
--all \
--skip world
```
### Merge Backups (Advanced)
Merge accounts/characters from another server:
```bash
./scripts/bash/backup-merge.sh \
--backup-dir ../old-server/backup \
--password YOUR_PASSWORD \
--all-accounts \
--all-characters \
--exclude-bots
```
This intelligently:
- Remaps GUIDs to avoid conflicts
- Preserves existing data
- Imports character progression (spells, talents, etc.)
- Handles item instances
Options:
- `--all-accounts` - Import all accounts
- `--all-characters` - Import all characters
- `--exclude-bots` - Skip playerbot characters
- `--account "name1,name2"` - Import specific accounts
- `--dry-run` - Show what would be imported
---
## Health Monitoring
### Database Health Check
Check overall database health:
```bash
./scripts/bash/db-health-check.sh
```
Output includes:
- ✅ Database status (exists, responsive)
- 📊 Update counts (released, module, custom)
- 🕐 Last update timestamp
- 💾 Database sizes
- 📦 Module update summary
- 👥 Account/character counts
**Options:**
- `-v, --verbose` - Show detailed information
- `-p, --pending` - Show pending updates
- `-m, --no-modules` - Hide module updates
- `-c, --container NAME` - Specify MySQL container
**Example output:**
```
🗄️ AZEROTHCORE DATABASE HEALTH CHECK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🗄️ Database Status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Auth DB (acore_auth)
🔄 Updates: 45 applied
🕐 Last update: 2025-01-14 14:30:22
💾 Size: 12.3 MB (23 tables)
✅ World DB (acore_world)
🔄 Updates: 1,234 applied (15 module)
🕐 Last update: 2025-01-14 14:32:15
💾 Size: 2.1 GB (345 tables)
✅ Characters DB (acore_characters)
🔄 Updates: 89 applied
🕐 Last update: 2025-01-14 14:31:05
💾 Size: 180.5 MB (67 tables)
📊 Server Statistics
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Accounts: 25
Characters: 145
Active (24h): 8
💾 Total Database Storage: 2.29 GB
```
### Backup Status
Check backup system status:
```bash
./scripts/bash/backup-status.sh
```
Shows:
- Backup tier summary (hourly, daily, manual)
- Latest backup timestamps
- Storage usage
- Next scheduled backups
**Options:**
- `-d, --details` - Show all available backups
- `-t, --trends` - Show size trends over time
### Query Applied Updates
Check which updates have been applied:
```sql
-- Show all updates for world database
USE acore_world;
SELECT name, state, timestamp FROM updates ORDER BY timestamp DESC LIMIT 20;
-- Show only module updates
SELECT name, state, timestamp FROM updates WHERE state='MODULE' ORDER BY timestamp DESC;
-- Count updates by state
SELECT state, COUNT(*) as count FROM updates GROUP BY state;
```
---
## Module SQL Management
### How Module SQL Works
When you enable a module that includes SQL changes:
1. **Module Installation:** Module is cloned to `modules/<module-name>/`
2. **SQL Detection:** SQL files are found in `data/sql/{base,updates,custom}/`
3. **SQL Staging:** SQL is copied to AzerothCore's update directories
4. **Auto-Application:** On next server startup, SQL is auto-applied
5. **Tracking:** Updates are tracked in `updates` table with `state='MODULE'`
### Module SQL Structure
Modules follow this structure:
```
modules/mod-example/
└── data/
└── sql/
├── base/ # Initial schema (runs once)
│ ├── db_auth/
│ ├── db_world/
│ └── db_characters/
├── updates/ # Incremental updates
│ ├── db_auth/
│ ├── db_world/
│ └── db_characters/
└── custom/ # Optional custom SQL
└── db_world/
```
### Verifying Module SQL
Check if module SQL was applied:
```bash
# Run health check with module details
./scripts/bash/db-health-check.sh --verbose
# Or query directly
mysql -e "SELECT * FROM acore_world.updates WHERE name LIKE '%mod-example%'"
```
### Manual SQL Execution
If you need to run SQL manually:
```bash
# Connect to database
docker exec -it ac-mysql mysql -uroot -p
# Select database
USE acore_world;
# Run your SQL
SOURCE /path/to/your/file.sql;
# Or pipe from host
docker exec -i ac-mysql mysql -uroot -pPASSWORD acore_world < yourfile.sql
```
### Module SQL Ledger & Deduplication
`./scripts/bash/stage-modules.sh` now keeps a lightweight ledger at `storage/modules/.modules-meta/module-sql-ledger.txt` (also mounted inside containers at `/azerothcore/modules/.modules-meta/module-sql-ledger.txt`). Each staged SQL file is recorded as:
```
<database-scope>|<module>|<base_filename>|<hash>
```
When the script runs again it hashes every module SQL file and skips any entry whose `(db, module, filename)` already matches with the same hash. This prevents re-copying identical SQL after a backup restore and stops worldserver from reapplying inserts that already exist in the database. If a database restore is detected (`local-storage/mysql-data/.restore-completed` changed), the ledger is automatically reset so every module SQL file is recopied exactly once. The ledger is automatically updated anytime a file changes so only the modified SQL is restaged.
The stage script also cross-checks MySQLs `updates` table before copying files and prunes any staged file whose identifier already exists there. That means even if a file gets stuck in `/azerothcore/data/sql/updates/<db>` (e.g., after an interrupted run), it is removed before worldserver starts if the database already recorded it.
### Restore-Time SQL Reconciliation
During a backup restore the `ac-db-import` service now runs `scripts/bash/restore-and-stage.sh`, which consolidates the old restore workflow with module SQL staging. Every backup created by the scheduler now includes a snapshot of the module ledger at `module-sql-ledger.txt` (for example `storage/backups/hourly/20250101_120000/module-sql-ledger.txt`). The restore script:
- Refreshes `storage/modules/.modules-meta/module-sql-ledger.txt` using the snapshot bundled with the backup (or rebuilds it from the modules directory if the snapshot is missing).
- Writes `storage/modules/.modules-meta/.restore-prestaged` to signal that the next `./scripts/bash/stage-modules.sh` run must repopulate `/azerothcore/data/sql/updates/*` before worldserver comes online.
The staging script now recopies every module SQL file—regardless of whether it has already been applied—using deterministic names like `MODULE_mod-npc-buffer_npc_buffer.sql`. AzerothCores built-in updater consults the `updates` tables to decide what should actually run, so already-applied files remain on disk purely to keep history intact and avoid “file missing” warnings. If a legacy backup doesnt contain the ledger snapshot the helper simply rebuilds it and still sets the flag, so the runtime staging pass behaves the same. Run `rm -f storage/modules/.modules-meta/module-sql-ledger.txt` and rerun `./scripts/bash/stage-modules.sh --yes` if you intentionally need to reseed the ledger from scratch.
This snapshot-driven workflow means restoring a new backup automatically replays any newly added module SQL while avoiding duplicate inserts for modules that were already present. See **[docs/ADVANCED.md](ADVANCED.md)** for a deeper look at the marker workflow and container responsibilities.
### Forcing a Module SQL Re-stage
If you intentionally need to reapply all module SQL (for example after manually cleaning tables):
1. Stop services: `docker compose down`
2. Remove the SQL ledger so the next run rehashes everything:
```bash
rm -f storage/modules/.modules-meta/module-sql-ledger.txt
```
3. (Optional) Drop the relevant records from the `updates` table if you want AzerothCore to rerun them, e.g.:
```bash
docker exec -it ac-mysql mysql -uroot -p \
-e "DELETE FROM acore_characters.updates WHERE name LIKE '%MODULE_mod-ollama-chat%';"
```
4. Run `./scripts/bash/stage-modules.sh --yes`
Only perform step 3 if you understand the impact—deleting entries causes worldserver to execute those SQL scripts again on next startup.
---
## Migration & Upgrades
### Upgrading from Older Backups
When restoring an older backup to a newer AzerothCore version:
1. **Restore the backup** as normal
2. **Verification happens automatically** - The system runs `dbimport` after restore
3. **Missing updates are applied** - Any new schema changes are detected and applied
4. **Check for errors** in worldserver logs
### Manual Migration Steps
If automatic migration fails:
```bash
# 1. Backup current state
./scripts/bash/manual-backup.sh --label pre-migration
# 2. Run dbimport manually
docker exec -it ac-worldserver /bin/bash
cd /azerothcore/env/dist/bin
./dbimport
# 3. Check for errors
tail -f /azerothcore/env/dist/logs/DBErrors.log
# 4. Verify with health check
./scripts/bash/db-health-check.sh --verbose --pending
```
### Schema Version Checking
Check your database version:
```sql
-- World database version
SELECT * FROM acore_world.version;
-- Check latest update
SELECT name, timestamp FROM acore_world.updates ORDER BY timestamp DESC LIMIT 1;
```
---
## Troubleshooting
### Database Won't Start
**Symptom:** MySQL container keeps restarting
**Solutions:**
1. Check logs:
```bash
docker logs ac-mysql
```
2. Check disk space:
```bash
df -h
```
3. Reset MySQL data (WARNING: deletes all data):
```bash
docker-compose down
rm -rf storage/mysql/*
docker-compose up -d
```
### Updates Not Applying
**Symptom:** SQL updates in `pending_db_*` not getting applied
**Solutions:**
1. Check `Updates.EnableDatabases` setting:
```bash
grep "Updates.EnableDatabases" storage/config/worldserver.conf
# Should be 7 (auth+char+world) or 15 (all including playerbots)
```
2. Check for SQL errors:
```bash
docker logs ac-worldserver | grep -i "sql error"
```
3. Manually run dbimport:
```bash
docker exec -it ac-worldserver /bin/bash
cd /azerothcore/env/dist/bin
./dbimport
```
### Backup Restore Fails
**Symptom:** Backup import reports errors
**Solutions:**
1. Verify backup integrity:
```bash
./scripts/bash/verify-backup-complete.sh /path/to/backup
```
2. Check SQL file format:
```bash
zcat backup.sql.gz | head -20
# Should see SQL statements like CREATE DATABASE, INSERT INTO
```
3. Check database names in manifest:
```bash
cat backup/manifest.json
# Verify database names match your .env
```
4. Try importing individual databases:
```bash
# Extract and import manually
zcat backup/acore_world.sql.gz | docker exec -i ac-mysql mysql -uroot -pPASSWORD acore_world
```
### Missing Characters After Restore
**Symptom:** Characters don't appear in-game
**Common Causes:**
1. **Wrong database restored** - Check you restored characters DB
2. **GUID mismatch** - Items reference wrong GUIDs
3. **Incomplete restore** - Check for SQL errors during restore
**Fix with backup-merge:**
```bash
# Use merge instead of import to remap GUIDs
./scripts/bash/backup-merge.sh \
--backup-dir ./path/to/backup \
--password PASSWORD \
--all-characters
```
### Duplicate SQL Execution
**Symptom:** "Duplicate key" errors in logs
**Cause:** SQL update ran twice
**Prevention:** The `updates` table prevents this, but if table is missing:
```sql
-- Recreate updates table
CREATE TABLE IF NOT EXISTS `updates` (
`name` varchar(200) NOT NULL,
`hash` char(40) DEFAULT '',
`state` enum('RELEASED','CUSTOM','MODULE','ARCHIVED','PENDING') NOT NULL DEFAULT 'RELEASED',
`timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`speed` int unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```
### Performance Issues
**Symptom:** Database queries are slow
**Solutions:**
1. Check database size:
```bash
./scripts/bash/db-health-check.sh
```
2. Optimize tables:
```sql
USE acore_world;
OPTIMIZE TABLE creature;
OPTIMIZE TABLE gameobject;
USE acore_characters;
OPTIMIZE TABLE characters;
OPTIMIZE TABLE item_instance;
```
3. Check MySQL configuration:
```bash
docker exec ac-mysql mysql -uroot -pPASSWORD -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size'"
```
4. Increase buffer pool (edit docker-compose.yml):
```yaml
environment:
MYSQL_INNODB_BUFFER_POOL_SIZE: 512M # Increase from 256M
```
---
## Best Practices
### Backup Strategy
✅ **DO:**
- Keep at least 3 days of daily backups
- Test restore procedures regularly
- Store backups in multiple locations
- Monitor backup size trends
- Verify backup completion
❌ **DON'T:**
- Rely solely on automated backups
- Store backups only on same disk as database
- Skip verification of backup integrity
- Ignore backup size growth warnings
### Update Management
✅ **DO:**
- Let AzerothCore's auto-updater handle SQL
- Review `DBErrors.log` after updates
- Keep `Updates.EnableDatabases` enabled
- Test module updates in development first
❌ **DON'T:**
- Manually modify core database tables
- Skip module SQL when installing modules
- Disable auto-updates in production
- Run untested SQL in production
### Module Installation
✅ **DO:**
- Enable modules via `.env` file
- Verify module SQL applied via health check
- Check module compatibility before enabling
- Test modules individually first
❌ **DON'T:**
- Copy SQL files manually
- Edit module source SQL
- Enable incompatible module combinations
- Skip SQL verification after module install
### Performance
✅ **DO:**
- Run `OPTIMIZE TABLE` on large tables monthly
- Monitor database size growth
- Set appropriate MySQL buffer pool size
- Use SSD storage for MySQL data
❌ **DON'T:**
- Store MySQL data on slow HDDs
- Run database on same disk as backup
- Ignore slow query logs
- Leave unused data unarchived
---
## Quick Reference
### Essential Commands
```bash
# Check database health
./scripts/bash/db-health-check.sh
# Check backup status
./scripts/bash/backup-status.sh
# Create manual backup
./scripts/bash/manual-backup.sh --label my-backup
# Restore from backup
./scripts/bash/backup-import.sh --backup-dir ./path/to/backup --password PASS --all
# Export portable backup
./scripts/bash/backup-export.sh --password PASS --all -o ./export
# Connect to MySQL
docker exec -it ac-mysql mysql -uroot -p
# View worldserver logs
docker logs ac-worldserver -f
# Restart services
docker-compose restart ac-worldserver ac-authserver
```
### Important File Locations
```
storage/
├── mysql/ # MySQL data directory
├── backups/
│ ├── hourly/ # Automated hourly backups
│ └── daily/ # Automated daily backups
├── config/ # Server configuration files
└── logs/ # Server log files
manual-backups/ # Manual backup storage
local-storage/
└── modules/ # Installed module files
```
### Support Resources
- **Health Check:** `./scripts/bash/db-health-check.sh --help`
- **Backup Status:** `./scripts/bash/backup-status.sh --help`
- **AzerothCore Wiki:** https://www.azerothcore.org/wiki
- **AzerothCore Discord:** https://discord.gg/gkt4y2x
- **Issue Tracker:** https://github.com/uprightbass360/AzerothCore-RealmMaster/issues
---
**End of Database Management Guide**