loyal_companion/docs/database.md

# Database Schema Guide

This document describes the database schema used by Daemon Boyfriend.

## Table of Contents

- [Overview](#overview)
- [Core Tables](#core-tables)
- [Living AI Tables](#living-ai-tables)
- [Indexes](#indexes)
- [Relationships](#relationships)
- [Schema Diagram](#schema-diagram)

---

## Overview

### Database Support

- **Primary:** PostgreSQL with async support (`asyncpg`)
- **Fallback:** In-memory (no persistence)
- **Testing:** SQLite (via `aiosqlite`)

### Key Features

- **PortableJSON:** Custom type for PostgreSQL JSONB / SQLite JSON compatibility
- **UTC Timestamps:** All timestamps are stored as timezone-aware UTC
- **Soft Deletes:** Many tables use `is_active` flag instead of hard deletes
- **Cascade Deletes:** Foreign keys cascade on user/conversation deletion

### Connection Configuration

```bash
# PostgreSQL connection string
DATABASE_URL=postgresql+asyncpg://user:password@localhost:5432/loyal_companion

# Optional settings
DATABASE_ECHO=false        # Log SQL queries
DATABASE_POOL_SIZE=5       # Connection pool size
DATABASE_MAX_OVERFLOW=10   # Max connections beyond pool
```

---

## Core Tables

### users

Stores Discord user information.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `discord_id` | BIGINT | Discord user ID (unique) |
| `discord_username` | VARCHAR(255) | Discord username |
| `discord_display_name` | VARCHAR(255) | Discord display name |
| `custom_name` | VARCHAR(255) | Custom name set by user/admin |
| `first_seen_at` | TIMESTAMPTZ | When user was first seen |
| `last_seen_at` | TIMESTAMPTZ | When user was last seen |
| `is_active` | BOOLEAN | Whether user is active |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

**Usage:**
```python
user = await user_service.get_or_create_user(
    discord_id=123456789,
    username="john_doe",
    display_name="John"
)
print(user.display_name)  # Returns custom_name or discord_display_name
```

---

### user_preferences

Key-value storage for user preferences.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `user_id` | BIGINT | Foreign key to users |
| `preference_key` | VARCHAR(100) | Preference name |
| `preference_value` | TEXT | Preference value |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

**Unique Constraint:** `(user_id, preference_key)`

---

### user_facts

Facts the bot knows about users.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `user_id` | BIGINT | Foreign key to users |
| `fact_type` | VARCHAR(50) | Type (hobby, work, family, etc.) |
| `fact_content` | TEXT | The fact itself |
| `confidence` | FLOAT | 0.0-1.0 confidence level |
| `source` | VARCHAR(50) | How learned (conversation, auto_extraction, manual) |
| `is_active` | BOOLEAN | Whether fact is active |
| `learned_at` | TIMESTAMPTZ | When fact was learned |
| `last_referenced_at` | TIMESTAMPTZ | When fact was last used |
| `category` | VARCHAR(50) | Category (same as fact_type) |
| `importance` | FLOAT | 0.0-1.0 importance level |
| `temporal_relevance` | VARCHAR(20) | past/present/future/timeless |
| `expiry_date` | TIMESTAMPTZ | When fact expires (optional) |
| `extracted_from_message_id` | BIGINT | Source Discord message ID |
| `extraction_context` | TEXT | Context of extraction |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

**Fact Types:**
- `hobby` - Activities, interests
- `work` - Job, career
- `family` - Family members
- `preference` - Likes, dislikes
- `location` - Places
- `event` - Life events
- `relationship` - Personal relationships
- `general` - Other facts

---

### guilds

Discord server (guild) information.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `discord_id` | BIGINT | Discord guild ID (unique) |
| `name` | VARCHAR(255) | Guild name |
| `joined_at` | TIMESTAMPTZ | When bot joined |
| `is_active` | BOOLEAN | Whether bot is active in guild |
| `settings` | JSONB | Guild-specific settings |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

---

### guild_members

User membership in guilds.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `guild_id` | BIGINT | Foreign key to guilds |
| `user_id` | BIGINT | Foreign key to users |
| `guild_nickname` | VARCHAR(255) | Nickname in this guild |
| `roles` | TEXT[] | Array of role names |
| `joined_guild_at` | TIMESTAMPTZ | When user joined guild |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

**Unique Constraint:** `(guild_id, user_id)`

---

### conversations

Conversation sessions between users and the bot.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `user_id` | BIGINT | Foreign key to users |
| `guild_id` | BIGINT | Guild ID (nullable for DMs) |
| `channel_id` | BIGINT | Discord channel ID |
| `started_at` | TIMESTAMPTZ | When conversation started |
| `last_message_at` | TIMESTAMPTZ | Last message timestamp |
| `message_count` | INTEGER | Number of messages |
| `is_active` | BOOLEAN | Whether conversation is active |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

**Conversation Timeout:**
A new conversation starts after 60 minutes of inactivity.

---

### messages

Individual messages in conversations.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `conversation_id` | BIGINT | Foreign key to conversations |
| `user_id` | BIGINT | Foreign key to users |
| `discord_message_id` | BIGINT | Discord message ID |
| `role` | VARCHAR(20) | user/assistant/system |
| `content` | TEXT | Message content |
| `has_images` | BOOLEAN | Whether message has images |
| `image_urls` | JSONB | Array of image URLs |
| `token_count` | INTEGER | Token count (estimated) |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

---

## Living AI Tables

### bot_states

Per-guild bot state (mood, statistics).

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `guild_id` | BIGINT | Guild ID (unique, NULL = global) |
| `mood_valence` | FLOAT | -1.0 to 1.0 (sad to happy) |
| `mood_arousal` | FLOAT | -1.0 to 1.0 (calm to excited) |
| `mood_updated_at` | TIMESTAMPTZ | When mood was last updated |
| `total_messages_sent` | INTEGER | Lifetime message count |
| `total_facts_learned` | INTEGER | Facts extracted count |
| `total_users_known` | INTEGER | Unique users count |
| `first_activated_at` | TIMESTAMPTZ | Bot "birth date" |
| `preferences` | JSONB | Bot preferences |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

---

### bot_opinions

Bot's opinions on topics.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `guild_id` | BIGINT | Guild ID (NULL = global) |
| `topic` | VARCHAR(255) | Topic name (lowercase) |
| `sentiment` | FLOAT | -1.0 to 1.0 |
| `interest_level` | FLOAT | 0.0 to 1.0 |
| `discussion_count` | INTEGER | Times discussed |
| `reasoning` | TEXT | AI explanation (optional) |
| `formed_at` | TIMESTAMPTZ | When opinion formed |
| `last_reinforced_at` | TIMESTAMPTZ | When last discussed |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

**Unique Constraint:** `(guild_id, topic)`

---

### user_relationships

Relationship depth with each user.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `user_id` | BIGINT | Foreign key to users |
| `guild_id` | BIGINT | Guild ID (NULL = global) |
| `relationship_score` | FLOAT | 0-100 score |
| `total_interactions` | INTEGER | Total interaction count |
| `positive_interactions` | INTEGER | Positive interaction count |
| `negative_interactions` | INTEGER | Negative interaction count |
| `avg_message_length` | FLOAT | Average message length |
| `conversation_depth_avg` | FLOAT | Average conversation turns |
| `shared_references` | JSONB | Inside jokes, nicknames, etc. |
| `first_interaction_at` | TIMESTAMPTZ | First interaction |
| `last_interaction_at` | TIMESTAMPTZ | Last interaction |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

**Unique Constraint:** `(user_id, guild_id)`

**Relationship Score Levels:**
- 0-20: Stranger
- 21-40: Acquaintance
- 41-60: Friend
- 61-80: Good Friend
- 81-100: Close Friend

---

### user_communication_styles

Learned communication preferences per user.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `user_id` | BIGINT | Foreign key to users (unique) |
| `preferred_length` | VARCHAR(20) | short/medium/long |
| `preferred_formality` | FLOAT | 0.0 (casual) to 1.0 (formal) |
| `emoji_affinity` | FLOAT | 0.0 (none) to 1.0 (lots) |
| `humor_affinity` | FLOAT | 0.0 (serious) to 1.0 (playful) |
| `detail_preference` | FLOAT | 0.0 (concise) to 1.0 (detailed) |
| `engagement_signals` | JSONB | Engagement signal data |
| `samples_collected` | INTEGER | Number of samples analyzed |
| `confidence` | FLOAT | 0.0-1.0 confidence level |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

---

### scheduled_events

Proactive events (birthdays, follow-ups).

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `user_id` | BIGINT | Foreign key to users |
| `guild_id` | BIGINT | Target guild ID |
| `channel_id` | BIGINT | Target channel ID |
| `event_type` | VARCHAR(50) | birthday/follow_up/reminder |
| `trigger_at` | TIMESTAMPTZ | When to trigger |
| `title` | VARCHAR(255) | Event title |
| `context` | JSONB | Additional context |
| `is_recurring` | BOOLEAN | Whether event recurs |
| `recurrence_rule` | VARCHAR(100) | yearly/monthly/weekly/etc. |
| `status` | VARCHAR(20) | pending/triggered/cancelled |
| `triggered_at` | TIMESTAMPTZ | When event was triggered |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

---

### fact_associations

Cross-user fact associations.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `fact_id_1` | BIGINT | Foreign key to user_facts |
| `fact_id_2` | BIGINT | Foreign key to user_facts |
| `association_type` | VARCHAR(50) | shared_interest/same_location/etc. |
| `strength` | FLOAT | 0.0-1.0 association strength |
| `discovered_at` | TIMESTAMPTZ | When discovered |
| `created_at` | TIMESTAMPTZ | Record creation time |

**Unique Constraint:** `(fact_id_1, fact_id_2)`

---

### mood_history

Historical mood changes.

| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `guild_id` | BIGINT | Guild ID |
| `valence` | FLOAT | Valence at this point |
| `arousal` | FLOAT | Arousal at this point |
| `trigger_type` | VARCHAR(50) | conversation/time_decay/event |
| `trigger_user_id` | BIGINT | User who triggered (optional) |
| `trigger_description` | TEXT | Description of trigger |
| `recorded_at` | TIMESTAMPTZ | When recorded |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Record update time |

---

## Indexes

All indexes are created with `IF NOT EXISTS` for idempotency.

### Core Table Indexes

| Table | Index | Columns |
|-------|-------|---------|
| users | ix_users_discord_id | discord_id |
| users | ix_users_last_seen_at | last_seen_at |
| user_preferences | ix_user_preferences_user_id | user_id |
| user_facts | ix_user_facts_user_id | user_id |
| user_facts | ix_user_facts_fact_type | fact_type |
| user_facts | ix_user_facts_is_active | is_active |
| guilds | ix_guilds_discord_id | discord_id |
| guild_members | ix_guild_members_guild_id | guild_id |
| guild_members | ix_guild_members_user_id | user_id |
| conversations | ix_conversations_user_id | user_id |
| conversations | ix_conversations_channel_id | channel_id |
| conversations | ix_conversations_last_message_at | last_message_at |
| conversations | ix_conversations_is_active | is_active |
| messages | ix_messages_conversation_id | conversation_id |
| messages | ix_messages_user_id | user_id |
| messages | ix_messages_created_at | created_at |

### Living AI Table Indexes

| Table | Index | Columns |
|-------|-------|---------|
| bot_states | ix_bot_states_guild_id | guild_id |
| bot_opinions | ix_bot_opinions_guild_id | guild_id |
| bot_opinions | ix_bot_opinions_topic | topic |
| user_relationships | ix_user_relationships_user_id | user_id |
| user_relationships | ix_user_relationships_guild_id | guild_id |
| user_communication_styles | ix_user_communication_styles_user_id | user_id |
| scheduled_events | ix_scheduled_events_user_id | user_id |
| scheduled_events | ix_scheduled_events_trigger_at | trigger_at |
| scheduled_events | ix_scheduled_events_status | status |
| fact_associations | ix_fact_associations_fact_id_1 | fact_id_1 |
| fact_associations | ix_fact_associations_fact_id_2 | fact_id_2 |
| mood_history | ix_mood_history_guild_id | guild_id |
| mood_history | ix_mood_history_recorded_at | recorded_at |

---

## Relationships

### Entity Relationship Diagram

```
┌──────────────────┐         ┌──────────────────┐
│      users       │         │      guilds      │
├──────────────────┤         ├──────────────────┤
│ id (PK)          │◄────┐   │ id (PK)          │◄────┐
│ discord_id       │     │   │ discord_id       │     │
│ discord_username │     │   │ name             │     │
│ custom_name      │     │   │ settings         │     │
└──────────────────┘     │   └──────────────────┘     │
         │               │            │               │
         │               │            │               │
         ▼               │            ▼               │
┌──────────────────┐     │   ┌──────────────────┐     │
│  user_facts      │     │   │  guild_members   │     │
├──────────────────┤     │   ├──────────────────┤     │
│ id (PK)          │     │   │ id (PK)          │     │
│ user_id (FK) ────┼─────┘   │ guild_id (FK) ───┼─────┘
│ fact_type        │         │ user_id (FK) ────┼─────┐
│ fact_content     │         │ guild_nickname   │     │
└──────────────────┘         └──────────────────┘     │
         ▲                                            │
         │                                            │
         │                                            │
┌────────┴─────────┐         ┌──────────────────┐     │
│fact_associations │         │  conversations   │     │
├──────────────────┤         ├──────────────────┤     │
│ id (PK)          │         │ id (PK)          │◄────┤
│ fact_id_1 (FK)   │         │ user_id (FK) ────┼─────┘
│ fact_id_2 (FK)   │         │ guild_id         │
│ association_type │         │ channel_id       │
└──────────────────┘         └──────────────────┘
                                     │
                                     │
                                     ▼
                             ┌──────────────────┐
                             │    messages      │
                             ├──────────────────┤
                             │ id (PK)          │
                             │ conversation_id  │
                             │ user_id (FK)     │
                             │ role             │
                             │ content          │
                             └──────────────────┘
```

### Living AI Relationships

```
┌──────────────────┐
│      users       │
├──────────────────┤
│ id (PK)          │◄─────────────────────────────────────┐
└──────────────────┘                                      │
         │                                                │
         │                                                │
    ┌────┴────────┬───────────────┬───────────────┐      │
    │             │               │               │      │
    ▼             ▼               ▼               ▼      │
┌────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐  │
│user_   │ │user_comm   │ │scheduled   │ │mood_       │  │
│relation│ │styles      │ │events      │ │history     │  │
│ships   │ │            │ │            │ │            │  │
├────────┤ ├────────────┤ ├────────────┤ ├────────────┤  │
│user_id │ │user_id     │ │user_id     │ │trigger_    │  │
│guild_id│ │preferred_  │ │guild_id    │ │user_id ────┼──┘
│score   │ │length      │ │event_type  │ │valence     │
│        │ │emoji       │ │trigger_at  │ │arousal     │
└────────┘ └────────────┘ └────────────┘ └────────────┘


┌──────────────────┐         ┌──────────────────┐
│   bot_states     │         │   bot_opinions   │
├──────────────────┤         ├──────────────────┤
│ guild_id (unique)│         │ guild_id         │
│ mood_valence     │         │ topic            │
│ mood_arousal     │         │ sentiment        │
│ total_messages   │         │ interest_level   │
│ total_facts      │         │ discussion_count │
└──────────────────┘         └──────────────────┘
```

---

## Schema Initialization

### From Code

```python
from loyal_companion.services.database import db

# Initialize connection
await db.init()

# Create tables from schema.sql
await db.create_tables()
```

### From SQL

```bash
# Create database
createdb loyal_companion

# Run schema
psql -U postgres -d loyal_companion -f schema.sql
```

### Docker

The `docker-compose.yml` handles database setup automatically:

```bash
docker-compose up -d
```

---

## Migrations

The project uses **Alembic** for migrations, but currently relies on `schema.sql` for initial setup.

```bash
# Generate migration
alembic revision --autogenerate -m "description"

# Run migrations
alembic upgrade head

# Rollback
alembic downgrade -1
```

---

## Type Compatibility

### PortableJSON

For JSONB (PostgreSQL) and JSON (SQLite) compatibility:

```python
from loyal_companion.models.base import PortableJSON

class MyModel(Base):
    settings = Column(PortableJSON, default={})
```

### ensure_utc()

Handles timezone-naive datetimes from SQLite:

```python
from loyal_companion.models.base import ensure_utc

# Safe for both PostgreSQL (already UTC) and SQLite (naive)
utc_time = ensure_utc(model.created_at)
```