# Secret Management API

A secure secret management system for Fraser that provides encryption at rest using envelope encryption (KEK + DEK) for storing sensitive data like LLM provider API keys, BYOK keys, OAuth tokens, and other credentials.

## Architecture Overview

The Secret API uses **envelope encryption** similar to AWS KMS, Google Cloud KMS, and HashiCorp Vault:

- **KEK (Key Encryption Key)**: Master key stored in external KMS (GCP Secret Manager or local file for dev)
- **DEK (Data Encryption Key)**: Randomly generated per secret, encrypted by KEK
- **Secrets**: User data encrypted with DEK

### Security Properties

✅ **KEK never touches disk** - Stored only in external KMS
✅ **DEK is random per secret** - Limits blast radius if one secret is compromised
✅ **AES-256-GCM** - Authenticated encryption for both KEK and DEK
✅ **Realm as AAD** - Prevents tampering and cross-realm attacks
✅ **KEK rotation supported** - Via `kek_version` field

## Visual Documentation

See the Excalidraw diagrams for visual architecture:

1. **[encryption-flow.excalidraw](encryption-flow.excalidraw)** - Shows envelope encryption flow
2. **[architecture-overview.excalidraw](architecture-overview.excalidraw)** - System architecture
3. **[user-workflows.excalidraw](user-workflows.excalidraw)** - Initialization and usage workflows

## API Reference

See [secret.proto](secret.proto) for the complete API specification.

### Endpoints

- `POST /v1/secrets` - Create or update a secret
- `GET /v1/secrets/{name}` - Get secret metadata (not plaintext)
- `GET /v1/secrets` - List secrets
- `DELETE /v1/secrets/{name}` - Delete a secret
- `POST /v1/secrets/{name}/rotate` - Rotate a secret
- `POST /v1/secrets/{name}/validate` - Validate secret can be decrypted

## Master Key Initialization

### One-Time Setup (System Administrator)

Before using the Secret API, initialize the master KEK:

```bash
# Step 1: Generate KEK
npm run secret:init-kek

# This generates a 256-bit AES key
```

```bash
# Step 2: Store KEK in external KMS

# For GCP Secret Manager (production):
gcloud secrets create master-key-v1 --data-file=kek.bin

# For local development:
# KEK is automatically saved to kms-store.json
```

```bash
# Step 3: Set environment variables
export SECRET_KEK=master-key-v1
export KMS_TYPE=gcp  # or 'local' for dev

# For GCP:
export GCP_PROJECT_ID=your-project
export GCP_CREDENTIALS_JSON=<base64-credentials>
```

### Environment Variables

| Variable | Description | Example |
|----------|-------------|---------|
| `SECRET_KEK` | Name of the KEK in KMS | `master-key-v1` |
| `KMS_TYPE` | KMS backend type | `gcp` or `local` |
| `GCP_PROJECT_ID` | GCP project (if using GCP) | `my-project-123` |
| `GCP_CREDENTIALS_JSON` | Base64-encoded service account JSON | `eyJ0eXBlIjoi...` |

## User Workflows

### 1. Storing LLM Provider API Keys

Users can securely store their API keys for OpenAI, Anthropic, or other LLM providers:

```bash
# Create a secret
curl -X POST https://api.example.com/v1/secrets \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "openai-key-prod",
    "realm": "llm-providers",
    "description": "Production OpenAI API key",
    "value": "sk-proj-..."
  }'
```

**What happens:**
1. System generates random 256-bit DEK
2. Encrypts secret value with DEK using AES-256-GCM
3. Fetches KEK from KMS
4. Encrypts DEK with KEK using AES-256-GCM (with realm as AAD)
5. Stores `encrypted_value`, `wrapped_dek`, and `kek_version` in PostgreSQL

**Response:**
```json
{
  "secret": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "user_id": "user_123",
    "name": "openai-key-prod",
    "realm": "llm-providers",
    "description": "Production OpenAI API key",
    "kek_version": "1",
    "status": "SECRET_STATUS_ACTIVE",
    "created_at": "2025-01-15T10:30:00Z",
    "updated_at": "2025-01-15T10:30:00Z"
  }
}
```

### 2. BYOK (Bring Your Own Key)

Customers can provide their own encryption keys for data encryption:

```bash
# Customer provides their encryption key
curl -X POST https://api.example.com/v1/secrets \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "customer-data-key",
    "realm": "byok",
    "description": "Customer-managed encryption key for compliance",
    "value": "<base64-encoded-256-bit-key>"
  }'
```

**Use case:**
- When encrypting customer data, the application retrieves this BYOK key
- Decrypts it using the Secret API's internal decryption
- Uses the plaintext key to encrypt customer data
- Customer maintains control over their encryption keys for compliance

### 3. Internal Secret Retrieval (Application Code)

Services retrieve secrets internally using the query functions:

```typescript
import { getSecret } from '@/db/queries/secret'
import { kekDecryptString } from '@/db/queries/kek'

// Fetch secret from database
const secret = await getSecret(userId, 'llm-providers', 'openai-key-prod')

if (!secret) {
  throw new Error('Secret not found')
}

// Decrypt the secret
const plaintextKey = await kekDecryptString({
  encryptedValue: secret.encryptedValue,
  wrappedDek: secret.wrappedDek,
  kekVersion: secret.kekVersion,
  kekKeyName: process.env.SECRET_KEK!,
  realm: secret.realm
})

// Use the plaintext key (in memory only, never persisted)
const response = await fetch('https://api.openai.com/v1/chat/completions', {
  headers: {
    'Authorization': `Bearer ${plaintextKey}`
  },
  // ...
})
```

## Realms

Secrets are organized into realms for access control and AAD in encryption:

| Realm | Purpose | Examples |
|-------|---------|----------|
| `llm-providers` | LLM API keys | OpenAI, Anthropic, Google keys |
| `byok` | Customer-managed keys | Data encryption keys |
| `mcp` | MCP server credentials | OAuth tokens, service accounts |
| `telemetry` | Monitoring/analytics | DataDog, Sentry keys |

The realm is used as **Additional Authenticated Data (AAD)** in AES-GCM encryption, which prevents:
- Cross-realm tampering (can't take a `mcp` secret and use it as `llm-providers`)
- Unauthorized modifications to the ciphertext

## Key Rotation

Rotate secrets regularly (recommended: every 90 days):

```bash
# Rotate a secret
curl -X POST https://api.example.com/v1/secrets/openai-key-prod/rotate \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "realm": "llm-providers",
    "new_value": "sk-proj-new-key-..."
  }'
```

**What happens:**
1. Creates new version with new DEK
2. Old secret marked as `SECRET_STATUS_ROTATED_PENDING`
3. Applications gracefully transition to new key
4. Delete old version after grace period

## Database Schema

The `secrets` table stores encrypted secrets:

```sql
CREATE TABLE secrets (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id TEXT NOT NULL,
  realm TEXT NOT NULL,
  name TEXT NOT NULL,
  description TEXT NOT NULL,
  encrypted_value TEXT NOT NULL,  -- base64 ciphertext
  wrapped_dek TEXT NOT NULL,      -- base64 encrypted DEK
  kek_version TEXT NOT NULL,      -- KEK version identifier
  status TEXT NOT NULL,           -- active | revoked | rotated_pending
  created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
  updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

CREATE INDEX idx_secrets_user_id ON secrets(user_id);
CREATE INDEX idx_secrets_realm ON secrets(user_id, realm);
CREATE INDEX idx_secrets_user_id_name ON secrets(user_id, name);
CREATE INDEX idx_secrets_user_id_status ON secrets(user_id, status);
```

## Security Best Practices

### For Administrators

1. **KEK Management**
   - Store KEK only in external KMS
   - Use GCP Secret Manager in production
   - Enable audit logging for KEK access
   - Rotate KEK annually

2. **Access Control**
   - Limit KMS access to service accounts only
   - Use least privilege IAM roles
   - Enable MFA for admin accounts

3. **Monitoring**
   - Log all secret access attempts
   - Alert on decryption failures
   - Track unusual access patterns
   - Monitor KEK access logs

### For Users

1. **Naming Conventions**
   - Use descriptive names: `openai-key-prod`, `anthropic-key-dev`
   - Include environment in name: `-prod`, `-staging`, `-dev`
   - Avoid special characters

2. **Key Rotation**
   - Rotate secrets every 90 days
   - Use `RotateSecret` API for smooth transitions
   - Delete old keys after grace period

3. **Realm Organization**
   - Use appropriate realm for each secret type
   - Don't mix secret types across realms
   - Document realm usage in your application

## Comparison with Other Systems

### vs. HashiCorp Vault

**Similarities:**
- Envelope encryption (KEK + DEK)
- Secrets scoped by path (realms)
- Key rotation support
- Audit logging

**Differences:**
- Fraser: Built into platform, PostgreSQL storage
- Vault: Standalone service, multiple storage backends

### vs. GCP Secret Manager

**Similarities:**
- External KMS for KEK
- Versioned secrets
- IAM-based access control

**Differences:**
- Fraser: Application-level secrets with user scoping
- GCP: Infrastructure-level secrets with project scoping

### vs. AWS Secrets Manager

**Similarities:**
- Automatic rotation support
- Encryption at rest
- Fine-grained access control

**Differences:**
- Fraser: Open-source, self-hosted
- AWS: Managed service, AWS-specific

## Implementation Status

- [x] Database schema (`secrets` table)
- [x] KEK/DEK encryption primitives
- [x] Basic CRUD operations
- [x] Protobuf API definition
- [ ] gRPC service implementation
- [ ] REST gateway
- [ ] Key rotation logic
- [ ] Audit logging
- [ ] Admin CLI tool
- [ ] User UI for secret management

## Migration from GCP Secret Manager

Fraser now uses a three-layer key hierarchy for improved security and management:

```
Master KEK (GCP/Vault) → Service KEKs (Fraser) → Secrets (Fraser)
```

**See [MIGRATION.md](MIGRATION.md) for complete migration guide.**

### Quick Start Migration

```bash
# 1. Initialize Master KEK
export MASTER_KEK=MASTER_KEK_V1
export MASTER_KMS_TYPE=gcp
npm run init-master-kek

# 2. Migrate LLM provider keys from GCP
npm run migrate-llm-keys

# 3. Validate migration
npm run validate-migration

# 4. Update services to use Secret API (see MIGRATION.md)
```

## Development

### Running Tests

```bash
# Test encryption/decryption
npm test src/db/queries/kek.test.ts

# Test secret CRUD operations
npm test src/db/queries/secret.test.ts
```

### Local Development Setup

```bash
# Initialize Master KEK (local file for dev)
export MASTER_KEK=dev-master-kek
export MASTER_KMS_TYPE=local
npm run init-master-kek

# Start development server
npm run dev

# Environment will use local file KMS (kms-store.json)
```

## References

- [NIST SP 800-38D: AES-GCM](https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38d.pdf)
- [Envelope Encryption](https://cloud.google.com/kms/docs/envelope-encryption)
- [Google Cloud Secret Manager](https://cloud.google.com/secret-manager/docs)
- [HashiCorp Vault](https://www.vaultproject.io/docs)

## Support

For issues or questions:
- File an issue in the Fraser repository
- Contact the security team for KEK management
- Review audit logs for access troubleshooting
