589 lines
15 KiB
Markdown
589 lines
15 KiB
Markdown
# du.pe - Knowledge Base & Development Guide
|
|
|
|
## 🎯 Quick Reference
|
|
|
|
### **What is du.pe?**
|
|
A self-hosted torrent-based file manager and media player that provides a clean web interface for managing torrents and streaming media content. Think of it as a personal Put.io alternative.
|
|
|
|
### **Core Features**
|
|
- 🧲 Torrent management (files & magnet links)
|
|
- 📥 Real-time download progress tracking
|
|
- 🎬 Instant video streaming
|
|
- 🗂️ Media library organization
|
|
- 🔍 Search functionality
|
|
- 🗑️ Trash management
|
|
- 📱 Mobile-friendly interface
|
|
|
|
### **Technology Stack**
|
|
- **Frontend**: Svelte + Vite
|
|
- **Backend**: Node.js + Express
|
|
- **Torrent Engine**: WebTorrent
|
|
- **Real-time**: WebSocket
|
|
- **Deployment**: Docker
|
|
|
|
---
|
|
|
|
## 🏗️ Architecture Overview
|
|
|
|
### **System Architecture Diagram**
|
|
```
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ Browser UI │ ←→ │ Express API │ ←→ │ WebTorrent │
|
|
│ (Svelte) │ │ (REST + WS) │ │ Engine │
|
|
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
|
↓ ↓ ↓
|
|
Real-time updates File operations Torrent management
|
|
(WebSocket) (API calls) (Download/Seed)
|
|
↓ ↓ ↓
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ State Stores │ │ File System │ │ External APIs │
|
|
│ (Media/Data) │ │ (Downloads) │ │ (TMDB/TVDB) │
|
|
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
|
```
|
|
|
|
### **Data Flow Patterns**
|
|
|
|
#### **Torrent Addition Flow**
|
|
1. User adds torrent (file/magnet) → Frontend
|
|
2. Frontend validates input → API endpoint
|
|
3. Server creates WebTorrent instance → Torrent engine
|
|
4. Download starts → WebSocket broadcasts progress
|
|
5. Files download → Organized in `/downloads/`
|
|
6. Metadata fetched → Cached in `/cache/`
|
|
7. UI updates → Real-time progress display
|
|
|
|
#### **Media Streaming Flow**
|
|
1. User clicks play → Frontend requests stream
|
|
2. API validates access → Checks file availability
|
|
3. Server streams file → HTTP Range requests supported
|
|
4. Browser plays → Native HTML5 video player
|
|
5. Optional transcoding → For codec compatibility
|
|
|
|
---
|
|
|
|
## 🔧 Component Deep Dive
|
|
|
|
### **Frontend Components**
|
|
|
|
#### **Core Application Structure**
|
|
```javascript
|
|
App.svelte (Root)
|
|
├── Router (svelte-routing)
|
|
├── Layout Components
|
|
│ ├── Sidebar (Navigation)
|
|
│ └── Topbar (Actions/Menu)
|
|
├── Route Components
|
|
│ ├── Files (File browser)
|
|
│ ├── Movies (Movie library)
|
|
│ ├── TvShows (TV series library)
|
|
│ ├── Transfers (Active torrents)
|
|
│ └── Trash (Deleted items)
|
|
└── Services
|
|
├── WebSocket client
|
|
├── API client
|
|
└── State management
|
|
```
|
|
|
|
#### **State Management (Svelte Stores)**
|
|
- **movieStore.js** - Movie library state and operations
|
|
- **tvStore.js** - TV show library state and operations
|
|
- **searchStore.js** - Search functionality and results
|
|
- **trashStore.js** - Trash management state
|
|
|
|
#### **Key Features by Component**
|
|
|
|
**Files.svelte** - File browser with:
|
|
- Directory navigation
|
|
- File type filtering
|
|
- Thumbnail previews
|
|
- Multi-select operations
|
|
- Drag & drop support
|
|
|
|
**Movies.svelte** - Movie library with:
|
|
- Grid/list view toggle
|
|
- Genre/year filtering
|
|
- Search functionality
|
|
- Metadata display
|
|
- Watch status tracking
|
|
|
|
**Transfers.svelte** - Torrent management with:
|
|
- Real-time progress bars
|
|
- Download/upload speed display
|
|
- Peer information
|
|
- Torrent control actions (pause, remove)
|
|
- File list within torrents
|
|
|
|
### **Backend Services**
|
|
|
|
#### **Server Architecture**
|
|
```javascript
|
|
server.js (Main server)
|
|
├── Express Application
|
|
│ ├── REST API endpoints
|
|
│ ├── Static file serving
|
|
│ ├── File upload handling
|
|
│ └── Error middleware
|
|
├── WebSocket Server
|
|
│ ├── Real-time progress events
|
|
│ ├── File update notifications
|
|
│ └── Connection management
|
|
├── WebTorrent Client
|
|
│ ├── Torrent management
|
|
│ ├── File downloading
|
|
│ └── Seeding functionality
|
|
└── File System Operations
|
|
├── Download organization
|
|
├── Thumbnail generation
|
|
├── Metadata caching
|
|
└── Trash management
|
|
```
|
|
|
|
#### **Key Server Features**
|
|
|
|
**Torrent Management**
|
|
- Add torrents via file upload or magnet links
|
|
- Real-time progress tracking via WebSocket
|
|
- File access within torrents
|
|
- Automatic file organization
|
|
|
|
**Media Processing**
|
|
- Automatic metadata fetching (TMDB/TVDB)
|
|
- Thumbnail generation for videos and images
|
|
- Library categorization (movies/TV shows)
|
|
- Search indexing
|
|
|
|
**File Operations**
|
|
- Streaming with HTTP Range support
|
|
- File browser with directory navigation
|
|
- Trash system with restore capability
|
|
- Disk space monitoring
|
|
|
|
---
|
|
|
|
## 🔄 Real-time Communication
|
|
|
|
### **WebSocket Events**
|
|
```javascript
|
|
// Client-side WebSocket handling
|
|
const ws = new WebSocket('ws://localhost:3001/?token=auth');
|
|
|
|
ws.onmessage = (event) => {
|
|
const message = JSON.parse(event.data);
|
|
|
|
switch(message.type) {
|
|
case 'progress':
|
|
// Update torrent progress UI
|
|
updateTorrentProgress(message.torrents);
|
|
break;
|
|
|
|
case 'fileUpdate':
|
|
// Refresh media libraries
|
|
refreshMediaLibrary();
|
|
break;
|
|
|
|
case 'error':
|
|
// Handle errors
|
|
showErrorMessage(message.message);
|
|
break;
|
|
}
|
|
};
|
|
```
|
|
|
|
### **Event Types**
|
|
- **progress** - Torrent download/upload progress updates
|
|
- **fileUpdate** - Media library changes (added/removed files)
|
|
- **torrentAdded** - New torrent added successfully
|
|
- **torrentRemoved** - Torrent removed or completed
|
|
- **error** - Various error notifications
|
|
|
|
---
|
|
|
|
## 📊 Data Models
|
|
|
|
### **Torrent Data Model**
|
|
```javascript
|
|
{
|
|
infoHash: "abc123def456...", // Unique identifier
|
|
name: "Example Movie 2024", // Torrent name
|
|
progress: 0.75, // Download progress (0-1)
|
|
downloadSpeed: 2097152, // Bytes per second
|
|
uploadSpeed: 1048576, // Bytes per second
|
|
numPeers: 25, // Connected peers
|
|
state: "downloading", // Current state
|
|
timeRemaining: 600, // Seconds remaining
|
|
files: [ // File array
|
|
{
|
|
name: "movie.mp4",
|
|
length: 1073741824,
|
|
path: "/downloads/example/",
|
|
type: "video"
|
|
}
|
|
],
|
|
magnetURI: "magnet:?xt=...", // Magnet link
|
|
added: "2024-01-15T10:30:00Z" // Addition timestamp
|
|
}
|
|
```
|
|
|
|
### **Movie Data Model**
|
|
```javascript
|
|
{
|
|
id: "movie_123",
|
|
title: "Example Movie",
|
|
year: 2024,
|
|
runtime: 120,
|
|
rating: 8.5,
|
|
genres: ["Action", "Drama"],
|
|
poster: "/cache/movie_data/123/poster.jpg",
|
|
backdrop: "/cache/movie_data/123/backdrop.jpg",
|
|
overview: "Movie description...",
|
|
filePath: "/downloads/example/movie.mp4",
|
|
fileSize: 1073741824,
|
|
added: "2024-01-15T10:30:00Z",
|
|
watched: false,
|
|
thumbnail: "/cache/thumbnails/videos/123/movie.jpg"
|
|
}
|
|
```
|
|
|
|
### **TV Show Data Model**
|
|
```javascript
|
|
{
|
|
id: "show_456",
|
|
title: "Example TV Series",
|
|
year: 2023,
|
|
rating: 9.0,
|
|
seasons: 2,
|
|
poster: "/cache/tv_data/456/poster.jpg",
|
|
backdrop: "/cache/tv_data/456/backdrop.jpg",
|
|
overview: "Series description...",
|
|
episodes: [
|
|
{
|
|
season: 1,
|
|
episode: 1,
|
|
title: "Pilot Episode",
|
|
overview: "Episode description...",
|
|
airDate: "2023-01-15",
|
|
filePath: "/downloads/example/s01e01.mp4",
|
|
thumbnail: "/cache/tv_data/456/episodes/s01e01.jpg"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 🛠️ Development Guide
|
|
|
|
### **Getting Started for Development**
|
|
|
|
#### **Prerequisites**
|
|
- Node.js 18+
|
|
- Docker & Docker Compose
|
|
- Git
|
|
|
|
#### **Local Development Setup**
|
|
```bash
|
|
# Clone the repository
|
|
git clone <repository-url>
|
|
cd dupe
|
|
|
|
# Start backend server
|
|
cd server
|
|
npm install
|
|
npm start
|
|
|
|
# Start frontend (in separate terminal)
|
|
cd client
|
|
npm install
|
|
npm run dev
|
|
|
|
# Access the application
|
|
# Frontend: http://localhost:5173
|
|
# Backend API: http://localhost:3001
|
|
```
|
|
|
|
#### **Docker Development**
|
|
```bash
|
|
# Build and run with Docker Compose
|
|
docker compose up -d --build
|
|
|
|
# View logs
|
|
docker compose logs -f
|
|
|
|
# Stop containers
|
|
docker compose down
|
|
```
|
|
|
|
### **Configuration**
|
|
|
|
#### **Environment Variables**
|
|
```bash
|
|
# .env file
|
|
USERNAME=admin # Basic auth username
|
|
PASSWORD=password # Basic auth password
|
|
TMDB_API_KEY=your_tmdb_key # The Movie Database API key
|
|
TVDB_API_KEY=your_tvdb_key # The TV Database API key
|
|
FANART_TV_API_KEY=your_fanart_key # Fanart.tv API key
|
|
VIDEO_THUMBNAIL_TIME=30 # Thumbnail timestamp (seconds)
|
|
PORT=3001 # Server port
|
|
```
|
|
|
|
#### **Frontend Configuration**
|
|
```javascript
|
|
// client/vite.config.js
|
|
export default {
|
|
server: {
|
|
host: '0.0.0.0',
|
|
port: 5173
|
|
},
|
|
define: {
|
|
'VITE_API': JSON.stringify('http://localhost:3001')
|
|
}
|
|
}
|
|
```
|
|
|
|
### **Code Structure Guidelines**
|
|
|
|
#### **Frontend (Svelte)**
|
|
- Use Svelte stores for state management
|
|
- Follow component-based architecture
|
|
- Implement proper error handling
|
|
- Use CSS variables for theming
|
|
- Maintain responsive design
|
|
|
|
#### **Backend (Node.js)**
|
|
- Use async/await for asynchronous operations
|
|
- Implement proper error middleware
|
|
- Follow RESTful API conventions
|
|
- Use meaningful HTTP status codes
|
|
- Implement proper logging
|
|
|
|
#### **File Organization**
|
|
```
|
|
client/src/
|
|
├── components/ # Reusable UI components
|
|
├── routes/ # Page components
|
|
├── stores/ # State management
|
|
├── utils/ # Utility functions
|
|
└── styles/ # CSS/styling
|
|
|
|
server/
|
|
├── utils/ # Server utilities
|
|
├── downloads/ # Downloaded files
|
|
├── cache/ # Cached data
|
|
└── trash/ # Deleted items
|
|
```
|
|
|
|
### **Testing Guidelines**
|
|
|
|
#### **Frontend Testing**
|
|
```bash
|
|
# Component testing (planned)
|
|
npm run test:unit
|
|
|
|
# E2E testing (planned)
|
|
npm run test:e2e
|
|
```
|
|
|
|
#### **Backend Testing**
|
|
```bash
|
|
# API endpoint testing (planned)
|
|
npm run test:api
|
|
|
|
# Integration testing (planned)
|
|
npm run test:integration
|
|
```
|
|
|
|
### **Common Development Tasks**
|
|
|
|
#### **Adding New API Endpoints**
|
|
1. Add route handler in `server/server.js`
|
|
2. Implement error handling
|
|
3. Add WebSocket events if needed
|
|
4. Update frontend API client
|
|
5. Add state management if required
|
|
|
|
#### **Adding New Frontend Pages**
|
|
1. Create route component in `client/src/routes/`
|
|
2. Add route in `App.svelte`
|
|
3. Update navigation in `Sidebar.svelte`
|
|
4. Add API calls as needed
|
|
5. Implement state management
|
|
|
|
#### **Adding New Features**
|
|
1. Plan API changes first
|
|
2. Implement backend logic
|
|
3. Add WebSocket events for real-time updates
|
|
4. Build frontend components
|
|
5. Add state management
|
|
6. Test thoroughly
|
|
|
|
---
|
|
|
|
## 🚀 Deployment Guide
|
|
|
|
### **Production Deployment**
|
|
|
|
#### **Docker Compose (Recommended)**
|
|
```bash
|
|
# Production environment
|
|
export USERNAME=admin
|
|
export PASSWORD=secure_password
|
|
export TMDB_API_KEY=your_key
|
|
|
|
# Deploy
|
|
docker compose -f docker-compose.prod.yml up -d
|
|
```
|
|
|
|
#### **Manual Deployment**
|
|
```bash
|
|
# Build frontend
|
|
cd client
|
|
npm run build
|
|
|
|
# Start server
|
|
cd ../server
|
|
npm start
|
|
```
|
|
|
|
#### **Reverse Proxy (Nginx)**
|
|
```nginx
|
|
server {
|
|
listen 80;
|
|
server_name your-domain.com;
|
|
|
|
location / {
|
|
proxy_pass http://localhost:3001;
|
|
proxy_http_version 1.1;
|
|
proxy_set_header Upgrade $http_upgrade;
|
|
proxy_set_header Connection 'upgrade';
|
|
proxy_set_header Host $host;
|
|
proxy_cache_bypass $http_upgrade;
|
|
}
|
|
}
|
|
```
|
|
|
|
### **Monitoring & Maintenance**
|
|
|
|
#### **Health Checks**
|
|
```bash
|
|
# Check server status
|
|
curl http://localhost:3001/api/system/info
|
|
|
|
# Check active torrents
|
|
curl http://localhost:3001/api/torrents
|
|
|
|
# Check disk usage
|
|
curl http://localhost:3001/api/system/disk
|
|
```
|
|
|
|
#### **Log Management**
|
|
```bash
|
|
# Docker logs
|
|
docker compose logs -f dupe
|
|
|
|
# Application logs
|
|
tail -f /var/log/dupe/app.log
|
|
```
|
|
|
|
#### **Backup Strategy**
|
|
```bash
|
|
# Backup downloads
|
|
tar -czf downloads-backup.tar.gz downloads/
|
|
|
|
# Backup application data
|
|
tar -czf app-backup.tar.gz cache/ trash/
|
|
```
|
|
|
|
---
|
|
|
|
## 🔍 Troubleshooting
|
|
|
|
### **Common Issues**
|
|
|
|
#### **Torrent Not Downloading**
|
|
- Check internet connection
|
|
- Verify torrent file/magnet link validity
|
|
- Check tracker status
|
|
- Review server logs for errors
|
|
|
|
#### **Video Not Streaming**
|
|
- Verify file exists and is accessible
|
|
- Check browser codec support
|
|
- Test with different browsers
|
|
- Review network connectivity
|
|
|
|
#### **WebSocket Connection Issues**
|
|
- Check authentication token
|
|
- Verify WebSocket server is running
|
|
- Review browser console for errors
|
|
- Check firewall/proxy settings
|
|
|
|
#### **Performance Issues**
|
|
- Monitor disk space usage
|
|
- Check network bandwidth
|
|
- Review system resources
|
|
- Consider cleanup operations
|
|
|
|
### **Debug Commands**
|
|
```bash
|
|
# Check Docker container status
|
|
docker ps
|
|
|
|
# Inspect container logs
|
|
docker logs dupe
|
|
|
|
# Test API endpoints
|
|
curl -v http://localhost:3001/api/torrents
|
|
|
|
# Monitor resource usage
|
|
docker stats dupe
|
|
```
|
|
|
|
---
|
|
|
|
## 📚 Cross-References
|
|
|
|
### **Documentation Links**
|
|
- [Project Structure](./project-structure.md) - Detailed architecture overview
|
|
- [API Documentation](./api-documentation.md) - Complete API reference
|
|
- [Component Guide](./components.md) - Frontend component documentation
|
|
- [Deployment Guide](./deployment.md) - Production deployment instructions
|
|
|
|
### **Key Implementation Files**
|
|
- [server/server.js](../server/server.js) - Main server implementation
|
|
- [client/src/App.svelte](../client/src/App.svelte) - Root application component
|
|
- [client/src/utils/api.js](../client/src/utils/api.js) - Frontend API client
|
|
- [docker-compose.yml](../docker-compose.yml) - Container configuration
|
|
|
|
### **External Dependencies**
|
|
- [Svelte Documentation](https://svelte.dev/docs) - Frontend framework
|
|
- [Express.js Documentation](https://expressjs.com/) - Backend framework
|
|
- [WebTorrent Documentation](https://webtorrent.io/docs) - Torrent engine
|
|
- [WebSocket API](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) - Real-time communication
|
|
|
|
---
|
|
|
|
## 🛣️ Development Roadmap
|
|
|
|
### **Planned Features**
|
|
- 🔐 Complete user authentication system
|
|
- 🎛️ Advanced video player controls
|
|
- 📱 Mobile app (React Native)
|
|
- 🔄 Automatic subtitle downloading
|
|
- 📊 Advanced analytics and reporting
|
|
- 🌐 Multi-language support
|
|
- ⚡ CDN integration for remote access
|
|
- 🔗 Share links for media content
|
|
|
|
### **Technical Improvements**
|
|
- 🧪 Comprehensive test suite
|
|
- 📈 Performance monitoring
|
|
- 🔒 Enhanced security measures
|
|
- 🗂️ Advanced file organization
|
|
- 🔄 Database integration
|
|
- 📊 Usage analytics
|
|
- 🛡️ Rate limiting and DDoS protection
|
|
|
|
---
|
|
|
|
*This knowledge base provides comprehensive documentation for understanding, developing, and deploying the du.pe application. For specific implementation details, refer to the cross-referenced files and external documentation.* |