663 lines
13 KiB
Markdown
663 lines
13 KiB
Markdown
# MetaScraper Deployment Guide
|
||
|
||
## 📦 Package Publishing
|
||
|
||
### Preparation Checklist
|
||
|
||
Before publishing, ensure:
|
||
|
||
- [ ] All tests pass: `npm test`
|
||
- [ ] Code is properly documented
|
||
- [ ] Version number follows semantic versioning
|
||
- [ ] CHANGELOG.md is updated
|
||
- [ ] Package.json is complete and accurate
|
||
- [ ] License file is present
|
||
- [ ] README.md is up to date
|
||
|
||
### Version Management
|
||
|
||
#### Semantic Versioning
|
||
|
||
```bash
|
||
# Patch version (bug fixes)
|
||
npm version patch
|
||
|
||
# Minor version (new features, backward compatible)
|
||
npm version minor
|
||
|
||
# Major version (breaking changes)
|
||
npm version major
|
||
```
|
||
|
||
#### Version Numbering Rules
|
||
|
||
- **MAJOR**: Breaking changes (API changes, Node.js version requirements)
|
||
- **MINOR**: New features (new Turkish patterns, performance improvements)
|
||
- **PATCH**: Bug fixes (error handling, small fixes)
|
||
|
||
### Package.json Configuration
|
||
|
||
```json
|
||
{
|
||
"name": "flixscaper",
|
||
"version": "1.0.0",
|
||
"description": "Netflix meta veri scraper.",
|
||
"type": "module",
|
||
"main": "src/index.js",
|
||
"exports": {
|
||
".": "./src/index.js",
|
||
"./parser": "./src/parser.js",
|
||
"./headless": "./src/headless.js"
|
||
},
|
||
"files": [
|
||
"src/",
|
||
"README.md",
|
||
"LICENSE",
|
||
"CHANGELOG.md"
|
||
],
|
||
"engines": {
|
||
"node": ">=18"
|
||
},
|
||
"keywords": [
|
||
"netflix",
|
||
"scraper",
|
||
"metadata",
|
||
"turkish",
|
||
"flixscaper"
|
||
],
|
||
"repository": {
|
||
"type": "git",
|
||
"url": "https://github.com/username/flixscaper.git"
|
||
}
|
||
}
|
||
```
|
||
|
||
### Publishing Process
|
||
|
||
#### 1. Local Testing
|
||
|
||
```bash
|
||
# Test package locally
|
||
npm pack
|
||
|
||
# Install in test project
|
||
npm install ./flixscaper-1.0.0.tgz
|
||
|
||
# Test functionality
|
||
node -e "import { scraperNetflix } from 'flixscaper'; console.log('Import successful')"
|
||
```
|
||
|
||
#### 2. NPM Registry Publishing
|
||
|
||
```bash
|
||
# Login to npm
|
||
npm login
|
||
|
||
# Publish to public registry
|
||
npm publish
|
||
|
||
# Publish with beta tag
|
||
npm publish --tag beta
|
||
|
||
# Publish dry run
|
||
npm publish --dry-run
|
||
```
|
||
|
||
#### 3. Private Registry Publishing
|
||
|
||
```bash
|
||
# Publish to private registry
|
||
npm publish --registry https://registry.yourcompany.com
|
||
|
||
# Configure default registry
|
||
npm config set registry https://registry.yourcompany.com
|
||
```
|
||
|
||
## 🏗️ Build & Distribution
|
||
|
||
### Source Distribution
|
||
|
||
MetaScraper is distributed as source code with minimal processing:
|
||
|
||
```bash
|
||
# Files included in distribution
|
||
src/
|
||
├── index.js # Main entry point
|
||
├── parser.js # HTML parsing logic
|
||
├── headless.js # Playwright integration
|
||
└── polyfill.js # Node.js compatibility
|
||
|
||
# Documentation files
|
||
README.md
|
||
LICENSE
|
||
CHANGELOG.md
|
||
|
||
# Configuration files
|
||
package.json
|
||
```
|
||
|
||
### Browser/Node.js Compatibility
|
||
|
||
#### Node.js Support Matrix
|
||
|
||
| Node.js Version | Support Status | Notes |
|
||
|-----------------|----------------|-------|
|
||
| 18.x | ✅ Full Support | Requires polyfill |
|
||
| 20.x | ✅ Full Support | Polyfill optional |
|
||
| 22.x | ✅ Full Support | Native support |
|
||
| 16.x | ❌ Not Supported | Use older version or upgrade |
|
||
| <16.x | ❌ Not Supported | Major compatibility issues |
|
||
|
||
#### Compatibility Layer
|
||
|
||
```javascript
|
||
// src/polyfill.js - Automatic compatibility handling
|
||
import { Blob } from 'node:buffer';
|
||
|
||
// Only apply polyfill if needed
|
||
if (typeof globalThis.File === 'undefined') {
|
||
class PolyfillFile extends Blob {
|
||
constructor(parts, name, options = {}) {
|
||
super(parts, options);
|
||
this.name = String(name);
|
||
this.lastModified = options.lastModified ?? Date.now();
|
||
}
|
||
}
|
||
globalThis.File = PolyfillFile;
|
||
}
|
||
|
||
globalThis.Blob = globalThis.Blob || Blob;
|
||
```
|
||
|
||
## 🔄 Continuous Integration/Deployment
|
||
|
||
### GitHub Actions Workflow
|
||
|
||
```yaml
|
||
# .github/workflows/deploy.yml
|
||
name: Deploy Package
|
||
|
||
on:
|
||
push:
|
||
tags:
|
||
- 'v*'
|
||
release:
|
||
types: [published]
|
||
|
||
jobs:
|
||
test:
|
||
runs-on: ubuntu-latest
|
||
strategy:
|
||
matrix:
|
||
node-version: [18.x, 20.x, 22.x]
|
||
|
||
steps:
|
||
- uses: actions/checkout@v3
|
||
|
||
- name: Setup Node.js
|
||
uses: actions/setup-node@v3
|
||
with:
|
||
node-version: ${{ matrix.node-version }}
|
||
cache: 'npm'
|
||
registry-url: 'https://registry.npmjs.org'
|
||
|
||
- name: Install dependencies
|
||
run: npm ci
|
||
|
||
- name: Run tests
|
||
run: npm test
|
||
|
||
- name: Run linting
|
||
run: npm run lint
|
||
|
||
- name: Check build
|
||
run: npm pack
|
||
|
||
publish:
|
||
needs: test
|
||
runs-on: ubuntu-latest
|
||
if: github.event_name == 'release' || startsWith(github.ref, 'refs/tags/')
|
||
|
||
steps:
|
||
- uses: actions/checkout@v3
|
||
|
||
- name: Setup Node.js
|
||
uses: actions/setup-node@v3
|
||
with:
|
||
node-version: '20.x'
|
||
cache: 'npm'
|
||
registry-url: 'https://registry.npmjs.org'
|
||
|
||
- name: Install dependencies
|
||
run: npm ci
|
||
|
||
- name: Build package
|
||
run: npm pack
|
||
|
||
- name: Publish to NPM
|
||
run: npm publish
|
||
env:
|
||
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
||
```
|
||
|
||
### Automated Testing Pipeline
|
||
|
||
```yaml
|
||
# .github/workflows/test.yml
|
||
name: Test Suite
|
||
|
||
on:
|
||
push:
|
||
branches: [ main, develop ]
|
||
pull_request:
|
||
branches: [ main ]
|
||
|
||
jobs:
|
||
test:
|
||
runs-on: ubuntu-latest
|
||
strategy:
|
||
matrix:
|
||
node-version: [18.x, 20.x, 22.x]
|
||
os: [ubuntu-latest, windows-latest, macos-latest]
|
||
|
||
steps:
|
||
- uses: actions/checkout@v3
|
||
|
||
- name: Setup Node.js ${{ matrix.node-version }}
|
||
uses: actions/setup-node@v3
|
||
with:
|
||
node-version: ${{ matrix.node-version }}
|
||
cache: 'npm'
|
||
|
||
- name: Install dependencies
|
||
run: npm ci
|
||
|
||
- name: Install Playwright (if needed)
|
||
run: npx playwright install chromium
|
||
|
||
- name: Run tests
|
||
run: npm test -- --coverage
|
||
|
||
- name: Upload coverage
|
||
uses: codecov/codecov-action@v3
|
||
```
|
||
|
||
## 🐳 Docker Deployment
|
||
|
||
### Dockerfile
|
||
|
||
```dockerfile
|
||
# Dockerfile
|
||
FROM node:18-alpine
|
||
|
||
WORKDIR /app
|
||
|
||
# Copy package files
|
||
COPY package*.json ./
|
||
|
||
# Install dependencies
|
||
RUN npm ci --only=production
|
||
|
||
# Copy source code
|
||
COPY src/ ./src/
|
||
|
||
# Create non-root user
|
||
RUN addgroup -g 1001 -S nodejs
|
||
RUN adduser -S flixscaper -u 1001
|
||
|
||
# Change ownership
|
||
RUN chown -R flixscaper:nodejs /app
|
||
USER flixscaper
|
||
|
||
# Health check
|
||
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
||
CMD node -e "import('flixscaper').then(() => process.exit(0)).catch(() => process.exit(1))"
|
||
|
||
EXPOSE 3000
|
||
|
||
CMD ["node", "-e", "import('flixscaper').then(m => console.log('MetaScraper ready'))"]
|
||
```
|
||
|
||
### Docker Compose
|
||
|
||
```yaml
|
||
# docker-compose.yml
|
||
version: '3.8'
|
||
|
||
services:
|
||
flixscaper:
|
||
build: .
|
||
container_name: flixscaper
|
||
environment:
|
||
- NODE_ENV=production
|
||
volumes:
|
||
- ./logs:/app/logs
|
||
restart: unless-stopped
|
||
|
||
flixscaper-test:
|
||
build: .
|
||
container_name: flixscaper-test
|
||
command: npm test
|
||
environment:
|
||
- NODE_ENV=test
|
||
volumes:
|
||
- .:/app
|
||
- /app/node_modules
|
||
```
|
||
|
||
### Building Docker Images
|
||
|
||
```bash
|
||
# Build image
|
||
docker build -t flixscaper:latest .
|
||
|
||
# Build with specific version
|
||
docker build -t flixscaper:1.0.0 .
|
||
|
||
# Run container
|
||
docker run --rm flixscaper:latest node -e "
|
||
import('flixscaper').then(async (m) => {
|
||
const result = await m.scraperNetflix('https://www.netflix.com/title/80189685');
|
||
console.log(result);
|
||
})
|
||
"
|
||
```
|
||
|
||
## 🔒 Security Considerations
|
||
|
||
### Package Security
|
||
|
||
#### Dependency Scanning
|
||
|
||
```bash
|
||
# Audit dependencies for vulnerabilities
|
||
npm audit
|
||
|
||
# Fix vulnerabilities
|
||
npm audit fix
|
||
|
||
# Generate security report
|
||
npm audit --json > security-report.json
|
||
```
|
||
|
||
#### Secure Publishing
|
||
|
||
```bash
|
||
# Use 2FA for npm account
|
||
npm profile enable-2fa
|
||
|
||
# Check package contents before publishing
|
||
npm pack --dry-run
|
||
|
||
# Verify no sensitive files included
|
||
tar -tzf flixscaper-*.tgz | grep -E "(key|secret|password|token)" || echo "No sensitive files found"
|
||
```
|
||
|
||
### Runtime Security
|
||
|
||
#### Input Validation
|
||
|
||
```javascript
|
||
// Ensure all inputs are validated
|
||
function validateInput(url, options = {}) {
|
||
if (!url || typeof url !== 'string') {
|
||
throw new Error('Invalid URL provided');
|
||
}
|
||
|
||
// Validate URL format
|
||
try {
|
||
new URL(url);
|
||
} catch {
|
||
throw new Error('Invalid URL format');
|
||
}
|
||
|
||
// Sanitize options
|
||
const safeOptions = {
|
||
headless: Boolean(options.headless),
|
||
timeoutMs: Math.max(1000, Math.min(60000, Number(options.timeoutMs) || 15000)),
|
||
userAgent: typeof options.userAgent === 'string' ? options.userAgent : undefined
|
||
};
|
||
|
||
return safeOptions;
|
||
}
|
||
```
|
||
|
||
#### Network Security
|
||
|
||
```javascript
|
||
// Secure request configuration
|
||
const secureHeaders = {
|
||
'User-Agent': userAgent || DEFAULT_USER_AGENT,
|
||
'Accept': 'text/html,application/xhtml+xml',
|
||
'Accept-Language': 'en-US,en;q=0.9',
|
||
'Cache-Control': 'no-cache',
|
||
'Pragma': 'no-cache'
|
||
};
|
||
|
||
// Rate limiting consideration
|
||
const requestDelay = 1000; // 1 second between requests
|
||
```
|
||
|
||
## 📊 Monitoring & Analytics
|
||
|
||
### Usage Analytics
|
||
|
||
#### Basic Metrics Collection
|
||
|
||
```javascript
|
||
// Optional analytics (user consent required)
|
||
function trackUsage(url, options, success, duration) {
|
||
if (!options.analytics) return;
|
||
|
||
const metrics = {
|
||
timestamp: Date.now(),
|
||
url: url.replace(/\/title\/\d+/, '/title/XXXXXX'), // Anonymize
|
||
headless: options.headless,
|
||
success: success,
|
||
duration: duration,
|
||
nodeVersion: process.version,
|
||
version: require('./package.json').version
|
||
};
|
||
|
||
// Send to analytics service (optional)
|
||
// analytics.track('flixscaper_usage', metrics);
|
||
}
|
||
```
|
||
|
||
#### Error Tracking
|
||
|
||
```javascript
|
||
function trackError(error, context) {
|
||
const errorInfo = {
|
||
message: error.message,
|
||
stack: error.stack,
|
||
context: context,
|
||
timestamp: Date.now(),
|
||
nodeVersion: process.version
|
||
};
|
||
|
||
// Log for debugging
|
||
console.error('MetaScraper Error:', errorInfo);
|
||
|
||
// Optional: Send to error tracking service
|
||
// errorTracker.captureException(error, { extra: context });
|
||
}
|
||
```
|
||
|
||
### Performance Monitoring
|
||
|
||
```javascript
|
||
// Performance metrics
|
||
class PerformanceMonitor {
|
||
constructor() {
|
||
this.metrics = {
|
||
totalRequests: 0,
|
||
successfulRequests: 0,
|
||
averageResponseTime: 0,
|
||
errorCounts: {}
|
||
};
|
||
}
|
||
|
||
recordRequest(duration, success, error = null) {
|
||
this.metrics.totalRequests++;
|
||
|
||
if (success) {
|
||
this.metrics.successfulRequests++;
|
||
} else {
|
||
this.metrics.errorCounts[error?.message] =
|
||
(this.metrics.errorCounts[error?.message] || 0) + 1;
|
||
}
|
||
|
||
// Update average response time
|
||
this.metrics.averageResponseTime =
|
||
(this.metrics.averageResponseTime * (this.metrics.totalRequests - 1) + duration)
|
||
/ this.metrics.totalRequests;
|
||
}
|
||
|
||
getMetrics() {
|
||
return {
|
||
...this.metrics,
|
||
successRate: (this.metrics.successfulRequests / this.metrics.totalRequests) * 100
|
||
};
|
||
}
|
||
}
|
||
```
|
||
|
||
## 🔄 Version Management
|
||
|
||
### Release Process
|
||
|
||
#### 1. Development Release
|
||
|
||
```bash
|
||
# Create feature branch
|
||
git checkout -b feature/new-patterns
|
||
|
||
# Implement changes
|
||
# Add tests
|
||
# Update documentation
|
||
|
||
# Create development release
|
||
npm version prerelease --preid=dev
|
||
git push --tags
|
||
npm publish --tag dev
|
||
```
|
||
|
||
#### 2. Production Release
|
||
|
||
```bash
|
||
# Merge to main
|
||
git checkout main
|
||
git merge develop
|
||
|
||
# Update version
|
||
npm version minor # or patch/major
|
||
|
||
# Create GitHub release
|
||
gh release create v1.1.0 --generate-notes
|
||
|
||
# Publish to npm
|
||
npm publish
|
||
```
|
||
|
||
#### 3. Hotfix Release
|
||
|
||
```bash
|
||
# Create hotfix branch from main
|
||
git checkout -b hotfix/critical-bug
|
||
|
||
# Fix issue
|
||
npm version patch
|
||
|
||
# Publish immediately
|
||
npm publish --tag latest
|
||
|
||
# Merge back to develop
|
||
git checkout develop
|
||
git merge main
|
||
git checkout main
|
||
git merge hotfix/critical-bug
|
||
```
|
||
|
||
### Changelog Management
|
||
|
||
```markdown
|
||
# CHANGELOG.md
|
||
|
||
## [1.1.0] - 2025-11-23
|
||
|
||
### Added
|
||
- New Turkish UI pattern: "yeni başlık"
|
||
- Performance monitoring API
|
||
- Docker support
|
||
|
||
### Fixed
|
||
- Memory leak in Playwright cleanup
|
||
- URL validation for Turkish Netflix domains
|
||
|
||
### Changed
|
||
- Improved error messages in Turkish
|
||
- Updated Node.js compatibility matrix
|
||
|
||
### Deprecated
|
||
- Support for Node.js 16.x (will be removed in 2.0.0)
|
||
|
||
## [1.0.1] - 2025-11-20
|
||
|
||
### Fixed
|
||
- Critical bug in title cleaning
|
||
- Missing year extraction for movies
|
||
```
|
||
|
||
## 🌐 Distribution Channels
|
||
|
||
### NPM Registry
|
||
|
||
```json
|
||
// package.json - publishing configuration
|
||
{
|
||
"publishConfig": {
|
||
"access": "public",
|
||
"registry": "https://registry.npmjs.org"
|
||
},
|
||
"repository": {
|
||
"type": "git",
|
||
"url": "https://github.com/username/flixscaper.git"
|
||
},
|
||
"bugs": {
|
||
"url": "https://github.com/username/flixscaper/issues"
|
||
},
|
||
"homepage": "https://github.com/username/flixscaper#readme"
|
||
}
|
||
```
|
||
|
||
### CDN Distribution
|
||
|
||
```javascript
|
||
// For browser usage (future enhancement)
|
||
// Available via CDN:
|
||
// https://cdn.jsdelivr.net/npm/flixscaper/dist/flixscaper.min.js
|
||
|
||
import('https://cdn.jsdelivr.net/npm/flixscaper@latest/dist/flixscaper.min.js')
|
||
.then(module => {
|
||
const { scraperNetflix } = module;
|
||
// Use in browser
|
||
});
|
||
```
|
||
|
||
### Private Distribution
|
||
|
||
```bash
|
||
# For enterprise/internal distribution
|
||
npm config set @company:registry https://npm.company.com
|
||
|
||
# Publish to private registry
|
||
npm publish --registry https://npm.company.com
|
||
|
||
# Install from private registry
|
||
npm install @company/flixscaper
|
||
```
|
||
|
||
---
|
||
|
||
*Deployment guide last updated: 2025-11-23* |