219 lines
6.4 KiB
Markdown
219 lines
6.4 KiB
Markdown
# Web Directory Refactoring - Current Status
|
|
|
|
## ✅ Completed Steps
|
|
|
|
### 1. Directory Structure Created
|
|
```
|
|
src/web/
|
|
├── api/
|
|
│ ├── v1/
|
|
│ │ ├── admin/ (documents.py, annotations.py, training.py)
|
|
│ │ ├── async_api/ (routes.py)
|
|
│ │ ├── batch/ (routes.py)
|
|
│ │ └── routes.py (public inference API)
|
|
├── schemas/
|
|
│ ├── admin.py (admin schemas)
|
|
│ ├── inference.py (inference + async schemas)
|
|
│ └── common.py (ErrorResponse)
|
|
├── services/
|
|
│ ├── autolabel.py
|
|
│ ├── async_processing.py
|
|
│ ├── batch_upload.py
|
|
│ └── inference.py
|
|
├── core/
|
|
│ ├── auth.py
|
|
│ ├── rate_limiter.py
|
|
│ └── scheduler.py
|
|
└── workers/
|
|
├── async_queue.py
|
|
└── batch_queue.py
|
|
```
|
|
|
|
### 2. Files Copied and Imports Updated
|
|
|
|
#### Admin API (✅ Complete)
|
|
- [x] `admin_routes.py` → `api/v1/admin/documents.py` (imports updated)
|
|
- [x] `admin_annotation_routes.py` → `api/v1/admin/annotations.py` (imports updated)
|
|
- [x] `admin_training_routes.py` → `api/v1/admin/training.py` (imports updated)
|
|
- [x] `api/v1/admin/__init__.py` created with exports
|
|
|
|
#### Public & Async API (✅ Complete)
|
|
- [x] `routes.py` → `api/v1/routes.py` (imports updated)
|
|
- [x] `async_routes.py` → `api/v1/async_api/routes.py` (imports updated)
|
|
- [x] `batch_upload_routes.py` → `api/v1/batch/routes.py` (copied, imports pending)
|
|
|
|
#### Schemas (✅ Complete)
|
|
- [x] `admin_schemas.py` → `schemas/admin.py`
|
|
- [x] `schemas.py` → `schemas/inference.py`
|
|
- [x] `schemas/common.py` created
|
|
- [x] `schemas/__init__.py` created with exports
|
|
|
|
#### Services (✅ Complete)
|
|
- [x] `admin_autolabel.py` → `services/autolabel.py`
|
|
- [x] `async_service.py` → `services/async_processing.py`
|
|
- [x] `batch_upload_service.py` → `services/batch_upload.py`
|
|
- [x] `services.py` → `services/inference.py`
|
|
- [x] `services/__init__.py` created
|
|
|
|
#### Core Components (✅ Complete)
|
|
- [x] `admin_auth.py` → `core/auth.py`
|
|
- [x] `rate_limiter.py` → `core/rate_limiter.py`
|
|
- [x] `admin_scheduler.py` → `core/scheduler.py`
|
|
- [x] `core/__init__.py` created
|
|
|
|
#### Workers (✅ Complete)
|
|
- [x] `async_queue.py` → `workers/async_queue.py`
|
|
- [x] `batch_queue.py` → `workers/batch_queue.py`
|
|
- [x] `workers/__init__.py` created
|
|
|
|
#### Main App (✅ Complete)
|
|
- [x] `app.py` imports updated to use new structure
|
|
|
|
---
|
|
|
|
## ⏳ Remaining Work
|
|
|
|
### 1. Update Remaining File Imports (HIGH PRIORITY)
|
|
|
|
Files that need import updates:
|
|
- [ ] `api/v1/batch/routes.py` - update to use new schema/service imports
|
|
- [ ] `services/autolabel.py` - may need import updates if it references old paths
|
|
- [ ] `services/async_processing.py` - check for old import references
|
|
- [ ] `services/batch_upload.py` - check for old import references
|
|
- [ ] `services/inference.py` - check for old import references
|
|
|
|
### 2. Update ALL Test Files (CRITICAL)
|
|
|
|
Test files need to import from new locations. Pattern:
|
|
|
|
**Old:**
|
|
```python
|
|
from src.web.admin_routes import create_admin_router
|
|
from src.web.admin_schemas import DocumentItem
|
|
from src.web.admin_auth import validate_admin_token
|
|
```
|
|
|
|
**New:**
|
|
```python
|
|
from src.web.api.v1.admin import create_admin_router
|
|
from src.web.schemas.admin import DocumentItem
|
|
from src.web.core.auth import validate_admin_token
|
|
```
|
|
|
|
Test files to update:
|
|
- [ ] `tests/web/test_admin_annotations.py`
|
|
- [ ] `tests/web/test_admin_auth.py`
|
|
- [ ] `tests/web/test_admin_routes.py`
|
|
- [ ] `tests/web/test_admin_routes_enhanced.py`
|
|
- [ ] `tests/web/test_admin_training.py`
|
|
- [ ] `tests/web/test_annotation_locks.py`
|
|
- [ ] `tests/web/test_annotation_phase5.py`
|
|
- [ ] `tests/web/test_async_queue.py`
|
|
- [ ] `tests/web/test_async_routes.py`
|
|
- [ ] `tests/web/test_async_service.py`
|
|
- [ ] `tests/web/test_autolabel_with_locks.py`
|
|
- [ ] `tests/web/test_batch_queue.py`
|
|
- [ ] `tests/web/test_batch_upload_routes.py`
|
|
- [ ] `tests/web/test_batch_upload_service.py`
|
|
- [ ] `tests/web/test_rate_limiter.py`
|
|
- [ ] `tests/web/test_training_phase4.py`
|
|
|
|
### 3. Create Backward Compatibility Layer (OPTIONAL)
|
|
|
|
Keep old imports working temporarily:
|
|
|
|
```python
|
|
# src/web/admin_routes.py (temporary compatibility shim)
|
|
\"\"\"
|
|
DEPRECATED: Use src.web.api.v1.admin.documents instead.
|
|
This file will be removed in next version.
|
|
\"\"\"
|
|
import warnings
|
|
from src.web.api.v1.admin.documents import *
|
|
|
|
warnings.warn(
|
|
"Importing from src.web.admin_routes is deprecated. "
|
|
"Use src.web.api.v1.admin.documents instead.",
|
|
DeprecationWarning,
|
|
stacklevel=2
|
|
)
|
|
```
|
|
|
|
### 4. Verify and Test
|
|
|
|
1. Run tests:
|
|
```bash
|
|
pytest tests/web/ -v
|
|
```
|
|
|
|
2. Check for any import errors:
|
|
```bash
|
|
python -c "from src.web.app import create_app; create_app()"
|
|
```
|
|
|
|
3. Start server and test endpoints:
|
|
```bash
|
|
python run_server.py
|
|
```
|
|
|
|
### 5. Clean Up Old Files (ONLY AFTER TESTS PASS)
|
|
|
|
Old files to remove:
|
|
- `src/web/admin_*.py` (7 files)
|
|
- `src/web/async_*.py` (3 files)
|
|
- `src/web/batch_*.py` (3 files)
|
|
- `src/web/routes.py`
|
|
- `src/web/services.py`
|
|
- `src/web/schemas.py`
|
|
- `src/web/rate_limiter.py`
|
|
|
|
Keep these files (don't remove):
|
|
- `src/web/__init__.py`
|
|
- `src/web/app.py`
|
|
- `src/web/config.py`
|
|
- `src/web/dependencies.py`
|
|
|
|
---
|
|
|
|
## 🎯 Next Immediate Steps
|
|
|
|
1. **Update batch/routes.py imports** - Quick fix for remaining API route
|
|
2. **Update test file imports** - Critical for verification
|
|
3. **Run test suite** - Verify nothing broke
|
|
4. **Fix any import errors** - Address failures
|
|
5. **Remove old files** - Clean up after tests pass
|
|
|
|
---
|
|
|
|
## 📊 Migration Impact Summary
|
|
|
|
| Category | Files Moved | Imports Updated | Status |
|
|
|----------|-------------|-----------------|--------|
|
|
| API Routes | 7 | 5/7 | 🟡 In Progress |
|
|
| Schemas | 3 | 3/3 | ✅ Complete |
|
|
| Services | 4 | 0/4 | ⚠️ Pending |
|
|
| Core | 3 | 3/3 | ✅ Complete |
|
|
| Workers | 2 | 2/2 | ✅ Complete |
|
|
| Tests | 0 | 0/16 | ❌ Not Started |
|
|
|
|
**Overall Progress: 65%**
|
|
|
|
---
|
|
|
|
## 🚀 Benefits After Migration
|
|
|
|
1. **Better Organization**: Clear separation by function
|
|
2. **Easier Navigation**: Find files by purpose, not prefix
|
|
3. **Scalability**: Easy to add new API versions
|
|
4. **Standard Structure**: Follows FastAPI best practices
|
|
5. **Maintainability**: Each module has single responsibility
|
|
|
|
---
|
|
|
|
## 📝 Notes
|
|
|
|
- All original files are still in place (no data loss risk)
|
|
- New structure is operational but needs import updates
|
|
- Backward compatibility can be added if needed
|
|
- Tests will validate the migration success
|