WIP
This commit is contained in:
Yaojia Wang
218
docs/web-refactoring-status.md
Normal file
218
docs/web-refactoring-status.md
Normal file
@@ -0,0 +1,218 @@
|
||||
# Web Directory Refactoring - Current Status
|
||||
|
||||
## ✅ Completed Steps
|
||||
|
||||
### 1. Directory Structure Created
|
||||
```
|
||||
src/web/
|
||||
├── api/
|
||||
│ ├── v1/
|
||||
│ │ ├── admin/ (documents.py, annotations.py, training.py)
|
||||
│ │ ├── async_api/ (routes.py)
|
||||
│ │ ├── batch/ (routes.py)
|
||||
│ │ └── routes.py (public inference API)
|
||||
├── schemas/
|
||||
│ ├── admin.py (admin schemas)
|
||||
│ ├── inference.py (inference + async schemas)
|
||||
│ └── common.py (ErrorResponse)
|
||||
├── services/
|
||||
│ ├── autolabel.py
|
||||
│ ├── async_processing.py
|
||||
│ ├── batch_upload.py
|
||||
│ └── inference.py
|
||||
├── core/
|
||||
│ ├── auth.py
|
||||
│ ├── rate_limiter.py
|
||||
│ └── scheduler.py
|
||||
└── workers/
|
||||
├── async_queue.py
|
||||
└── batch_queue.py
|
||||
```
|
||||
|
||||
### 2. Files Copied and Imports Updated
|
||||
|
||||
#### Admin API (✅ Complete)
|
||||
- [x] `admin_routes.py` → `api/v1/admin/documents.py` (imports updated)
|
||||
- [x] `admin_annotation_routes.py` → `api/v1/admin/annotations.py` (imports updated)
|
||||
- [x] `admin_training_routes.py` → `api/v1/admin/training.py` (imports updated)
|
||||
- [x] `api/v1/admin/__init__.py` created with exports
|
||||
|
||||
#### Public & Async API (✅ Complete)
|
||||
- [x] `routes.py` → `api/v1/routes.py` (imports updated)
|
||||
- [x] `async_routes.py` → `api/v1/async_api/routes.py` (imports updated)
|
||||
- [x] `batch_upload_routes.py` → `api/v1/batch/routes.py` (copied, imports pending)
|
||||
|
||||
#### Schemas (✅ Complete)
|
||||
- [x] `admin_schemas.py` → `schemas/admin.py`
|
||||
- [x] `schemas.py` → `schemas/inference.py`
|
||||
- [x] `schemas/common.py` created
|
||||
- [x] `schemas/__init__.py` created with exports
|
||||
|
||||
#### Services (✅ Complete)
|
||||
- [x] `admin_autolabel.py` → `services/autolabel.py`
|
||||
- [x] `async_service.py` → `services/async_processing.py`
|
||||
- [x] `batch_upload_service.py` → `services/batch_upload.py`
|
||||
- [x] `services.py` → `services/inference.py`
|
||||
- [x] `services/__init__.py` created
|
||||
|
||||
#### Core Components (✅ Complete)
|
||||
- [x] `admin_auth.py` → `core/auth.py`
|
||||
- [x] `rate_limiter.py` → `core/rate_limiter.py`
|
||||
- [x] `admin_scheduler.py` → `core/scheduler.py`
|
||||
- [x] `core/__init__.py` created
|
||||
|
||||
#### Workers (✅ Complete)
|
||||
- [x] `async_queue.py` → `workers/async_queue.py`
|
||||
- [x] `batch_queue.py` → `workers/batch_queue.py`
|
||||
- [x] `workers/__init__.py` created
|
||||
|
||||
#### Main App (✅ Complete)
|
||||
- [x] `app.py` imports updated to use new structure
|
||||
|
||||
---
|
||||
|
||||
## ⏳ Remaining Work
|
||||
|
||||
### 1. Update Remaining File Imports (HIGH PRIORITY)
|
||||
|
||||
Files that need import updates:
|
||||
- [ ] `api/v1/batch/routes.py` - update to use new schema/service imports
|
||||
- [ ] `services/autolabel.py` - may need import updates if it references old paths
|
||||
- [ ] `services/async_processing.py` - check for old import references
|
||||
- [ ] `services/batch_upload.py` - check for old import references
|
||||
- [ ] `services/inference.py` - check for old import references
|
||||
|
||||
### 2. Update ALL Test Files (CRITICAL)
|
||||
|
||||
Test files need to import from new locations. Pattern:
|
||||
|
||||
**Old:**
|
||||
```python
|
||||
from src.web.admin_routes import create_admin_router
|
||||
from src.web.admin_schemas import DocumentItem
|
||||
from src.web.admin_auth import validate_admin_token
|
||||
```
|
||||
|
||||
**New:**
|
||||
```python
|
||||
from src.web.api.v1.admin import create_admin_router
|
||||
from src.web.schemas.admin import DocumentItem
|
||||
from src.web.core.auth import validate_admin_token
|
||||
```
|
||||
|
||||
Test files to update:
|
||||
- [ ] `tests/web/test_admin_annotations.py`
|
||||
- [ ] `tests/web/test_admin_auth.py`
|
||||
- [ ] `tests/web/test_admin_routes.py`
|
||||
- [ ] `tests/web/test_admin_routes_enhanced.py`
|
||||
- [ ] `tests/web/test_admin_training.py`
|
||||
- [ ] `tests/web/test_annotation_locks.py`
|
||||
- [ ] `tests/web/test_annotation_phase5.py`
|
||||
- [ ] `tests/web/test_async_queue.py`
|
||||
- [ ] `tests/web/test_async_routes.py`
|
||||
- [ ] `tests/web/test_async_service.py`
|
||||
- [ ] `tests/web/test_autolabel_with_locks.py`
|
||||
- [ ] `tests/web/test_batch_queue.py`
|
||||
- [ ] `tests/web/test_batch_upload_routes.py`
|
||||
- [ ] `tests/web/test_batch_upload_service.py`
|
||||
- [ ] `tests/web/test_rate_limiter.py`
|
||||
- [ ] `tests/web/test_training_phase4.py`
|
||||
|
||||
### 3. Create Backward Compatibility Layer (OPTIONAL)
|
||||
|
||||
Keep old imports working temporarily:
|
||||
|
||||
```python
|
||||
# src/web/admin_routes.py (temporary compatibility shim)
|
||||
\"\"\"
|
||||
DEPRECATED: Use src.web.api.v1.admin.documents instead.
|
||||
This file will be removed in next version.
|
||||
\"\"\"
|
||||
import warnings
|
||||
from src.web.api.v1.admin.documents import *
|
||||
|
||||
warnings.warn(
|
||||
"Importing from src.web.admin_routes is deprecated. "
|
||||
"Use src.web.api.v1.admin.documents instead.",
|
||||
DeprecationWarning,
|
||||
stacklevel=2
|
||||
)
|
||||
```
|
||||
|
||||
### 4. Verify and Test
|
||||
|
||||
1. Run tests:
|
||||
```bash
|
||||
pytest tests/web/ -v
|
||||
```
|
||||
|
||||
2. Check for any import errors:
|
||||
```bash
|
||||
python -c "from src.web.app import create_app; create_app()"
|
||||
```
|
||||
|
||||
3. Start server and test endpoints:
|
||||
```bash
|
||||
python run_server.py
|
||||
```
|
||||
|
||||
### 5. Clean Up Old Files (ONLY AFTER TESTS PASS)
|
||||
|
||||
Old files to remove:
|
||||
- `src/web/admin_*.py` (7 files)
|
||||
- `src/web/async_*.py` (3 files)
|
||||
- `src/web/batch_*.py` (3 files)
|
||||
- `src/web/routes.py`
|
||||
- `src/web/services.py`
|
||||
- `src/web/schemas.py`
|
||||
- `src/web/rate_limiter.py`
|
||||
|
||||
Keep these files (don't remove):
|
||||
- `src/web/__init__.py`
|
||||
- `src/web/app.py`
|
||||
- `src/web/config.py`
|
||||
- `src/web/dependencies.py`
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Next Immediate Steps
|
||||
|
||||
1. **Update batch/routes.py imports** - Quick fix for remaining API route
|
||||
2. **Update test file imports** - Critical for verification
|
||||
3. **Run test suite** - Verify nothing broke
|
||||
4. **Fix any import errors** - Address failures
|
||||
5. **Remove old files** - Clean up after tests pass
|
||||
|
||||
---
|
||||
|
||||
## 📊 Migration Impact Summary
|
||||
|
||||
| Category | Files Moved | Imports Updated | Status |
|
||||
|----------|-------------|-----------------|--------|
|
||||
| API Routes | 7 | 5/7 | 🟡 In Progress |
|
||||
| Schemas | 3 | 3/3 | ✅ Complete |
|
||||
| Services | 4 | 0/4 | ⚠️ Pending |
|
||||
| Core | 3 | 3/3 | ✅ Complete |
|
||||
| Workers | 2 | 2/2 | ✅ Complete |
|
||||
| Tests | 0 | 0/16 | ❌ Not Started |
|
||||
|
||||
**Overall Progress: 65%**
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Benefits After Migration
|
||||
|
||||
1. **Better Organization**: Clear separation by function
|
||||
2. **Easier Navigation**: Find files by purpose, not prefix
|
||||
3. **Scalability**: Easy to add new API versions
|
||||
4. **Standard Structure**: Follows FastAPI best practices
|
||||
5. **Maintainability**: Each module has single responsibility
|
||||
|
||||
---
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
- All original files are still in place (no data loss risk)
|
||||
- New structure is operational but needs import updates
|
||||
- Backward compatibility can be added if needed
|
||||
- Tests will validate the migration success
|
||||
Reference in New Issue
Block a user