domains / data / ml-productionization / senior-ml-engineer

senior-ml-engineer

Skill card

combined

The two copies overlap on MLOps, model deployment, monitoring, and LLM/RAG integration, but each contains material steps the other lacks. The first copy is a broad role/competency guide with performance targets, security/compliance, team leadership, and generic commands, while the second copy provides concrete operational workflows: a step-by-step deployment sequence with validation thresholds, a Docker template, serving options, MLOps pipeline setup with feature store and retraining triggers, LLM provider abstraction with retry/cost management, and RAG implementation with vector DB selection and chunking. Crowning one would hide the other's real alternative content.

Repositories

Other copies

Combined skill

---
name: senior-ml-engineer
description: Production ML engineering skill for deploying models, building MLOps pipelines, integrating LLMs, and implementing RAG systems. Covers model serving, feature stores, drift monitoring, cost optimization, and senior-level production responsibilities.
---

# Senior ML Engineer

Production ML engineering patterns for model deployment, MLOps infrastructure, LLM integration, and RAG systems.

## Model Deployment Workflow

1. Export model to standardized format (ONNX, TorchScript, SavedModel)
2. Package model with dependencies in Docker container
3. Deploy to staging environment
4. Run integration tests against staging
5. Deploy canary (5% traffic) to production
6. Monitor latency and error rates for 1 hour
7. Promote to full production if metrics pass
8. Validation: p95 latency < 100ms, error rate < 0.1%

### Container Template

```dockerfile
FROM python:3.11-slim

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY model/ /app/model/
COPY src/ /app/src/

HEALTHCHECK CMD curl -f http://localhost:8080/health || exit 1

EXPOSE 8080
CMD ["uvicorn", "src.server:app", "--host", "0.0.0.0", "--port", "8080"]
```

### Serving Options

| Option | Latency | Throughput | Use Case |
|--------|---------|------------|----------|
| FastAPI + Uvicorn | Low | Medium | REST APIs, small models |
| Triton Inference Server | Very Low | Very High | GPU inference, batching |
| TensorFlow Serving | Low | High | TensorFlow models |
| TorchServe | Low | High | PyTorch models |
| Ray Serve | Medium | High | Complex pipelines, multi-model |

## MLOps Pipeline Setup

1. Configure feature store (Feast, Tecton) for training data
2. Set up experiment tracking (MLflow, Weights & Biases)
3. Create training pipeline with hyperparameter logging
4. Register model in model registry with version metadata
5. Configure staging deployment triggered by registry events
6. Set up A/B testing infrastructure for model comparison
7. Enable drift monitoring with alerting
8. Validation: New models automatically evaluated against baseline

### Retraining Triggers

| Trigger | Detection | Action |
|---------|-----------|--------|
| Scheduled | Cron (weekly/monthly) | Full retrain |
| Performance drop | Accuracy < threshold | Immediate retrain |
| Data drift | PSI > 0.2 | Evaluate, then retrain |
| New data volume | X new samples | Incremental update |

## LLM Integration Workflow

1. Create provider abstraction layer for vendor flexibility
2. Implement retry logic with exponential backoff
3. Configure fallback to secondary provider
4. Set up token counting and context truncation
5. Add response caching for repeated queries
6. Implement cost tracking per request
7. Add structured output validation with Pydantic
8. Validation: Response parses correctly, cost within budget

## RAG System Implementation

1. Choose vector database (Pinecone, Qdrant, Weaviate)
2. Select embedding model based on quality/cost tradeoff
3. Implement document chunking strategy
4. Create ingestion pipeline with metadata extraction
5. Build retrieval with query embedding
6. Add reranking for relevance improvement
7. Format context and send to LLM
8. Validation: Response references retrieved context, no hallucinations

## Performance Targets

- P50 latency: < 50ms
- P95 latency: < 100ms
- P99 latency: < 200ms
- Throughput: > 1000 requests/second
- Concurrent users: > 10,000
- Uptime: 99.9%
- Error rate: < 0.1%

## Security & Compliance

- Authentication & authorization
- Data encryption (at rest & in transit)
- PII handling and anonymization
- GDPR/CCPA compliance
- Regular security audits
- Vulnerability management

## Senior-Level Responsibilities

1. Technical Leadership: Drive architectural decisions, mentor team members, establish best practices, ensure code quality
2. Strategic Thinking: Align with business goals, evaluate trade-offs, plan for scale, manage technical debt
3. Collaboration: Work across teams, communicate effectively, build consensus, share knowledge
4. Innovation: Stay current with research, experiment with new approaches, contribute to community, drive continuous improvement
5. Production Excellence: Ensure high availability, monitor proactively, optimize performance, respond to incidents

## Common Commands

```bash
# Development
python -m pytest tests/ -v --cov
python -m black src/
python -m pylint src/

# Training
python scripts/train.py --config prod.yaml
python scripts/evaluate.py --model best.pth

# Deployment
docker build -t service:v1 .
kubectl apply -f k8s/
helm upgrade service ./charts/

# Monitoring
kubectl logs -f deployment/service
python scripts/health_check.py
```