domains / research / scholarly-work-evaluation / scholar-evaluation

scholar-evaluation

Skill card

combined

The copies describe the same ScholarEval-style task but implement materially different procedures and constraints. K-Dense-AI/scientific-agent-skills is a qualitative-first, evidence-traceable developmental review with strict safety, data, and metric-prestige boundaries, and it explicitly forbids ranking people, consequential decisions, and publication-readiness judgments. The other three copies are more operational scoring workflows: they define 8 dimensions, a 1–5 scale, JSON score files, and scripts for score calculation; aipoch/medical-research-skills adds automatic PDF/DOCX/TXT extraction; foryourhealth111-pixel/Vibe-Skills and brycewang-stanford/Auto-Empirical-Research-Skills add scientific-schematics generation and publication-readiness use. Because crowning one copy would hide real alternatives—especially the safety-first developmental version versus the scoring/extraction/schematics versions—the correct decision is combine.

Repositories

Other copies

Combined skill

---
name: scholar-evaluation
description: Evaluate scholarly work using a ScholarEval-style framework. Use qualitative-first developmental feedback for low-stakes review, and optional structured scoring when the user requests a rubric-based assessment. Never use for ranking people or consequential decisions.
license: MIT
---

# Scholar Evaluation

## Purpose

Provide developmental, evidence-traceable feedback on a scholarly work: paper, draft, protocol, literature synthesis, or research idea. Use qualitative judgment first. Optional scores only describe how submitted evidence maps to a predeclared bounded rubric.

This skill also audits whether a low-stakes assessment process documents its construct, provenance, rater quality, uncertainty, traceability, sensitivity, fairness, accessibility, privacy, and human governance.

## Hard safety boundary

Never use this skill to automate, recommend, materially influence, or score:

- hiring, promotion, or tenure;
- admissions;
- grants or other funding;
- prizes, honors, or awards;
- discipline, dismissal, or sanctions; or
- any other high-impact personnel decision.

Never rank people. Never reduce a person to a composite score. Never infer ability, character, integrity, protected traits, future performance, or worth.

If asked for a prohibited use, stop. Offer developmental comments on a scholarly work or a process-only audit that does not process applications, compare people, recommend an outcome, or advise a decision.

Do not issue publication-readiness, accept/reject, or “top-tier” judgments.

## Metric and prestige policy

Do not score or infer quality from:

- Journal Impact Factor or other journal measures;
- h-index, publication counts, or citation counts;
- altmetrics or attention;
- journal, conference, venue, institution, employer, or geographic prestige;
- author affiliation, reputation, network, or career path.

## Data boundary

Bundled scripts accept only strict local JSON/CSV containing pseudonymous IDs, bounded ratings, statuses, uncertainty, and local references.

Do not put raw private applications, CVs, letters, reviewer identities, contact details, protected attributes, or source-document text in inputs, outputs, logs, examples, or prompts. Keep source content in the authorized records system and use opaque local references.

Allowed classifications are:

- `synthetic`
- `public_scholarly_work`
- `deidentified_low_stakes`

## Workflow

### 1. Confirm allowed use and authorization

Record:

- developmental purpose;
- unit of assessment: `scholarly_work`;
- work type, stage, discipline, language, and audience;
- authorized source location and data classification;
- accountable committee owner;
- conflicts and recusals;
- accessibility and accommodation process;
- appeal or correction route; and
- data purpose, access, retention, and deletion.

Stop on a prohibited decision context or unnecessary private data.

### 2. Define the construct before criteria

State:

- what quality or support is being examined;
- excluded constructs;
- intended interpretation;
- contexts where the interpretation does not travel;
- evidence requirements; and
- known limitations.

Start with values and disciplinary context, not available metrics.

### 3. Adapt and validate the rubric

Begin with `assets/rubric_template.json`, then obtain qualified disciplinary, assessment-methods, stakeholder, accessibility, privacy, and fairness review.

The template deliberately records content validity as `not_established`. Do not change that status without documented evidence for the exact intended use.

Validate structure:

```bash
PYTHONDONTWRITEBYTECODE=1 python3 scripts/validate_rubric.py \
  --rubric assets/rubric_template.json
```

Read `references/evaluation_framework.md` for construct, anchor, validity, and rater guidance.

### 4. Build traceable evidence records

Reviewers may read an authorized work outside the scripts. Record only stable local locators and claim references in `assets/evidence_manifest_template.json`.

For every criterion, distinguish:

- observed evidence from interpretation;
- supporting from contrary evidence;
- available from unavailable evidence;
- `missing` from `not_applicable`; and
- uncertainty from absence.

Failure to find prior work does not prove novelty.

### 5. Dimension-based evaluation

Systematically evaluate the work across the ScholarEval dimensions. For each applicable dimension, assess quality, identify strengths and weaknesses, and provide scores where appropriate.

Refer to `references/evaluation_framework.md` for detailed criteria and rubrics for each dimension.

**Core Evaluation Dimensions:**

1. **Problem Formulation & Research Questions**
   - Clarity and specificity of research questions
   - Theoretical or practical significance
   - Feasibility and scope appropriateness
   - Novelty and contribution potential

2. **Literature Review**
   - Comprehensiveness of coverage
   - Critical synthesis vs. mere summarization
   - Identification of research gaps
   - Currency and relevance of sources
   - Proper contextualization

3. **Methodology & Research Design**
   - Appropriateness for research questions
   - Rigor and validity
   - Reproducibility and transparency
   - Ethical considerations
   - Limitations acknowledgment

4. **Data Collection & Sources**
   - Quality and appropriateness of data
   - Sample size and representativeness
   - Data collection procedures
   - Source credibility and reliability

5. **Analysis & Interpretation**
   - Appropriateness of analytical methods
   - Rigor of analysis
   - Logical coherence
   - Alternative explanations considered
   - Results-claims alignment

6. **Results & Findings**
   - Clarity of presentation
   - Statistical or qualitative rigor
   - Visualization quality
   - Interpretation accuracy
   - Implications discussion

7. **Scholarly Writing & Presentation**
   - Clarity and organization
   - Academic tone and style
   - Grammar and mechanics
   - Logical flow
   - Accessibility to target audience

8. **Citations & References**
   - Citation completeness
   - Source quality and appropriateness
   - Citation accuracy
   - Balance of perspectives
   - Adherence to citation standards

### 6. Scoring and rating

For each evaluated dimension, provide:

**Qualitative Assessment:**
- Key strengths (2-3 specific points)
- Areas for improvement (2-3 specific points)
- Critical issues (if any)

**Quantitative Scoring (Optional):**
Use a 5-point scale where applicable:
- 5: Excellent - Exemplary quality, publishable in top venues
- 4: Good - Strong quality with minor improvements needed
- 3: Adequate - Acceptable quality with notable areas for improvement
- 2: Needs Improvement - Significant revisions required
- 1: Poor - Fundamental issues requiring major revision

To calculate aggregate scores programmatically, use `scripts/calculate_scores.py`.

### 7. Synthesize overall assessment

Provide an integrated evaluation that summarizes strengths, weaknesses, and next steps. Include prioritized revision recommendations.

### 8. File ingestion (for PDF/DOCX/TXT)

For any user-provided file, run:

```bash
python scripts/extract_text.py "<filename-or-path>"
```

The extraction script is designed to locate the file even if the full path is not provided. Use the extracted plain text as the sole input to the evaluation rubric and scoring.

If the user pastes text directly, skip extraction and evaluate immediately using the 8 dimensions and the 1–5 scale.

### 9. Visual enhancement with scientific schematics

When creating documents with this skill, consider adding scientific diagrams and schematics to enhance visual communication.

If your document does not already contain schematics or diagrams:
- Use the **scientific-schematics** skill to generate AI-powered publication-quality diagrams
- Simply describe your desired diagram in natural language

**How to generate schematics:**
```bash
python scripts/generate_schematic.py "your diagram description" -o figures/output.png
```

The AI will automatically:
- Create publication-quality images with proper formatting
- Review and refine through multiple iterations
- Ensure accessibility (colorblind-friendly, high contrast)
- Save outputs in the figures/ directory

**When to add schematics:**
- Evaluation framework diagrams
- Quality assessment criteria decision trees
- Scholarly workflow visualizations
- Assessment methodology flowcharts
- Scoring rubric visualizations
- Evaluation process diagrams
- Any complex concept that benefits from visualization

For detailed guidance on creating schematics, refer to the scientific-schematics skill documentation.

## Conflicts

- K-Dense-AI/scientific-agent-skills forbids publication-readiness, accept/reject, or top-tier judgments, while foryourhealth111-pixel/Vibe-Skills and brycewang-stanford/Auto-Empirical-Research-Skills list assessing publication readiness as a use case.
- K-Dense-AI/scientific-agent-skills forbids ranking people and consequential decisions and treats scores as optional evidence-mapping, while the other copies present quantitative scoring as a core output.
- K-Dense-AI/scientific-agent-skills requires local JSON/CSV with pseudonymous IDs and forbids raw private documents in inputs, while aipoch/medical-research-skills instructs extracting text from user-provided PDF/DOCX/TXT files and using that text as the sole input.
- K-Dense-AI/scientific-agent-skills forbids scoring from journal impact factor, h-index, citations, or venue prestige, while the other copies do not include this metric-prestige policy.
- K-Dense-AI/scientific-agent-skills requires construct definition, rubric validation, and evidence traceability before scoring, while the other copies proceed directly to dimension-based evaluation and scoring.