Score Call-for-Papers Submissions

Pre-scores CFP submissions with an anchored rubric and reasons, flags product pitches and duplicates, builds a track-balanced shortlist and prepares reviewer packets.

Example prompts

Once the skill is added, start a chat with one of these and attach your file if you have one.

  • Score these 310 CFP submissions with our rubric and give me the top 60 with a balanced track mix.

  • Flag any submissions that are really product pitches or from vendors who aren't sponsors.

  • Write a scoring rubric with anchors for 1 to 5 so our 8 reviewers score consistently.

What this skill does

This skill pre-scores call-for-papers submissions so your program committee can focus on the close calls. It scores each one on relevance, originality, practical takeaways, clarity and speaker credibility with an anchored 1 to 5 scale and a reason that points at the submission text, then computes a weighted total.

Before scoring it flags duplicates, missing info, over-limit abstracts, product pitches and vendors who aren't sponsors. It fills each track's slots in score order with caps per company and speaker, builds a waitlist, marks borderline rows for committee review, and prepares reviewer packets and note drafts.

When to use it

  • Reviewing hundreds of CFP or call-for-speakers submissions
  • Writing an anchored scoring rubric for your review committee
  • Spotting product pitches and duplicate submissions
  • Balancing the shortlist across tracks, levels and companies
  • Preparing reviewer packets and decline notes

How it works

  1. 1Load and cleanMaps your export's columns, counts rows and flags duplicates, missing info, pitches and conflicts with the reason for each.
  2. 2Score with anchorsRates each criterion 1 to 5 against written anchors and gives a reason for every score.
  3. 3Build the shortlistFills track slots in score order with company and speaker caps, then a waitlist, using an optional Python script.
  4. 4Route the close callsSends borderline and judgment-call submissions to committee review instead of declining them.
  5. 5Prepare the committeeCreates reviewer packets with conflicts avoided and drafts accept, waitlist and decline notes.

What you'll need

  • Submissions export (CSV or XLSX)
  • Tracks, slots and formats
  • Rubric and weights, or use the default
  • Sponsor list, balance goals and reviewer conflicts

What you get

An intake summary with rows in and out, a scored table (also as CSV) with per-criterion scores, total, flags, recommendation and reasons, a shortlist by track with cut lines, a balance check, a committee review list, reviewer packets and draft accept, waitlist and decline notes. Everything is labeled as a pre-score for committee review.

Tips

  • Share your sponsor list so vendor sessions from sponsors aren't treated like unsolicited pitches.
  • Keep at least three human reviewers per final decision; the pre-score is one input.
  • Use blind review packets if you want reviewers to judge content before names.
  • Decide caps per company and per speaker before you look at the results.

What's in the download

  • cfp-review-scorer/SKILL.md8.5 KB
  • cfp-review-scorer/assets/sample-cfp-submissions.csv12.7 KB
  • cfp-review-scorer/references/reviewer-packet.md2.4 KB
  • cfp-review-scorer/references/scoring-rubric.md2.5 KB
  • cfp-review-scorer/scripts/cfp_shortlist.py7.9 KB
Read the SKILL.md
---
name: cfp-review-scorer
description: Reviews call-for-papers and call-for-speakers submissions for conferences, summits and user conferences. Applies an anchored 1 to 5 scoring rubric (relevance, originality, practical takeaways, speaker credibility, clarity) with reasons, flags thinly veiled product pitches, duplicates and missing info, balances tracks, levels and speaker mix, and produces reviewer packets plus accept, waitlist and decline recommendations with notes. Use for CFP review, abstract scoring, session selection or speaker selection.
license: MIT
metadata:
  author: eventmarketingsoftware.com
  version: "1.0"
  category: content-and-speakers
  source: https://eventmarketingsoftware.com/skills/cfp-review-scorer
---

# CFP Review Scorer

Pre-scores a pile of session submissions so the program committee spends its time on the close calls. Every score comes with a reason tied to the rubric anchors, every flag says why, and the shortlist respects your track slots. Humans make the final decisions; this skill makes them faster and more consistent.

## Intake

If the user has an `event-context.md` file, read it first and don't ask for anything it already answers (audience, sponsor list, themes).

Ask for whatever is missing in ONE message:

1. The submissions export (CSV or XLSX from your CFP or abstract management tool, a form export or a pasted table).
2. Tracks or themes, total session slots and slots per track, formats and lengths.
3. The rubric: use theirs if they have one, or the default in [references/scoring-rubric.md](references/scoring-rubric.md). Any criterion weights.
4. The audience: roles, experience level, what they come for.
5. Sponsor list and rules for vendor speakers (e.g. vendors only via sponsor slots, vendors must co-present with a customer).
6. Balance goals: max sessions per company, share of first-time speakers, level mix, any self-reported diversity goals.
7. Reviewers: how many, their names or initials, and known conflicts (their own company, close partners).

## Data sources (connectors optional)

- If a Google Sheets connector is available, read the submissions sheet and offer to write scores to a new tab. Never overwrite the source tab.
- Otherwise, work from an uploaded CSV or XLSX or a pasted table and return Markdown tables plus downloadable CSVs.
- Never require a paid service. Don't fetch speakers' websites or social profiles.

Sample file for testing: [assets/sample-cfp-submissions.csv](assets/sample-cfp-submissions.csv).

## Method

1. **Load and count.** Report rows in. Map columns to: submission ID, title, abstract, outcomes, format, track, level, speaker name, title, company, email, co-presenters, prior speaking, sponsor status. Ask only if a critical column (title, abstract, speaker) can't be mapped.
2. **Clean and flag before scoring.**
   - `Duplicate`: same or near-identical title or abstract from the same speaker or company. Keep the most complete or most recent, list the others.
   - `Multiple from speaker`: one person on several submissions. Fine, but only accept one unless the user allows more.
   - `Missing info`: no abstract, abstract under about 40 words, no outcomes when required, no speaker details.
   - `Over limit`: above the CFP's stated word limit.
   - `Vendor pitch`: product names, feature lists, "our platform", demo promised, or a vendor speaker with no customer and no sponsor slot. Quote the phrase that triggered it.
   - `Vendor not sponsor`: speaker's company sells to this audience and isn't on the sponsor list. Flag for the committee, don't auto-decline.
   - `Conflict`: a reviewer works at or partners with the speaker's company.
3. **Score each submission** on each criterion, 1 to 5, using the anchors. Write one short reason per criterion that points at the text ("Outcomes are specific: build a lead tier model"). Score only what's written. Don't reward famous companies or penalize unknown speakers beyond the credibility criterion.
4. **Compute the weighted total** (0 to 100): sum of (score / 5 x weight). Default weights in the rubric reference. If code execution is available, run `python3 scripts/cfp_shortlist.py` (see below) for totals, ties and the shortlist; otherwise do it in-chat and show the math for the top and borderline rows.
5. **Build the shortlist.** Fill each track's slots in score order, applying caps (max per company, one per speaker), then fill the waitlist (default 25% of slots) in score order. Note where balance goals changed the order and why. Ties: use the rubric's tiebreak question.
6. **Recommend.** Each submission gets `Accept`, `Waitlist`, `Decline` or `Committee review` (flags that need a human: vendor not sponsor, conflict, borderline score within 3 points of the cut line).
7. **Prepare reviewer packets.** Assign each `Committee review` and top-scoring submission to 3 reviewers (or as many as the user has), avoiding conflicts and balancing load. Use the packet format in [references/reviewer-packet.md](references/reviewer-packet.md).
8. **Draft notes.** Short, kind decline and waitlist notes per reason group (fit, capacity, vendor pitch), plus an acceptance note template. Drafts only.

### Script

`scripts/cfp_shortlist.py` takes a scored CSV and the slot plan and returns totals, duplicate flags, the shortlist and the waitlist. Python standard library only.

```
python3 scripts/cfp_shortlist.py scored.csv --slots "Demand Gen=4,Field Events=3,Ops and Data=3" --max-per-company 1 --waitlist-share 0.25 --out shortlist.csv
```

Scored CSV columns: `submission_id,title,track,speaker_email,company,relevance,originality,takeaways,credibility,clarity` (extra columns are kept). Several rows per submission (one per reviewer) are averaged. The script adds `total`, `rank_in_track`, `recommendation` (Accept, Waitlist, Decline), `borderline` and `flags` (duplicates, caps, tracks with no slots). You then move borderline rows and rows with human-judgment flags (vendor not sponsor, conflict) to `Committee review`. If code execution isn't available, follow the same steps in-chat.

Domain rules that matter:
- Anchored scales keep reviewers consistent. Every 1 and 5 needs a reason.
- At least 3 reviewers per final decision is a common standard; the pre-score is one input, not a reviewer.
- Never infer gender, ethnicity or other protected traits from names or photos. Use self-reported fields only if the CFP collected them and the user asks.
- Blind review: if the user wants it, hide speaker name and company in packets and score credibility separately.
- Vendor sessions aren't banned; they're judged on whether a competitor's customer would learn something.

## Output templates

**1. Intake summary:** rows in, rows scored, rows flagged by flag type, rows out (must equal rows in).

**2. Scored table** (also as CSV):

| ID | Title | Track | Format | Speaker, company | Relevance | Originality | Takeaways | Credibility | Clarity | Total | Flags | Recommendation | Reasons |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

**3. Shortlist by track:** slots, accepted, waitlist, and the cut-line score per track.

**4. Balance check:**

| Measure | Goal | Shortlist | Note |
|---|---|---|---|

(level mix, sessions per company, first-time speakers, customer vs vendor speakers, formats)

**5. Committee review list**, **6. Reviewer packets**, **7. Decline, waitlist and accept note drafts**.

## Validation checklist

- [ ] Rows in equals rows out; every row has a recommendation.
- [ ] Every flag quotes or cites what triggered it.
- [ ] Every 1 and every 5 has a reason that points at the submission text.
- [ ] Totals recompute correctly from scores and weights for a spot check of 3 rows.
- [ ] No track exceeds its slots; caps per company and per speaker hold.
- [ ] Borderline rows (within 3 points of the cut line) are in Committee review, not silently declined.
- [ ] No protected traits inferred.
- [ ] House style in notes: contractions, plain words, no dashes as punctuation.

## Approval gates

- The skill recommends; the committee decides. Label the output "Pre-score for committee review".
- Never send acceptance, waitlist or decline emails, update a CFP tool or write to a sheet without showing the content and getting a clear yes.

## Privacy

- Submissions contain personal data. Keep the file in the user's tools and don't repeat speaker emails in summaries beyond what's needed.
- Reviewer packets should include only what reviewers need. For blind review, remove names, companies and emails.
- Decline notes shouldn't share scores of other submissions.

## Related skills

session-abstract-writer (the speaker's side), agenda-builder (scheduling accepted sessions), speaker-management (onboarding accepted speakers).

FAQ

How do you score call-for-papers submissions consistently?
Use an anchored rubric: a 1 to 5 scale where 1, 3 and 5 are described in words for each criterion, plus weights and a tiebreak question. Ask reviewers to justify every 1 and 5. The skill ships a default rubric for relevance, originality, takeaways, clarity and credibility that you can replace with your own.
Can AI pick our conference sessions?
It shouldn't make the final call. This skill pre-scores and flags submissions with reasons so your committee can spend time on the close calls. Borderline rows and judgment calls like vendors who aren't sponsors go to committee review, and output is labeled as a pre-score.
How does the skill spot product pitches?
It looks for product or company names as the subject, feature lists, promised demos, phrases like 'our platform' and outcomes that need the speaker's product. It quotes the phrase that triggered the flag. Vendor sessions aren't banned; they're judged on whether a competitor's customer would still learn something.
How do I balance tracks and speakers in a CFP shortlist?
Set slots per track and caps per company and per speaker before scoring. The skill fills each track in score order, skips anyone over a cap, builds a waitlist and shows a balance check for levels, formats, first-time speakers and customer versus vendor speakers. It never infers protected traits from names.