i have been building a small tool for mapping recurring limitations, missing evaluations, open problems, and future-work claims across collections of academic papers.
the obvious version of this tool would ask an LLM to read the papers and generate a list of research gaps. that version would also be the easiest one to demo. give it a folder of PDFs, wait a few minutes, and show a clean list of promising directions.
i did not build that version.
Atlas starts with a narrower job: find what the paper authors actually said, preserve where they said it, and make every later interpretation point back to that evidence.
a generated research gap can sound convincing. a source statement can be checked.
the first Atlas pipeline was barely a pipeline. it read UTF-8 text files, detected headings such as Limitations, Future Work, and Threats to Validity, then ran explicit regex rules over paragraphs and sentences.
the rules looked for phrases academic writing already repeats:
a limitation of this work,we did not evaluate,our study is limited to,remains unclear,future work,more work is needed.
this works because academic limitations are not always subtle.
papers often announce them using formulaic language, especially in
dedicated sections. regex is weak at meaning, but it is good at
finding repeated surface forms. section context makes those forms
more useful: limited to means something different in a
Limitations section than it does in a dataset description.
the early version scored each candidate from two sources: the section heading and the matched rule categories. it filtered short fragments, deduplicated normalized text within each paper, and wrote JSONL artifacts plus a static HTML report.
no embeddings. no API calls. no generated summary. the interesting part was not the sophistication of the model. there was no model. the interesting part was that every candidate had a filename, section heading, start offset, end offset, and exact source text.
because a limitation statement and a research gap are not the same object.
if a paper says it did not evaluate robustness under domain shift, that is evidence about what the authors did not test. it is not yet proof that nobody has tested it, that the problem remains open, or that another study would be novel. answering those questions needs literature search, comparison across papers, and scientific judgment.
a language model can collapse those steps into one fluent sentence. the sentence may be useful. it may also quietly turn one paper's local limitation into a field-level novelty claim.
Atlas extracts stated research debt. it does not claim that the extracted item is unresolved, important, or novel across the literature.
this boundary made the tool less impressive in screenshots. it also made the output much easier to audit.
version 0.1.0 was extraction. the versions after that mostly added the workflow needed to treat extraction as evidence rather than an answer.
0.2.0 added pattern-level rule details and manual review. instead of
only saying that a candidate matched future_work, Atlas
recorded the stable pattern ID and exact substring that fired. a
reviewer could then label the span or reject it before it became
anything stronger.
later versions added promotion, deterministic map nodes, manifests, validation, safer edits, and release diagnostics. a map node is a group of reviewed evidence: for example, several papers reporting missing dialect evaluation can sit under one node without Atlas claiming that the issue remains unresolved across the literature. every added layer put more friction between raw regex output and a finished research claim.
Atlas sorts filenames, headings, candidates, rules, tags, and map nodes explicitly. IDs are derived from normalized inputs with truncated SHA-256. JSON uses stable ordering. extraction artifacts do not contain timestamps. the same corpus and version should produce the same evidence files.
this matters for a boring reason: if a candidate changes, i want to know whether the paper changed, the rule changed, the section boundary changed, or the runtime changed. stochastic extraction makes that comparison harder.
determinism does not make regex correct. it makes regex errors repeatable, inspectable, and measurable.
a false positive you can reproduce is easier to fix than a plausible answer you cannot reconstruct.
the original MVP only accepted pre-converted .txt
files. that kept the extraction logic simple, but pushed an
important problem onto the user. academic corpora arrive as PDFs.
Atlas 1.0.0 added local PDF ingestion through Poppler's
pdftotext. the converter runs with fixed reading-order
arguments, and Atlas preserves the exact converted text used for
every character offset. the output does not pretend an offset points
into PDF bytes or page coordinates. it points into a saved UTF-8
artifact that can be opened and checked.
the first real-corpus run was ugly. three papers produced 293 detected "sections." author names, affiliations, table rows, diagram labels, and wrapped prose looked enough like headings to the permissive detector. two-column layout also interleaved unrelated text into candidate spans.
this was useful. the failure was concrete. i could point to the converted text, the bad boundary, the rule hit, and the candidate ID. changing PDF extraction from layout-preserving output to reading order removed the worst cross-column spans. tightening heading rules reduced the same corpus from 293 detected sections to 126.
126 is still noisy. PDF text is not a semantic document tree, and a deterministic converter cannot recover structure the file does not encode. Atlas reports that limitation instead of hiding it behind a fluent summary.
the real papers exposed two different false-positive classes.
first, phrases such as future research can describe a
topic without stating the authors' future work. the early rule was
too broad. it now requires a more action-like context, such as
future research would or future studies should.
second, a paper about limitation generation contains the word limitation everywhere: prompts, labels, rubric definitions, model outputs, and dataset examples. substring scoring treated headings such as Limitation Subtype: Limited Datasets as if they were the paper's own Limitations section. during the final pre-release hardening for the public 1.0.0 tag, that shortcut was replaced with exact canonical heading scores, and references were removed from candidate scanning.
the corpus also exposed misses. explicit paragraphs beginning with Threats to construct validity and Threats to internal validity were not retained unless another rule happened to match. those phrases are now first-class evidence patterns.
none of these changes required a better prompt. they required a failure log, source spans, and a test that failed for the same reason the real paper failed.
Atlas 1.0.0 accepts English text and text-bearing PDFs. it scans detected headed sections, applies seven rule categories, scores and deduplicates candidates, preserves PDF conversion artifacts, and writes reviewable JSONL and HTML reports.
the full workflow goes from papers to sections, candidates, manual reviews, promoted debt records, and deterministic map nodes. it has 155 tests, release checks across Python 3.11, 3.12, and 3.13, and a clean-project smoke workflow that ends with validated artifacts.
it still has real limits:
1.0.0 does not mean the extractor is complete. it means the evidence contract is stable enough to test honestly.
keeping extraction deterministic is not an argument against machine learning. it is an argument about order.
semantic clustering could group reviewed statements that use different language. summarization could produce compact descriptions of evidence groups. retrieval could test whether a stated limitation recurs across papers or has been addressed elsewhere. an LLM could help a reviewer navigate a large map.
but those layers are easier to evaluate after the extraction layer has a measurable precision/recall story and a set of reviewed source statements. otherwise the semantic layer is judging its own generated premises.
Atlas started as a small regex experiment because academic papers often state their limitations more directly than we assume. it grew into a review workflow because finding a sentence is not the same as establishing a gap.
the deterministic version is less magical. it returns source statements, not research ideas. it leaves ambiguity visible. it makes the reviewer do work.
that is the point: the first useful map of research debt should show where every claim came from before it tries to tell us where research should go next.
source and release artifacts: github.com/voidwest/atlas.