Package 'textpress' reference manual

Title:	A Lightweight and Versatile NLP Toolkit
Description:	An R toolkit for building text corpora and searching them. Collect URLs and provenance from RSS and Atom feeds, Wikipedia and its citations, or supply existing URLs. No custom object classes, just plain data frames from start to finish. Covers the full arc from URL to retrieved passage through a consistent four-step API: Fetch, Read, Process, Search. Traditional tools (KWIC, BM25, dictionary matching) sit alongside modern ones (semantic search, LLM-ready chunking), all compatible with the native R pipe ('\|>').
Authors:	Jason Timm [aut, cre]
Maintainer:	Jason Timm <[email protected]>
License:	MIT + file LICENSE
Version:	1.1.2
Built:	2026-09-28 12:50:52 UTC
Source:	https://github.com/jaytimm/textpress

Common abbreviations for NLP

Description

Common abbreviations for NLP (e.g. sentence splitting). Named list; used by nlp_split_sentences.

Usage

abbreviations
abbreviations

Format

A named list with the following components:

abbreviations: A character vector of common abbreviations, including titles, months, and standard abbreviations.

Source

Internally compiled linguistic resource.

Demo dictionary of generation-name variants for NER

Description

A small dictionary of generational cohort terms (Greatest, Silent, Boomers, Gen X, Millennials, Gen Z, Alpha, etc.) and spelling/variant forms, for use with search_dict. Built in-package (no data()).

Usage

dict_generations
dict_generations

Format

A data frame with columns variant (surface form to match), TermName (standardized label), is_cusp (logical), start and end (birth year range; Pew definitions where applicable, see https://github.com/jaytimm/AmericanGenerations/blob/main/data/pew-generations.csv).

Examples

head(dict_generations)
# use as term list: search_dict(corpus, by = "doc_id", terms = dict_generations$variant)
head(dict_generations)
# use as term list: search_dict(corpus, by = "doc_id", terms = dict_generations$variant)

Demo dictionary of political / partisan term variants for NER

Description

A small dictionary of political party and ideology terms (Democrat, Republican, MAGA, Liberal, Conservative, Christian Nationalist, White Supremacist, etc.) and spelling/variant forms, for use with search_dict. Built in-package (no data()).

Usage

dict_political
dict_political

Format

A data frame with columns variant (surface form to match) and TermName (standardized label).

Examples

head(dict_political)
# search_dict(corpus, by = "doc_id", terms = dict_political$variant)
head(dict_political)
# search_dict(corpus, by = "doc_id", terms = dict_political$variant)

Fetch recent articles from RSS and Atom feeds

Description

Fetches RSS or Atom feeds and returns one row per article entry. The result includes feed and article metadata and can be passed directly to read_urls through its url column.

Usage

fetch_rss(feed_url, cores = 1)
fetch_rss(feed_url, cores = 1)

Arguments

feed_url

Character vector of RSS or Atom feed URLs.

cores

Number of cores for parallel feed requests (default 1).

Value

A data.table with common discovery columns doc_id, url, title, published_at, source, language, and country, followed by RSS-specific feed and article metadata.

Examples

## Not run: 
feeds <- subset(rss_politics, category == "polling")
items <- fetch_rss(feeds$url, cores = 4)
corpus <- read_urls(items)

## End(Not run)
## Not run: 
feeds <- subset(rss_politics, category == "polling")
items <- fetch_rss(feeds$url, cores = 4)
corpus <- read_urls(items)

## End(Not run)

Fetch external citation URLs from Wikipedia article(s)

Description

Wikipedia. Extracts external citation URLs from the References section of one or more Wikipedia article URLs. Use read_urls to scrape content from those URLs.

Usage

fetch_wiki_refs(url, n = NULL)
fetch_wiki_refs(url, n = NULL)

Arguments

url

Character vector of full Wikipedia article URLs (e.g. from fetch_wiki_urls).

n

Maximum number of citation URLs to return per source page. Default NULL returns all; use a number (e.g. 10) to limit.

Value

A data.table with columns source_url, ref_id, and ref_url, one row per citation across all input URLs. Use split(x, x$source_url) to group by article.

Examples

## Not run: 
wiki_urls <- fetch_wiki_urls("January 6 Capitol attack")
refs_dt <- fetch_wiki_refs(wiki_urls)
articles <- read_urls(refs_dt$ref_url)

## End(Not run)
## Not run: 
wiki_urls <- fetch_wiki_urls("January 6 Capitol attack")
refs_dt <- fetch_wiki_refs(wiki_urls)
articles <- read_urls(refs_dt$ref_url)

## End(Not run)

Fetch Wikipedia page URLs by search query

Description

Wikipedia. Uses the MediaWiki API to get Wikipedia article URLs matching a search phrase. Does not search your local corpus. Use read_urls to get article content from these URLs.

Usage

fetch_wiki_urls(query, limit = 10)
fetch_wiki_urls(query, limit = 10)

Arguments

query

Search phrase (e.g. "117th Congress").

limit

Number of page URLs to return (default 10).

Value

Character vector of full Wikipedia article URLs.

Examples

## Not run: 
wiki_urls <- fetch_wiki_urls("January 6 Capitol attack")
corpus <- read_urls(wiki_urls[1])

## End(Not run)
## Not run: 
wiki_urls <- fetch_wiki_urls("January 6 Capitol attack")
corpus <- read_urls(wiki_urls[1])

## End(Not run)

Convert token list to data frame

Description

Convert the token list returned by nlp_tokenize_text into a data frame (long format), with identifiers and optional spans.

Usage

nlp_cast_tokens(tok)
nlp_cast_tokens(tok)

Arguments

tok

List with at least a tokens element (and optionally spans), e.g. output of nlp_tokenize_text(..., include_spans = TRUE).

Value

Data frame with columns for unit id, token, and optionally start/end spans.

Examples

tok <- list(
  tokens = list(
    "1.1" = c("Hello", "world", "."),
    "1.2" = c("This", "is", "an", "example", "."),
    "2.1" = c("This", "is", "a", "party", "!")
  )
)
dtm <- nlp_cast_tokens(tok)

tok <- list(
  tokens = list(
    "1.1" = c("Hello", "world", "."),
    "1.2" = c("This", "is", "an", "example", "."),
    "2.1" = c("This", "is", "a", "party", "!")
  )
)
dtm <- nlp_cast_tokens(tok)

Build a BM25 index for ranked keyword search

Description

Build a weighted BM25 index for ranked keyword search. Creates a searchable index from a named list of token vectors. The unit-id column name is taken from attr(tokens, "id_col") when present (e.g. from nlp_tokenize_text), else "uid".

Usage

nlp_index_tokens(tokens, k1 = 1.2, b = 0.75, stem = FALSE)
nlp_index_tokens(tokens, k1 = 1.2, b = 0.75, stem = FALSE)

Arguments

tokens

Named list of character vectors (e.g. from nlp_tokenize_text).

k1

BM25 saturation parameter (default 1.2).

b

BM25 length normalization (default 0.75).

stem

Logical. If TRUE, stem tokens (default FALSE).

Value

Data.table with unit-id column, token, score; attr(., "id_col") set for search_index.

Roll units into fixed-size chunks with optional context

Description

Roll units (e.g. sentences) into fixed-size chunks with optional context (RAG-style). Groups consecutive rows at the finest by level into chunks and optionally adds surrounding context.

Usage

nlp_roll_chunks(corpus, by, chunk_size, context_size, id_col = "uid")
nlp_roll_chunks(corpus, by, chunk_size, context_size, id_col = "uid")

Arguments

corpus

Data frame or data.table with a text column and the identifier columns specified in by.

by

Character vector of identifier columns that define the text unit (e.g. doc_id or c("url", "node_id")). The last column is the level rolled into chunks (e.g. sentences).

chunk_size

Integer. Number of units per chunk.

context_size

Integer. Number of units of context around each chunk.

id_col

Character. Name of the column holding the unique chunk id (default "uid").

Value

Data.table with id_col (pasted grouping + chunk index), grouping columns from by, and text (chunk plus context). Unique on by[1] and text.

Examples

corpus <- data.frame(doc_id = c('1', '1', '2'),
                    sentence_id = c('1', '2', '1'),
                    text = c("Hello world.",
                             "This is an example.",
                             "This is a party!"))
chunks <- nlp_roll_chunks(corpus, by = c('doc_id', 'sentence_id'),
                          chunk_size = 2, context_size = 1)
corpus <- data.frame(doc_id = c('1', '1', '2'),
                    sentence_id = c('1', '2', '1'),
                    text = c("Hello world.",
                             "This is an example.",
                             "This is a party!"))
chunks <- nlp_roll_chunks(corpus, by = c('doc_id', 'sentence_id'),
                          chunk_size = 2, context_size = 1)

Split text into paragraphs

Description

Break documents into structural blocks (paragraphs). Splits text from the text column by a paragraph delimiter.

Usage

nlp_split_paragraphs(corpus, by = c("doc_id"), paragraph_delim = "\\n+")
nlp_split_paragraphs(corpus, by = c("doc_id"), paragraph_delim = "\\n+")

Arguments

corpus

Data frame or data.table with a text column and the identifier columns specified in by.

by

Character vector of identifier columns that define the text unit (e.g. doc_id or c("url", "node_id")). Default c("doc_id").

paragraph_delim

Regular expression used to split text into paragraphs (default "\\n+").

Value

Data.table with the by columns, paragraph_id, and text. One row per paragraph.

Examples

corpus <- data.frame(doc_id = c('1', '2'),
                     text = c("Hello world.\n\nMind your business!",
                              "This is an example.n\nThis is a party!"))
paragraphs <- nlp_split_paragraphs(corpus)

corpus <- data.frame(doc_id = c('1', '2'),
                     text = c("Hello world.\n\nMind your business!",
                              "This is an example.n\nThis is a party!"))
paragraphs <- nlp_split_paragraphs(corpus)

Split text into sentences

Description

Refine blocks into individual sentences. Splits text into sentences with accurate start/end offsets; handles abbreviations (Wikipedia and web optimized).

Usage

nlp_split_sentences(
  corpus,
  by = c("doc_id"),
  abbreviations = textpress::abbreviations
)
nlp_split_sentences(
  corpus,
  by = c("doc_id"),
  abbreviations = textpress::abbreviations
)

Arguments

corpus

Data frame or data.table with a text column and the identifier columns specified in by.

by

Character vector of identifier columns that define the text unit (e.g. doc_id or c("url", "node_id")). Default c("doc_id").

abbreviations

Character vector of abbreviations to protect (default textpress::abbreviations).

Value

Data.table with by columns, sentence_id, text, start, end.

Tokenize text into a clean token stream

Description

Normalize text into a clean token stream. Tokenizes corpus text, preserving structure (capitalization, punctuation). The last column in by determines the tokenization unit.

Usage

nlp_tokenize_text(
  corpus,
  by = c("doc_id", "paragraph_id", "sentence_id"),
  id_col = "uid",
  include_spans = TRUE,
  method = "word"
)
nlp_tokenize_text(
  corpus,
  by = c("doc_id", "paragraph_id", "sentence_id"),
  id_col = "uid",
  include_spans = TRUE,
  method = "word"
)

Arguments

corpus

Data frame or data.table with a text column and the identifier columns specified in by.

by

Character vector of identifier columns that define the text unit (e.g. doc_id or c("url", "node_id")). Default c("doc_id", "paragraph_id", "sentence_id"). The last column is the finest granularity.

id_col

Character. Name of the column (and list names) used for the unit id (default "uid").

include_spans

Logical. Include start/end character spans for each token (default TRUE).

method

Character. "word" or "biber".

Value

Named list of tokens; or list of tokens and spans if include_spans = TRUE.

Examples

corpus <- data.frame(doc_id = c('1', '1', '2'),
                    sentence_id = c('1', '2', '1'),
                    text = c("Hello world.",
                             "This is an example.",
                             "This is a party!"))
tokens <- nlp_tokenize_text(corpus, by = c('doc_id', 'sentence_id'))
corpus <- data.frame(doc_id = c('1', '1', '2'),
                    sentence_id = c('1', '2', '1'),
                    text = c("Hello world.",
                             "This is an example.",
                             "This is a party!"))
tokens <- nlp_tokenize_text(corpus, by = c('doc_id', 'sentence_id'))

Read content from URLs

Description

Input: character vector of URLs. Output: structured data frame (one row per node: headings, paragraphs, lists). Like read_csv or read_html: bring an external resource into R. Follows fetch_rss or fetch_wiki_urls in the pipeline—fetch gets locations, read gets text. Wikipedia uses high-fidelity selectors; use parent_heading to see which section each node belongs to. External links and empty text rows are omitted; optionally exclude References/See also/Bibliography/Sources sections for wiki URLs.

Usage

read_urls(
  x,
  cores = 1,
  detect_boilerplate = TRUE,
  remove_boilerplate = TRUE,
  exclude_wiki_refs = TRUE
)
read_urls(
  x,
  cores = 1,
  detect_boilerplate = TRUE,
  remove_boilerplate = TRUE,
  exclude_wiki_refs = TRUE
)

Arguments

x

Character vector of URLs, or a data frame containing a url column. Data-frame metadata is preserved in the returned meta table; an existing doc_id is honored.

cores

Number of cores for parallel requests (default 1).

detect_boilerplate

Logical. Detect boilerplate (e.g. sign-up, related links).

remove_boilerplate

Logical. If detect_boilerplate is TRUE, remove boilerplate rows; if FALSE, keep them and add is_boilerplate.

exclude_wiki_refs

Logical. For Wikipedia URLs only, drop nodes whose parent_heading is References, See also, Bibliography, or Sources. Default TRUE.

Value

A list with text (node-level data: doc_id, url, node_id, parent_heading, text, and optionally type, is_boilerplate) and meta (one row per URL with doc_id, url, scraped fields, and any metadata supplied in x). For character input, doc_id is an integer key in first-appearance order.

Examples

## Not run: 
feeds <- subset(rss_politics, category == "polling")
articles <- fetch_rss(feeds$url)
out <- read_urls(articles, cores = 1)
nodes <- out$text
meta <- out$meta

## End(Not run)
## Not run: 
feeds <- subset(rss_politics, category == "polling")
articles <- fetch_rss(feeds$url)
out <- read_urls(articles, cores = 1)
nodes <- out$text
meta <- out$meta

## End(Not run)

Curated US local-news RSS feeds

Description

A pinned snapshot of the canonical Local Rags RSS output. Each row identifies one unique US local-news feed with publisher, feed, verification, Census division, county, and rural-urban geography. Access it directly as textpress::rss_local_rags. Full discovery and 3DLNews provenance remain in the source project.

Usage

rss_local_rags
rss_local_rags

Format

A data frame with one row per feed and 11 variables: source, feed, url, verified_at, latest_item_at, census_division, state_abbr, county, five-character fips county code, rucc_2023, and metro_status.

Source

Local Rags RSS, https://github.com/jaytimm/local-rags-rss, derived from 3DLNews, https://github.com/wm-newslab/3DLNews.

Examples

head(textpress::rss_local_rags)
head(textpress::rss_local_rags)

Curated US politics RSS feeds

Description

A verified collection of non-government RSS and Atom feeds covering US politics. The collection includes general political news, Congress, elections, polling, public opinion, and state politics. Feed activity was checked on the date recorded in verified_at; publisher availability can change after that date. Access it directly as textpress::rss_politics.

Usage

rss_politics
rss_politics

Format

A data frame with one row per feed and seven variables:

source: Publisher or organization name.
feed: Publisher-supplied or descriptive feed name.
category: Primary subject category assigned by textpress.
source_type: Type of publisher or organization.
url: RSS or Atom feed URL.
verified_at: UTC date and time when the feed was last checked.
latest_item_at: UTC publication date and time of the newest item observed during verification.

Source

Publisher-provided RSS and Atom endpoints.

Examples

head(textpress::rss_politics)
head(textpress::rss_politics)

Exact phrase / MWE matcher

Description

Exact phrase or multi-word expression (MWE) matcher; no partial-match risk. Tokenizes corpus, builds n-grams, and exact-joins against terms. Word boundaries respected. N-gram range is set from the min and max word count of terms. Good for deterministic entity extraction (e.g. before an LLM call).

Usage

search_dict(corpus, by = c("doc_id"), terms)
search_dict(corpus, by = c("doc_id"), terms)

Arguments

corpus

Data frame or data.table with a text column and the identifier columns specified in by.

by

Character vector of identifier columns that define the text unit (e.g. doc_id or c("url", "node_id")). Default c("doc_id").

terms

Character vector of terms or phrases to match exactly. N-gram range derived from word counts of terms.

Value

Data.table with id, start, end, n, ngram, term.

Examples

corpus <- data.frame(doc_id = "1", text = "Gen Z and Millennials use social media.")
search_dict(corpus, by = "doc_id", terms = c("Gen Z", "Millennials", "social media"))
corpus <- data.frame(doc_id = "1", text = "Gen Z and Millennials use social media.")
search_dict(corpus, by = "doc_id", terms = c("Gen Z", "Millennials", "social media"))

Search the BM25 index

Description

BM25 ranked retrieval. Search the index produced by nlp_index_tokens with a keyword query. The unit-id column in results is taken from attr(index, "id_col") when present, else "uid".

Usage

search_index(index, query, n = 10, stem = FALSE)
search_index(index, query, n = 10, stem = FALSE)

Arguments

index

Object created by nlp_index_tokens.

query

Character string (keywords).

n

Number of results to return (default 10).

stem

Logical; must match the setting used during indexing (default FALSE).

Value

Data.table with columns query, method (“bm25”), score (3 significant figures), and the unit-id column (e.g. uid), ranked by score.

Search corpus by regex

Description

Search corpus by regex. Specific strings/patterns; good for KWIC-style results. Returns matches with optional highlighting.

Usage

search_regex(corpus, query, by = c("doc_id"), highlight = c("<b>", "</b>"))
search_regex(corpus, query, by = c("doc_id"), highlight = c("<b>", "</b>"))

Arguments

corpus

Data frame or data.table with a text column and the identifier columns specified in by.

query

Search pattern (regex).

by

Character vector of identifier columns that define the text unit (e.g. doc_id or c("url", "node_id")). Default c("doc_id").

highlight

Length-two character vector for wrapping matches (default c("<b>", "</b>")).

Value

Data.table with id, by columns, text, start, end, pattern.

Semantic search by cosine similarity

Description

Semantic search by cosine similarity. Returns top-n matches from an embedding matrix for one or more query vectors. Subject-first: embeddings (haystack) then query (needle). Pipe-friendly.

Usage

search_vector(embeddings, query, n = 10)
search_vector(embeddings, query, n = 10)

Arguments

embeddings

Numeric matrix of embeddings; rows are searchable units (row names used as identifiers).

query

Row name in embeddings, a numeric vector (single query), or a numeric matrix (multiple queries).

n

Number of results to return per query (default 10).

Value

Data frame with columns query, method (“cosine”), score (3 significant figures), and the unit-id column (e.g. uid). For multiple queries, a list of such data frames.

Fetch embeddings from a Hugging Face inference endpoint

Description

Builds a numeric matrix of embeddings for each text unit. Row names come from by (data frame) or from names(corpus) / corpus (character vector). Use the result with search_vector for semantic search.

Usage

util_fetch_embeddings(
  corpus,
  by = NULL,
  api_token,
  api_url = "https://router.huggingface.co/hf-inference/models/BAAI/bge-small-en-v1.5"
)
util_fetch_embeddings(
  corpus,
  by = NULL,
  api_token,
  api_url = "https://router.huggingface.co/hf-inference/models/BAAI/bge-small-en-v1.5"
)

Arguments

corpus

A data frame with text and by columns, or a character vector of texts. If a named character vector, names become row names; if unnamed, the strings themselves are used as row names.

by

Character vector of identifier columns; required when corpus is a data frame (row names), ignored when corpus is a character vector.

api_token

Hugging Face API token.

api_url

Inference endpoint URL (default BAAI/bge-small-en-v1.5).

Value

Numeric matrix with row names (unit ids).

Package 'textpress'

Help Index

Common abbreviations for NLP

Description

Usage

Format

Source

Demo dictionary of generation-name variants for NER

Description

Usage

Format

Examples

Demo dictionary of political / partisan term variants for NER

Description

Usage

Format

Examples

Fetch recent articles from RSS and Atom feeds

Description

Usage

Arguments

Value

Examples

Fetch external citation URLs from Wikipedia article(s)

Description

Usage

Arguments

Value

Examples

Fetch Wikipedia page URLs by search query

Description

Usage

Arguments

Value

Examples

Convert token list to data frame

Description

Usage

Arguments

Value

Examples

Build a BM25 index for ranked keyword search

Description

Usage

Arguments

Value

Roll units into fixed-size chunks with optional context

Description

Usage

Arguments

Value

Examples

Split text into paragraphs

Description

Usage

Arguments

Value

Examples

Split text into sentences

Description

Usage

Arguments

Value

Tokenize text into a clean token stream

Description

Usage

Arguments

Value

Examples

Read content from URLs

Description

Usage

Arguments

Value

Examples

Curated US local-news RSS feeds

Description

Usage

Format

Source