---
title: "Basic NLP"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Basic NLP}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(
  message = FALSE,
  warning = FALSE,
  comment = "#>"
)
```

Three `nlp_*` functions form the core processing pipeline: `nlp_split_sentences()` segments text into sentence rows; `nlp_tokenize_text()` tokenizes each sentence into a named list with optional character spans; `nlp_cast_tokens()` flattens that list into a long-format data frame ready for counting, filtering, or indexing. This vignette walks through each step on a Wikipedia corpus, then collapses them into a single pipe.

Load packages.

```{r libs}
library(textpress)
library(dplyr)
library(DT)
```

## Corpus

Fetch Wikipedia articles on the Strauss-Howe generational theory and scrape the article bodies.

```{r corpus}
wiki_urls <- textpress::fetch_wiki_urls("Strauss Howe generational theory", limit = 5)

wiki_text_list <- textpress::read_urls(wiki_urls, exclude_wiki_refs = TRUE)

corpus <- wiki_text_list$text
nrow(corpus)
```



## Sentence split

`nlp_split_sentences()` segments each text into individual sentences, adding `sentence_id` and `start`/`end` character offsets. Known abbreviations (*Mr., U.S., vs.*) and single-letter initials are protected from false boundary detection via a built-in `abbreviations` vector, which can be extended as needed.

```{r split-sentences}
web_ss <- corpus |>
  textpress::nlp_split_sentences(by = c("doc_id", "node_id"))

web_ss |>
  slice(1:5) |>
  mutate(text = {
    words <- strsplit(text, "\\s+")
    sapply(words, function(w) paste(paste(w[seq_len(min(15, length(w)))], 
                                          collapse = " "), "..."))
  }) |>
  DT::datatable(rownames = FALSE)
```

## Tokenize

`nlp_tokenize_text()` tokenizes each sentence into a named list. With `include_spans = TRUE` the result carries two parallel lists -- `tokens` and `spans` -- keyed by a composite `uid` built from the `by` columns.

```{r tokenize}
tokens <- web_ss |>
  textpress::nlp_tokenize_text(
    by = c("doc_id", "node_id", "sentence_id"),
    method = 'biber',
    include_spans = TRUE
  )

# tokens list: first unit
head(tokens$tokens[[1]])

# spans matrix: matching start/end positions
head(tokens$spans[[1]])
```

## Cast to data frame

`nlp_cast_tokens()` flattens the token list into a long-format data table: one row per token, with `id`, `token`, `start`, and `end`.

```{r cast}
token_df <- tokens |> textpress::nlp_cast_tokens()

token_df |>
  slice(1:20) |>
  DT::datatable(rownames = FALSE)
```



## Full pipeline

All three steps as a single pipe from the abstract corpus.

```{r pipeline}
corpus |>
  textpress::nlp_split_sentences(by = c("doc_id", "node_id")) |>
  textpress::nlp_tokenize_text(
    by            = c("doc_id", "node_id", "sentence_id"),
    include_spans = TRUE
  ) |>
  textpress::nlp_cast_tokens()
```

## Summary

`nlp_split_sentences()` → `nlp_tokenize_text()` → `nlp_cast_tokens()` is the standard processing path from raw text to a token-level data frame. The `start`/`end` spans thread through each step, keeping token positions recoverable relative to the source text.