---
title: "Wikipedia data"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Wikipedia data}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(
  eval = identical(Sys.getenv("TEXTPRESS_BUILD_LIVE_VIGNETTES"), "true"),
  message = FALSE,
  warning = FALSE,
  comment = "#>"
)
```

Same fetch-then-read pipeline as the web-data vignette, but starting from a topic rather than a known set of publishers. `fetch_wiki_urls()` finds Wikipedia article URLs; `fetch_wiki_refs()` follows each article's References section to collect cited external URLs -- primary sources, reports, and news. `read_urls()` handles both kinds of URL in the same way.

## Expand via Wikipedia

**Start from a topic.** `fetch_wiki_urls(query, limit)` returns Wikipedia article URLs matching a search phrase. Use that as your seed set.

```{r fetch-wiki-urls, message=FALSE, warning=FALSE}
library(textpress)
library(dplyr)

wiki_urls <- textpress::fetch_wiki_urls("January 6 Capitol attack", limit = 5)
wiki_urls
```

**Follow References.** Articles cite external sources; those URLs are often high-value (primary sources, reports, news). `fetch_wiki_refs(url, n)` returns one flat `data.table` with `source_url` (the Wikipedia page) and `ref_url` (the cited link), whether `url` contains one Wikipedia page or several. Use `n` to cap references per page.

```{r fetch-wiki-refs}
refs <- textpress::fetch_wiki_refs(wiki_urls[1:3], n = 15)

refs |>
  slice(1:5) |>
  select(-source_url) |> 
  DT::datatable(rownames = F)
```

So: **wiki article URLs** + **URLs from their References sections** = an expanded URL list. You can then read both the articles and the cited pages.

## Ingest with read_urls()

Character vector of URLs → data frame (one row per node: headings, paragraphs, lists). Use `read_urls()` on any of the URLs you collected above.

**Wikipedia has special status in `read_urls()`.** When the URL is a Wikipedia page, the function uses Wikipedia’s main-content selector (`div.mw-parser-output`) and preserves section structure via `parent_heading`. Boilerplate detection is off for Wikipedia. **`exclude_wiki_refs = TRUE`** (default) drops nodes under References, See also, Bibliography, and Sources so you get article body only; set to `FALSE` to include those sections.

```{r read-urls}
# Article body only (no References / See also); wiki_read$text, wiki_read$meta
wiki_text_list <- textpress::read_urls(wiki_urls, exclude_wiki_refs = TRUE)

wiki_text_list$text |>
  select(-url) |>
  slice(1:5) |>
  mutate(text = {
    words <- strsplit(text, "\\s+")
    sapply(words, function(w) paste(paste(w[seq_len(min(15, length(w)))], collapse = " "), "..."))
  }) |>
  DT::datatable(rownames = F)
```





## Summary

`fetch_wiki_urls()` and `fetch_wiki_refs()` extend the same fetch-then-read pipeline to Wikipedia -- useful when you want a topic-seeded corpus that includes both article text and cited primary sources. `read_urls()` handles the result identically to any other URL list, returning `$text` and `$meta`.