---
title: "Web data"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Web data}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(
  eval = identical(Sys.getenv("TEXTPRESS_BUILD_LIVE_VIGNETTES"), "true"),
  message = FALSE,
  warning = FALSE,
  comment = "#>"
)
```

`textpress` is organized around four actions -- fetch, read, process, search. This vignette covers the first two. `fetch_rss()` retrieves recent entries from publisher-supported RSS and Atom feeds; `read_urls()` scrapes their content into a node-level data frame while retaining that provenance. Together they turn selected news sources into an analysis-ready corpus in a few lines.

## Select, fetch, and read

Filter the bundled `rss_politics` catalog directly, use `fetch_rss()` to retrieve recent entries from those publishers, then filter the returned metadata locally before `read_urls()` scrapes and parses the selected articles. The filtering below is ordinary R filtering, not a query sent to a search service.

```{r web-urls, message=FALSE, warning=FALSE}
library(textpress)
library(dplyr)
library(DT)

feeds <- textpress::rss_politics |>
  filter(category %in% c("polling", "public_opinion"))

web_urls <- textpress::fetch_rss(feeds$url) |>
  filter(grepl(
    "immigration|poll|survey",
    paste(title, description),
    ignore.case = TRUE
  ))
```



Scrape and parse the URLs returned above into a node-level data frame with `$text` and `$meta` components.

```{r web-text}
web_text_list <- web_urls |>
  textpress::read_urls(cores = 4)
```

Build a text snippet from the first 15 words of each article, join to metadata, and display as an interactive table.

```{r}
snippets <- web_text_list$text |>
  group_by(doc_id) |>
  summarise(text = {
    words <- unlist(strsplit(paste(text, collapse = " "), "\\s+"))
    paste(paste(words[seq_len(min(15, length(words)))], collapse = " "), "...")
  }, .groups = "drop")

metas_dt <- web_text_list$meta |>
  filter(!is.na(h1_title) & nzchar(trimws(h1_title))) |>
  left_join(snippets, by = "doc_id") |>
  arrange(desc(date)) |>
  mutate(
    title_link = paste0(
      '<a href="', url, '" target="_blank">', h1_title, '</a>'
    )
  )

DT::datatable(
  metas_dt |> select(date, source, title_link, text),
  options = list(columnDefs = list(
    list(targets = 2, orderable = FALSE)
  )),
  escape   = FALSE,
  rownames = FALSE
)
```

## Summary

`fetch_rss()` and `read_urls()` are source-to-corpus entry points for a `textpress` pipeline. `fetch_rss()` returns publisher-provided article URLs with feed metadata; `read_urls()` scrapes and parses them into `$text` (one row per node) and `$meta` (one row per URL, including discovery provenance).