Define the intended task and the evidence you need
Begin with a local dataset specification: the model task, unit of annotation, intended source community and inclusion rules. For a small first exercise, inspect comments received from MachineLearning rather than promising a dataset representing all machine learning discussions. The archive is independent and historical, not official Reddit or live data. It does not certify provenance, labels, representativeness or fitness for a model.
Consider the linguistic diversity within your chosen subreddits. Some communities have distinct slang, abbreviations, or grammatical quirks that can skew language models if not properly accounted for. Document the patterns you actually observe in the received sample; do not assume the community name proves language, expertise or uniform quality. Any cleaning rules and their effect on inclusion belong in your own specification.
Choose which available fields you need to retain and what to do when they are absent. Record route, filters, retrieval time and actual response files alongside the sample. Metadata can be missing and references unresolved. This is documentation you create from observed evidence, not a guaranteed source certificate supplied with every record. Save the received material because another remote retrieval need not return the same input.
Retrieve a bounded page and keep source records separate from labels
Start with https://api.redditdatasets.com/v1/comments?subreddit=MachineLearning&limit=3, the homepage example. Use X-API-Key: <key> or Authorization: Bearer <key> on the server. The comments route restricts results to comments, subreddit chooses the community and limit asks for at most three records. The Get API key page explains registration before payment and saving the once-shown key outside public annotation files.
Content routes accept q, subreddit, author, kind, from, to, min_score, limit and page. kind uses post or comment; specialized routes retain their type restriction. Dates use YYYY-MM-DD UTC on created_utc and fully include the final to day. min_score filters a numeric threshold without sorting, verifying expertise or removing spam. These filters perform retrieval, not random sampling or automatic balancing of your dataset.
Validate data, then copy selected records into your own annotation queue while preserving a local mapping to their received source. Keep labels and rejection reasons separate from original text. Flat records with present parent_id or link_id can be grouped locally, without complete discussion context. There are no search filters for those identifiers. /v1/subreddits accepts prefix and limit for name discovery, not guaranteed topical coverage.
limit and page range from 1 to 100, with one page per request. Internal _doc order is not chronology, relevance, score or random order. Read total with relation: eq is an exact query total, gte a lower bound and unknown unreliable or null. A complete local queue of what you received does not prove a complete archive download. Save the boundaries encountered rather than presenting pagination as a bulk export.
Audit local splits and explain exclusions
Before assigning training and evaluation partitions, inspect duplicate identifiers and available conversation links among the records you received. You may choose to keep locally recognized related records in one partition to reduce a particular overlap risk. That is your own split rule, not an API feature or a guarantee that all duplicates, related conversations or earlier model-training exposure have been found.
Document which records entered annotation, which were excluded and why. Distinguish missing metadata from zero values, and your labels from the archived text. Archived score metadata is not an ability to cast votes and is not proof of correctness. Local sorting, deduplication and date partitioning can organize a sample without certifying completeness, lack of bias or identical repeated retrieval.
Every successful retrieval costs one credit, even empty or repeated. Invalid requests do not debit; upstream failures refund reservations. Client response loss may conceal a billed success, and local annotation or storage failure does not refund it. Handle 400 by correcting parameters, 401 by checking key and domain, 402 by checking balance and 429 with bounded delayed retries. Reprocessing a saved local response does not call the API.
Read the documentation for route constraints and pricing for $1 per 1000 successes and the $50 minimum for 50000 paid credits. The key page explains conditional trial eligibility and password-confirmed regeneration, which immediately invalidates the old key. Public /v1/status only confirms HTTP response and static scope, not dataset quality or search health. Respect applicable law and third-party rights when retaining, annotating or redistributing your sample.
