Basango

I started Basango because following the war and security situation in eastern DRC meant checking many sources by hand. The project collects articles from selected Congolese media, puts them into one data model, removes duplicates, assigns topics, and makes the result searchable.

The hard part is not downloading pages. Sources use different publishing systems, metadata is uneven, and the same report may appear several times. Basango handles that work as a pipeline that can run continuously without requiring a large operating budget.

The problem

News is available. Finding a small set of relevant reports from sources worth checking still takes daily work.

The Democratic Republic of Congo has many online news outlets, social media accounts, and informal channels publishing political, economic, social, cultural, environmental, and security-related information. Some sources are reputable and consistent. Others are unreliable, politically biased, poorly structured, or difficult to evaluate.

Following a specific topic manually, such as the eastern DRC conflict, means moving between multiple news websites, Twitter/X feeds, and search results. Each day, the user has to filter what is relevant, compare sources, remove duplicates, identify noise, and decide which reports are credible enough to use.

Staying informed becomes a daily process of collecting and filtering information by hand.

RSS readers and feed aggregators solve part of this problem. RSS provides a standardized way for applications to consume website updates without manually checking each website. However, many Congolese media websites either do not provide usable RSS feeds, do not expose topic-level subscription models, or publish too broadly for focused monitoring.

Generic tools like Feedly aggregate sources, but they do not answer Basango's question:

What is happening now, from credible sources, about the topics I care about?

That turned Basango from a simple news app idea into an information retrieval and curation system.

In Basango, information retrieval means collecting articles from selected sources, normalizing them into a unified structure, classifying them by topic, and making them searchable, filterable, and eventually recommendable.

Product direction

The first brief was short:

Build an aggregator for Congolese news.

That idea quickly became too limited. A useful system had to do more than collect links. It needed to:

• collect articles from selected legitimate sources;
• support different website structures;
• normalize article data into a consistent format;
• classify articles by topic;
• reduce duplication and information noise;
• support search and filtering;
• prepare the dataset for semantic search and recommendation;
• remain cheap enough to run continuously;
• expose processed news through web and mobile applications.

I needed a low-cost pipeline that could handle different media websites without hard-coding the full workflow for each source. The same stored data also had to support search, reporting, and later experiments with recommendations.

System architecture

Basango uses four parts:

1. Crawler: collects and normalizes articles
2. Database: stores, deduplicates, indexes, and classifies data
3. API: validates, authenticates, and exposes backend capabilities
4. Consumers: provide dashboard and mobile access to curated information

The crawler handles differences between sources. PostgreSQL stores, deduplicates, classifies, and indexes articles. The API validates requests and applies authentication. The dashboard and mobile app read the processed data.

Crawler engine

The crawler is the first major technical component of Basango. Its role is to transform a fragmented set of Congolese news websites into a unified stream of structured articles.

The sources required two extraction strategies. WordPress sites exposed structured JSON through /wp-json, including posts, dates, titles, links, and metadata.

Other websites did not expose a usable API. For those sources, Basango had to parse HTML directly, extract article links from listing pages, visit detail pages, and extract fields such as title, body, publication date, categories, and canonical URL.

Downloading pages was straightforward. Supporting both API-based and HTML-based sources without duplicating the crawler for every media outlet required a shared engine.

Design goal

The crawler follows one rule:

Source-specific behavior should live in configuration, while the crawling engine should remain generic.

Instead of writing a custom crawler class for every media outlet, Basango defines a source configuration schema. Each source declares its identifier, base URL, source kind, pagination strategy, date format, category support, rate-limiting needs, detail-fetching behavior, and, for HTML sources, the CSS selectors required to extract article data.

Adding or repairing a source changes its configuration. The crawl sequence stays the same.

Parser abstraction

Basango uses two concrete parser implementations behind a shared crawling model:

1. HtmlCrawler extracts article data from HTML selectors.

2. WordPressCrawler fetches article lists and metadata from WordPress REST endpoints.

The sourceKind field selects the parser:

type AnySourceOptions = HtmlSourceOptions | WordPressSourceOptions;

type SourceKind = "html" | "wordpress";

The sourceKind value selects the parser at runtime.

Configuration-driven crawling

The crawler validates configuration with Zod before execution.

Crawler configuration is fragile. A broken selector, invalid source URL, unsupported source kind, or malformed date format can silently corrupt extracted data, so the application validates configuration before a crawl starts.

HTML sources require explicit selectors because each website has its own structure:

{
  "articleBody": ".field-name-body",
  "articleCategories": ".views-field-field-cat-gorie a",
  "articleDate": "head > meta[property=\"article:published_time\"]",
  "articleLink": ".views-field-title a",
  "articles": ".view-content > .views-row.content-row",
  "articleTitle": "h1.page-header",
  "pagination": "ul.pagination > li.pager-last > a"
}

For example, one source may expose article cards under:

.view-content > .views-row.content-row

while another may use:

.for_aitems > .article_other_item

The crawler does not need to know those details. It only consumes the normalized configuration.

WordPress sources require less extraction configuration because their public REST API already exposes structured content:

{
  "sourceId": "example.com",
  "sourceKind": "wordpress",
  "sourceUrl": "https://example.com"
}

The WordPress API removed the need for custom selectors on those sources.

Parser selection flow

Crawling workflow

The crawler processes a source in three steps:

1. Discover article links
2. Fetch article details
3. Persist or forward the normalized article

Listing pages discover links with few requests. Detail jobs make one request per article and parse more HTML. A separate persistence step can retry storage failures without repeating discovery.

Synchronous and asynchronous execution

Basango supports both synchronous and asynchronous crawling.

Synchronous mode runs one source from the command line without Redis workers. I use it to test selectors, pagination, date parsing, and normalized fields while adding a source.

Asynchronous mode lets the crawler add workers as the workload grows. Basango uses BullMQ with Redis to split the crawling process into background jobs.

The async pipeline separates listing discovery from article detail extraction:

When detail requests become the bottleneck, the runtime can add article workers without adding listing workers.

The QueueManager exposes only the operations used by the crawler:

enqueueListing(payload)
enqueueArticle(payload)
iterQueueNames()
queueName(suffix)
close()

This hides BullMQ-specific implementation details from the rest of the crawler.

Persistence strategy

The crawler persists articles through a Persistor interface:

interface Persistor {
  persist(record: Partial<Article>): Promise<void> | void;
  close(): Promise<void> | void;
}

The first concrete implementation is JsonlPersistor, which writes articles as newline-delimited JSON.

JSON Lines lets the crawler append one article at a time and replay the file without loading the full dataset into memory.

Each persisted article is sanitized before storage:

• non-breaking spaces are normalized;
• zero-width characters are removed;
• line endings are normalized;
• repeated newlines are collapsed;
• title, body, and categories are cleaned.

The crawler also generates a stable hash from the article link:

hash: md5(data.link)

The resulting identifier supports deduplication and idempotent processing.

After local persistence, the crawler forwards the article to the Basango API. The split provides two guarantees:

• crawled data can be stored locally for inspection, replay, or dataset generation;
• successfully normalized articles can be pushed into the main application backend through a controlled ingestion boundary.

Incremental crawling and HTTP reliability

News crawling is both a data extraction problem and an operational problem. A crawler that repeatedly fetches the same content wastes bandwidth, CPU, storage, and downstream processing.

Basango supports constraints such as page range, date range, category, source update dates, update direction; These controls allow the system to avoid blindly crawling everything every time.

The HTTP client is also configurable with production-oriented behavior:

• request timeout;
• maximum retries;
• retry backoff;
• redirect support;
• SSL verification;
• Retry-After handling;
• user-agent configuration;
• optional user-agent rotation.

Media websites are unreliable infrastructure targets. They may be slow, temporarily unavailable, misconfigured, rate-limited, or inconsistent when automated clients connect.

Some sources also require more careful crawling behavior. The configuration supports:

requiresRateLimit: boolean

Per-source settings control rate limits, retries, and request behavior.

Why the crawler design matters

The crawler keeps source changes local.

Adding a new WordPress-based source can be as simple as adding a configuration entry:

{
  "sourceId": "example.com",
  "sourceKind": "wordpress",
  "sourceUrl": "https://example.com"
}

Adding a custom HTML source requires more configuration, but not a new crawler implementation:

{
  "sourceId": "example.com",
  "sourceKind": "html",
  "sourceUrl": "https://example.com",
  "paginationTemplate": "actualite",
  "sourceSelectors": {
    "articles": ".article-list .item",
    "articleTitle": "h1",
    "articleLink": "a",
    "articleDate": "meta[property=\"article:published_time\"]",
    "articleBody": ".article-body",
    "pagination": ".pagination a:last-child"
  }
}

Source variability lives in configuration. The runtime keeps the same discovery, detail, normalization, and persistence sequence. The result is a configuration-driven ingestion engine that supports heterogeneous news sources, normalizes them into a common article representation, and can run either synchronously for development or asynchronously for production-scale crawling.

Database layer

With the crawler defining how information enters Basango, the next major challenge is what happens after collection.

PostgreSQL stores each normalized article, rejects duplicates, builds the search index, and links the record to sources, categories, bookmarks, and reports.

The database code lives in its own monorepo package. The API imports its queries, while the crawler, dashboard, and mobile app go through the API.

The repository groups the API, crawler, dashboard, and mobile applications with shared database, domain, logging, encryption, and UI packages.

Design goal

The database follows one rule:

Store articles as structured knowledge objects, not as scraped text blobs.

A basic aggregator could store every article as { title, body, url }. That is enough for a feed, but not for source analysis, topic classification, deduplication, search, reporting, or later research.

Basango models the database around the main entities of a news intelligence system:

Source
Article
Category
User
Bookmark
Comment
FollowedSource
RefreshToken
VerificationToken
LoginHistory

These records support crawler ingestion, search, reports, bookmarks, comments, and source following.

The database has to support three different workloads:

Technology choice

Basango uses PostgreSQL with Drizzle ORM.

PostgreSQL stores the relations between articles, sources, categories, users, bookmarks, and comments. Reports use joins, grouping, sorting, filtering, and time windows over those records.

The database package keeps Drizzle schema definitions and migrations in packages/db/src/schema.ts. packages/db/src/client.ts creates a PostgreSQL pool and exports the typed Drizzle client.

The packages depend on one another as follows:

The API imports the database model. The dashboard uses shared backend types. The crawler sends records through the ingestion endpoint instead of writing to PostgreSQL.

Core data model

At the center of the database is the article table.

Each article stores its text with source information, publication date, categories, sentiment, credibility metadata, token statistics, reading time, deduplication hash, search index, and crawler timestamps.

The schema also includes related tables for sources, categories, users, bookmarks, followed sources, comments, login history, verification tokens, and refresh tokens.

Conceptually, the model looks like this:

The same records support the current feed, dashboard reports, and later retrieval experiments.

For example, bookmarks and followed_sources make the system ready for personalization. credibility creates room for source quality scoring. token_statistics prepares the dataset for NLP and LLM processing. categories and category_id support both source-provided labels and normalized platform-level classification.

Article as a canonical document

The article is the central database record.

The crawler receives article data from different websites, but the database stores it through one canonical model:

id
sourceId
categoryId
title
body
link
hash
publishedAt
crawledAt
categories
metadata
credibility
sentiment
readingTime
tokenStatistics
clustered
tsv

This is where normalization becomes real.

A publisher may classify articles using labels like:

Politique
Actualité
Sécurité
Nord-Kivu
RDC

But Basango needs a cleaner product-level taxonomy:

Politics
Security
Economy
Society
Environment
Culture
International affairs

Basango handles this by preserving raw article categories while also assigning a normalized canonical category.

That is a strong design decision. Raw categories preserve how the original publisher described the article. Canonical categories make the product consistent across different sources.

Without this split, the application would either lose valuable source metadata or expose users to a messy category system.

Deduplication and idempotency

News crawling is repetitive by nature.

A crawler may fetch the same article multiple times because of pagination changes, retries, update crawls, category pages, or source-specific duplication. The database therefore needs idempotency.

Basango generates a stable hash from the article link and uses it to detect duplicates before insertion. In the article creation query, the system first checks whether an article with the same hash already exists. If it does, it returns the existing article reference instead of inserting a duplicate.

That decision makes the crawler safe to rerun.

A crawler should be able to fail, retry, restart, and reprocess pages without corrupting the dataset. Idempotency is what makes that possible.

Classification strategy

The first version of classification is deliberately pragmatic.

Instead of starting with an expensive LLM-based classifier, Basango uses a deterministic category classifier based on canonical categories, candidates, normalization, and weighted matching. The classifier normalizes source-provided category labels, compares them against known candidates, scores matches, and assigns the best canonical category.

This is the right engineering move for the current phase.

LLMs can be useful later, especially for ambiguous articles or richer topic modeling. But for a continuous ingestion system, every automated decision has a cost. Rule-based classification is cheaper, predictable, debuggable, and good enough as a first layer.

The classification strategy is layered:

An LLM can later handle low-confidence matches without replacing the rule-based path.

Search and retrieval

Basango uses PostgreSQL full-text search as the first search layer.

PostgreSQL full-text search identifies natural-language documents that match a query and can rank them by relevance. It preprocesses text into lexemes, stores searchable vectors, and supports indexed search through tsvector and tsquery.

PostgreSQL covers the current search requirements without another service to deploy or synchronize.

Instead of immediately introducing Elasticsearch, Meilisearch, or a vector database, Basango starts with what PostgreSQL already provides:

GIN index on search vector
trigram indexes on title and link
source/date/id index for feed pagination
category index for filtering
hash uniqueness for deduplication

The schema includes indexes for article categories, title, link, search vector, source, publication date, and article ID.

This matches the main read paths:

Latest articles
Articles by source
Articles by category
Articles by sentiment
Search by keyword
Publication graph
Source distribution

The database is optimized for the workflows Basango actually needs instead of becoming a generic dump of scraped records.

Pagination strategy

Feeds need pagination, but offset pagination becomes expensive and unstable when data changes frequently.

Basango uses a keyset-style pagination strategy based on publication date and article ID. The article query builds pagination state, applies filters, orders by publishedAt and id, and fetches one extra record to determine whether another page exists.

This is a strong backend decision for a news system.

In a constantly growing dataset, offset pagination can skip or duplicate records when new articles arrive. Keyset pagination is better because it uses a cursor based on stable ordered fields.

The query model supports filters such as:

source
sentiment
category
search query
cursor
limit

The API can use the same pagination contract for the dashboard and mobile feed.

Analytics and reporting

Basango is also designed for analysis.

The database query layer exposes reporting functions for:

article publication graph
article source distribution
source publication graph
source category shares
dashboard overview

The database package owns these aggregate queries. The API exposes their results through typed procedures.

The reporting layer computes metrics such as total articles, total users, total sources, active sources in a period, publication count over time, and percentage distribution by source or category.

Basango supports public reading, source monitoring, and research queries, so it needs both article retrieval and aggregate reports.

The dashboard needs to answer operational questions:

Are crawlers producing data?
Which sources are active?
Which topics dominate a source?
How much content was collected this week?
Are some sources overrepresented?

Those questions require aggregate queries beyond record creation and retrieval.

Database query layer

The database package separates schema definitions from query functions.

The database package exposes this interface:

createArticle()
getArticles()
getArticleById()
getArticlesPublicationGraph()
getArticlesSourceDistribution()

createSource()
updateSource()
getSources()
getSourceById()
getSourcePublicationGraph()
getSourceCategoryShares()

getCategories()
getDashboardOverview()
getUserByEmail()
getUserById()

Route handlers call database functions instead of embedding SQL. Query tests can therefore run without invoking the HTTP transport.

It also means the same query functions can be reused by different entry points: REST ingestion endpoints, tRPC procedures, background jobs, CLI scripts, or future workers.

Database flow

Why the database design matters

The database design makes Basango resilient.

The crawler can be noisy, websites can change, and article categories can be inconsistent, but the database provides a stable model for the rest of the system.

The database uses these rules:

• use PostgreSQL as the core persistence and search layer;
• use Drizzle to keep schema and queries typed;
• store articles as structured documents, not text blobs;
• preserve raw source metadata while adding canonical platform metadata;
• use hashes for idempotent ingestion;
• use indexed search before adding more expensive infrastructure;
• keep analytics queries close to the database;
• model user-facing features early enough to support personalization later.

The database package now owns storage, search, reports, source monitoring, and topic assignments. Future personalization can read those same records without changing the crawler.

API layer

The API accepts crawler records and serves articles, reports, and account data to the clients.

The API receives normalized articles from the crawler, validates inputs, exposes authenticated data access to first-party clients, and provides reporting endpoints for the dashboard.

It protects the database and groups product capabilities into typed procedures.

Basango uses Hono with secure headers, logging, CORS configuration, REST routers, and a tRPC server mounted under /trpc/*.

The API has two roles:

The crawler posts JSON to one HTTP endpoint. The dashboard and mobile app use tRPC procedures with shared TypeScript types.

Design goal

The API follows one rule:

Keep the product API type-safe, but keep ingestion simple and explicit.

Basango has different consumers:

Crawler
Dashboard
Mobile app
Future external tools

The crawler does not need the same API experience as the dashboard. It needs a stable ingestion endpoint. The dashboard and mobile app need typed queries, authentication, pagination, filtering, analytics, and safe mutations.

Using both REST and tRPC lets each consumer use the right interface.

Why tRPC

tRPC fits Basango because the project is a TypeScript monorepo.

tRPC is an implementation of Remote Procedure Call for TypeScript applications. Instead of calling URLs directly and manually maintaining DTOs, clients call typed functions exposed by the server.

The web dashboard and mobile app consume the same first-party API, so contract changes must reach both clients.

The dashboard and mobile app should not manually duplicate backend DTOs. They should consume the server contract directly.

With tRPC, a backend router like this:

articles.list
articles.create
articles.getPublications
sources.list
sources.update
reports.getDashboardOverview
auth.session

becomes a typed client API.

This reduces integration bugs. If the backend input changes, the client breaks at compile time instead of failing at runtime.

Router structure

The tRPC API groups procedures by domain.

The application router combines:

articles
auth
categories
reports
sources

These routers expose the major product capabilities: article ingestion and listing, authentication, category retrieval, dashboard reporting, and source management. The API is organized around product capabilities instead of technical folders such as controllers and handlers.

Context, middleware, and authentication

Every serious API needs a context model.

Basango's tRPC context includes:

database connection
session
geolocation context

The API extracts an access token from the Authorization header, resolves the session, attaches the database instance, and derives geo context from the request. The tRPC initialization also defines public and protected procedures. Protected procedures require both database access and authentication.

Public and protected procedures use the same authentication context.

Instead of checking authentication manually in every procedure, Basango centralizes it through procedure composition:

publicProcedure
protectedProcedure

The authentication router exposes login, refresh, and session procedures. Login validates credentials and locked status before returning tokens. Refresh validates the refresh token and issues a new session. Session returns the authenticated user.

Basango also has bookmarks, followed sources, comments, personalized feeds, dashboard access, and source administration. Those features require authenticated user context.

Those features require a reliable identity layer.

API and database separation

The API delegates database work to query functions from @basango/db.

A procedure should usually be thin:

The article router calls functions such as:

createArticle()
getArticles()
getArticlesPublicationGraph()
getArticlesSourceDistribution()

The source router calls functions such as:

createSource()
updateSource()
getSources()
getSourceById()
getSourcePublicationGraph()
getSourceCategoryShares()

Database functions contain the queries. Routers validate input, authorize the caller, and return the result.

The API should not care about SQL details. The database package should not care whether the caller is tRPC, REST, a worker, or a CLI command.

New transports can reuse the database functions without copying queries.

Validation strategy

The API uses shared domain schemas from @basango/domain.

Validation belongs at the boundary. The crawler may send malformed article data. The dashboard may send invalid filters. A mobile client may call a procedure with an expired token. Every input crossing into the backend needs validation.

Basango avoids duplicating validation logic by keeping domain models and schemas in a shared package. The API imports those schemas and applies them at procedure boundaries.

The pattern is:

The crawler and both clients are checked against the same schemas.

REST boundary for ingestion

The crawler forwards normalized articles to the backend API after local persistence.

That is a deliberate design decision.

The crawler does not write directly to the database. It treats the API as the ingestion boundary.

Routing ingestion through the API has several effects:

• the database stays private;
• ingestion can be authenticated;
• validation happens in one place;
• crawler failures are isolated from database access;
• future ingestion clients can reuse the same boundary.

Direct database writes from every internal component are tempting, but they create tight coupling and inconsistent validation. Basango avoids that by making the API the controlled entry point.

API runtime flow

Why the API design matters

The API is the single write path for crawler records.

The crawler can evolve independently. The database schema can evolve behind query functions. The dashboard and mobile app can consume strongly typed procedures. Authentication, session resolution, CORS, secure headers, and request logging are centralized.

The API uses these rules:

• use REST where ingestion simplicity matters;
• use tRPC where first-party type safety matters;
• centralize authentication through protected procedures;
• keep database queries outside API route handlers;
• reuse domain schemas for input validation;
• expose analytical queries as product capabilities;
• keep the API thin, typed, and composable.

The result is one backend boundary for ingestion, reporting, authentication, search, and future personalization.

Consumers: web dashboard and mobile app

The crawler, database, and API feed two client applications.

The web and mobile applications let people search, filter, save, and read the processed articles.

Basango has two clients:

That distinction matters.

A dashboard is for understanding the system. A mobile app is for benefiting from the system.

Open image preview: Basango dashboard sources viewOpen image preview: Basango dashboard articles viewOpen image preview: Basango dashboard overviewOpen image preview: Basango dashboard detail view

Web dashboard

The web dashboard supports Basango's internal operations.

It exposes capabilities such as:

total articles
total sources
active sources
publication trends
source distribution
category shares
article lists
source management
crawler outputs

The dashboard uses Next.js, tRPC, TanStack Query, React Table, Recharts, Shadcn UI components, and shared domain packages.

The dashboard consumes the same API contract as the other clients instead of reading raw database tables.

It uses reporting procedures such as:

reports.getDashboardOverview
articles.getPublications
articles.getSourceDistribution
sources.getPublications
sources.getCategoryShares

The dashboard uses the reporting procedures that also expose crawler health and source activity.

Dashboard as an observability tool

For a crawler-based system, an admin dashboard is not optional.

Crawlers fail silently if nobody monitors them. A source can change its HTML structure. A WordPress API can stop responding. A date format can break. A source can suddenly produce far fewer articles than expected.

The dashboard helps answer operational questions:

Did the crawler collect articles today?
Which sources are active?
Which sources are overrepresented?
Are categories being assigned correctly?
Is a source producing mostly one kind of content?
Did publication volume change compared to the previous period?

The dashboard therefore helps diagnose the ingestion system as well as present its data.

The dashboard exposes crawl volume, active sources, category distribution, and recent ingestion results.

Mobile application

The mobile app lets users read and save articles.

Its role is different from the dashboard. The dashboard helps operate the system. The mobile app helps users stay informed without overconsuming information.

The mobile app uses Expo, React Native, Expo Router, and React Navigation.

The mobile product direction is:

personalized news feed
topic-specific reading
source following
saved articles
article details
search
recommendations
notifications

The database already contains structures that support this direction, including users, bookmarks, followed sources, comments, and authentication-related tables.

The mobile client reads the existing article, category, source, and bookmark records.

Shared API contract

Both clients call Basango through the API.

tRPC lets both TypeScript clients call backend procedures through shared types. A procedure rename or input change then fails during type checking in the affected client.

The clients share this integration path:

api.articles.list()
api.sources.list()
api.reports.getDashboardOverview()
api.auth.login()
api.auth.session()

Instead of duplicating request and response types in every client, the server router becomes the contract.

That reduces integration bugs and makes refactoring safer. If an API input changes, the dashboard and mobile app receive type errors during development.

That is the correct failure mode.

Consumer architecture

The consumer layer is organized around shared contracts:

The web dashboard can focus on tables, charts, filters, and source administration.

The mobile app can focus on reading, saved articles, preferences, and notifications.

Both clients depend on the same backend concepts:

Article
Source
Category
User
Session
Bookmark
Report

Both clients use the same names and validation rules for articles, sources, and categories.

Web and mobile responsibilities

The dashboard should not try to be the mobile app. The mobile app should not try to be the dashboard.

Web-specific tables and mobile-specific navigation remain inside their clients.

The dashboard answers:

What is the system collecting?
Which sources are active?
How is the dataset evolving?
Are categories and sources healthy?

The mobile app answers:

What should I read now?
What happened about the topics I follow?
Which articles are relevant to me?
What should I save or revisit later?

This distinction prevents the product from becoming one overloaded interface.

Clients call API procedures. API routers validate and authorize requests, then call the database package. Database functions return records without knowing which client requested them.

Key engineering decisions

Start with the real workflow, not the technology

The project started with the work of following DRC news, especially the conflict in the east, without checking dozens of sources by hand.

That problem shaped the architecture.

The goal was not "build a crawler" or "use an LLM." The goal was to reduce the cost of staying informed.

That is why the system includes crawling, normalization, search, classification, dashboard reporting, and mobile consumption.

Use configuration for source variability

News websites change. HTML structures differ. Some sites expose WordPress APIs, others do not.

A configuration-driven crawler makes source onboarding and maintenance easier.

Source configuration changes more often than the crawl sequence, so selectors and request settings belong in configuration.

Keep ingestion idempotent

The crawler will retry. Pages will overlap. Sources will duplicate articles.

A hash-based deduplication strategy allows the crawler to run repeatedly without corrupting the dataset.

This is one of those small backend decisions that separates a prototype from a real system.

Use deterministic classification before LLM classification

The first classification layer is rule-based and candidate-driven.

That is not less sophisticated. It is more responsible.

LLMs can improve ambiguous classification later, but they should not be the first dependency in a system that needs to run continuously and cheaply.

Use PostgreSQL before adding search infrastructure

PostgreSQL already provides relational modeling, indexes, aggregation, and full-text search.

Adding Elasticsearch, Meilisearch, or a vector database too early would increase operational cost and complexity.

The better decision was to exhaust PostgreSQL first, then add specialized systems only when the product proves they are needed.

Keep API transport separate from database logic

The API exposes product capabilities.

The database package owns queries.

Route handlers validate and authorize requests, while database functions contain the SQL.

Use tRPC for first-party clients

The web dashboard and mobile app are first-party TypeScript clients.

tRPC makes sense because it gives them a typed contract without code generation or duplicated DTOs.

For external integrations or ingestion, REST still makes sense.

Using both is not inconsistency. It is using the right interface for the right consumer.

Use the dashboard to inspect ingestion

The dashboard helps users browse articles and helps me operate the crawler and inspect data quality.

Without these reports, a broken selector can keep producing empty jobs without a visible error on the public feed.

What I learned

The biggest lesson from Basango is that downloading pages is the easy part. The difficult work starts when the system must retry failed requests, compare duplicates, normalize inconsistent metadata, and show whether the latest crawl actually produced useful articles.

Crawling must be treated as an unreliable process. Websites change, metadata is inconsistent, pagination is not always clean, and source categories cannot be trusted as a product taxonomy.

I also learned to delay AI until there is a specific reason to use it. Rule-based classification is cheaper to run and easier to inspect. It also gives me a baseline against which I can measure a later model.

Once collection worked, retrieval became the larger task. The database had to deduplicate articles, normalize categories, index text, filter results, and compute reports.

The API layer also became a major design boundary. By separating REST ingestion from tRPC client access, Basango supports both machine workflows and first-party application development without forcing one interface style everywhere.

The dashboard is also an operating tool, not decoration. It shows whether jobs are running, when each source last produced data, and where an extraction rule has stopped working.

Current state

Basango now has the main pieces of the curation pipeline: source adapters, queued crawling, idempotent ingestion, PostgreSQL search, topic classification, typed APIs, and web and mobile clients. The next work is empirical. I need to measure source coverage, extraction failures, duplicate rates, and classification quality before adding semantic search or recommendations.

The project should reduce time spent collecting reports and leave more time to compare what credible sources say about the DRC.