Basango

Basango started as a personal need to follow critical news about the Democratic Republic of Congo more responsibly, especially the war and security situation in eastern DRC. It evolved into a backend-driven information system for automated news crawling, normalization, classification, search, analytics, and future recommendation.

The goal was not simply to build another news aggregator. The real goal was to reduce the manual work required to find credible, relevant, non-duplicated information across a fragmented media ecosystem.

Basango is designed as an ingestion-to-consumption pipeline and combines system design, backend architecture, data modeling, web crawling, information retrieval, and applied AI readiness.

The Problem

The core problem behind Basango was not access to news. The real problem was reliable, low-noise, topic-specific news retrieval.

The Democratic Republic of Congo has many online news outlets, social media accounts, and informal channels publishing political, economic, social, cultural, environmental, and security-related information. Some sources are reputable and consistent. Others are unreliable, politically biased, poorly structured, or difficult to evaluate.

Following a specific topic manually, such as the eastern DRC conflict, means moving between multiple news websites, Twitter/X feeds, and search results. Each day, the user has to filter what is relevant, compare sources, remove duplicates, identify noise, and decide which reports are credible enough to use.

In practice, staying informed becomes daily manual information retrieval work.

RSS readers and feed aggregators solve part of this problem. RSS provides a standardized way for applications to consume website updates without manually checking each website. However, many Congolese media websites either do not provide usable RSS feeds, do not expose topic-level subscription models, or publish too broadly for focused monitoring.

Generic tools like Feedly can aggregate sources, but they do not reliably answer the question Basango is built around:

What is happening now, from credible sources, about the topics I care about?

That turned Basango from a simple news app idea into an information retrieval and curation system.

In Basango, information retrieval means collecting articles from selected sources, normalizing them into a unified structure, classifying them by topic, and making them searchable, filterable, and eventually recommendable.

Product Direction

Basango started with a simple idea:

Build an aggregator for Congolese news.

That idea quickly became too limited. A useful system had to do more than collect links. It needed to:

• collect articles from selected legitimate sources;
• support different website structures;
• normalize article data into a consistent format;
• classify articles by topic;
• reduce duplication and information noise;
• support search and filtering;
• prepare the dataset for semantic search and recommendation;
• remain cheap enough to run continuously;
• expose processed news through web and mobile applications.

The engineering challenge was to design a low-cost, extensible news curation pipeline capable of crawling heterogeneous Congolese media websites, extracting article content through source-specific adapters, normalizing data into a unified model, classifying articles by topic, and preparing the dataset for search, analytics, recommendation, and downstream text analysis.

System Architecture

Basango is organized around four main layers:

1. Crawler: collects and normalizes articles
2. Database: stores, deduplicates, indexes, and classifies data
3. API: validates, authenticates, and exposes backend capabilities
4. Consumers: provide dashboard and mobile access to curated information

Each layer has a clear responsibility.

The crawler solves source heterogeneity. The database solves persistence, deduplication, classification, search, and analytics. The API solves validation, authentication, orchestration, and typed access. The consumers make the curated information usable through operational and user-facing interfaces.

Crawler Engine

The crawler is the first major technical component of Basango. Its role is to transform a fragmented set of Congolese news websites into a unified stream of structured articles.

The initial research showed that news sources could not be handled with a single extraction strategy. Some websites were built on WordPress and exposed structured JSON through the WordPress REST API, usually available under /wp-json. That made article data easier to retrieve because posts, dates, titles, links, and metadata could be fetched in a machine-readable format.

Other websites did not expose a usable API. For those sources, Basango had to parse HTML directly, extract article links from listing pages, visit detail pages, and extract fields such as title, body, publication date, categories, and canonical URL.

The real challenge was not simply downloading pages. It was designing a unified crawling engine that could support both API-based and HTML-based sources without duplicating crawler logic for every media outlet.

Design Goal

The crawler was designed around one core idea:

Source-specific behavior should live in configuration, while the crawling engine should remain generic.

Instead of writing a custom crawler class for every media outlet, Basango defines a source configuration schema. Each source declares its identifier, base URL, source kind, pagination strategy, date format, category support, rate-limiting needs, detail-fetching behavior, and, for HTML sources, the CSS selectors required to extract article data.

This creates a clean separation between the source-specific extraction rules and the generic crawling workflow.

Parser Abstraction

Basango uses two concrete parser implementations behind a shared crawling model:

1. HtmlCrawler is used for websites where article data must be extracted from HTML using selectors.

2. WordPressCrawler is used for websites exposing WordPress REST endpoints, where article listings and metadata can be fetched as JSON.

Both parsers are selected from the source configuration using the sourceKind field:

type AnySourceOptions = HtmlSourceOptions | WordPressSourceOptions;

type SourceKind = "html" | "wordpress";

This allows the crawler engine to resolve a source dynamically and delegate extraction to the correct parser.

Configuration-Driven Crawling

Crawler configuration is validated with Zod. This gives the crawler a type-safe boundary between external configuration and runtime execution.

That matters because crawling configuration is fragile. A broken selector, invalid source URL, unsupported source kind, or malformed date format can silently corrupt extracted data if it is not caught early.

HTML sources require explicit selectors because each website has its own structure:

{
  "articleBody": ".field-name-body",
  "articleCategories": ".views-field-field-cat-gorie a",
  "articleDate": "head > meta[property=\"article:published_time\"]",
  "articleLink": ".views-field-title a",
  "articles": ".view-content > .views-row.content-row",
  "articleTitle": "h1.page-header",
  "pagination": "ul.pagination > li.pager-last > a"
}

For example, one source may expose article cards under:

.view-content > .views-row.content-row

while another may use:

.for_aitems > .article_other_item

The crawler does not need to know those details. It only consumes the normalized configuration.

WordPress sources require less extraction configuration because their public REST API already exposes structured content:

{
  "sourceId": "example.com",
  "sourceKind": "wordpress",
  "sourceUrl": "https://example.com"
}

This was an important discovery during the research phase because it reduced the amount of custom extraction work required for WordPress-based media outlets.

Parser Selection Flow

Crawling Workflow

Crawling a source is divided into three independent steps:

1. Discover article links
2. Fetch article details
3. Persist or forward the normalized article

This separation matters because each step has different performance characteristics.

Listing pages are lightweight and mostly used for discovery. Article detail pages are more expensive because they require additional HTTP requests and parsing. Persistence is isolated so storage or backend failures do not have to crash the entire crawling operation.

Synchronous and Asynchronous Execution

Basango supports both synchronous and asynchronous crawling.

Synchronous mode was implemented for development and iteration. It allows a source to be crawled immediately from the command line without Redis workers or queue orchestration. This is useful for testing a new source configuration, debugging broken selectors, validating pagination, checking date parsing, and inspecting extracted article fields.

Asynchronous mode is used for scalable crawling. Basango uses BullMQ with Redis to split the crawling process into background jobs.

The async pipeline separates listing discovery from article detail extraction:

listing queue  -> discovers article URLs
details queue  -> fetches and persists article content

This makes the crawler easier to scale horizontally. More workers can be added when article fetching becomes the bottleneck.

The queue layer is intentionally small. The QueueManager exposes only the operations the crawler needs:

enqueueListing(payload)
enqueueArticle(payload)
iterQueueNames()
queueName(suffix)
close()

This hides BullMQ-specific implementation details from the rest of the crawler.

Persistence Strategy

The crawler persists articles through a Persistor interface:

interface Persistor {
  persist(record: Partial<Article>): Promise<void> | void;
  close(): Promise<void> | void;
}

The first concrete implementation is JsonlPersistor, which writes articles as newline-delimited JSON.

JSON Lines is a strong fit for this use case because it is append-friendly, easy to stream, easy to inspect, and convenient for dataset-oriented workflows.

Each persisted article is sanitized before storage:

• non-breaking spaces are normalized;
• zero-width characters are removed;
• line endings are normalized;
• repeated newlines are collapsed;
• title, body, and categories are cleaned.

The crawler also generates a stable hash from the article link:

hash: md5(data.link)

This provides a deterministic identifier that can be used for deduplication and idempotent processing.

After local persistence, the article is forwarded to the Basango backend API. This creates two useful guarantees:

• crawled data can be stored locally for inspection, replay, or dataset generation;
• successfully normalized articles can be pushed into the main application backend through a controlled ingestion boundary.

Incremental Crawling and HTTP Reliability

News crawling is both a data extraction problem and an operational problem. A crawler that repeatedly fetches the same content wastes bandwidth, CPU, storage, and downstream processing.

Basango supports constraints such as page range, date range, category, source update dates, update direction; These controls allow the system to avoid blindly crawling everything every time.

The HTTP client is also configurable with production-oriented behavior:

• request timeout;
• maximum retries;
• retry backoff;
• redirect support;
• SSL verification;
• Retry-After handling;
• user-agent configuration;
• optional user-agent rotation.

This matters because media websites are not stable infrastructure targets. They may be slow, temporarily unavailable, misconfigured, protected by rate limits, or inconsistent in how they respond to automated clients.

Some sources also require more careful crawling behavior. The configuration supports:

requiresRateLimit: boolean

This allows the crawler to treat sensitive sources more responsibly instead of applying the same behavior to every website.

Why the Crawler Design Matters

The main value of the crawler is extensibility.

Adding a new WordPress-based source can be as simple as adding a configuration entry:

{
  "sourceId": "example.com",
  "sourceKind": "wordpress",
  "sourceUrl": "https://example.com"
}

Adding a custom HTML source requires more configuration, but not a new crawler implementation:

{
  "sourceId": "example.com",
  "sourceKind": "html",
  "sourceUrl": "https://example.com",
  "paginationTemplate": "actualite",
  "sourceSelectors": {
    "articles": ".article-list .item",
    "articleTitle": "h1",
    "articleLink": "a",
    "articleDate": "meta[property=\"article:published_time\"]",
    "articleBody": ".article-body",
    "pagination": ".pagination a:last-child"
  }
}

That is the important engineering decision: source variability is handled declaratively through configuration, while the crawling runtime remains reusable. The result is a configuration-driven ingestion engine that supports heterogeneous news sources, normalizes them into a common article representation, and can run either synchronously for development or asynchronously for production-scale crawling.

Database Layer

With the crawler defining how information enters Basango, the next major challenge is what happens after collection.

The database layer is responsible for turning raw collected news into a durable, queryable, deduplicated, searchable, classifiable, and analyzable information base.

The crawler produces normalized article records, but those records only become useful once they can be deduplicated, indexed, searched, classified, filtered, and connected to sources, categories, users, bookmarks, and reporting views.

Basango implements the database layer as its own package inside the monorepo, separate from the API, crawler, dashboard, and mobile applications. This is an important architectural decision: the database model is treated as a shared backend capability, not as private code hidden inside one application.

The repository is organized around applications such as API, crawler, dashboard, and mobile, and shared packages such as database, domain, logger, encryption, and UI.

Design Goal

The database was designed around one core idea:

Store articles as structured knowledge objects, not as scraped text blobs.

A naive aggregator could store every article as { title, body, url } and call it a day. That would work for displaying a basic feed, but it would fail for the real goals of Basango: source analysis, topic classification, deduplication, search, recommendation, dashboards, and future research usage.

Basango models the database around the main entities of a news intelligence system:

Source
Article
Category
User
Bookmark
Comment
FollowedSource
RefreshToken
VerificationToken
LoginHistory

This structure shows that the project is not just a crawler. It is a platform foundation.

The database has to support three different workloads:

Ingestion  -> idempotent article creation from crawler output
Retrieval  -> paginated and filtered access for clients
Analysis   -> aggregation by source, date, category, and publication volume

Technology Choice

Basango uses PostgreSQL with Drizzle ORM.

PostgreSQL is a strong fit because the data is highly relational. Articles belong to sources. Articles can be classified into categories. Users can follow sources, bookmark articles, and comment on content. Reports need joins, grouping, sorting, filtering, and time-windowed aggregations.

The database package uses Drizzle schema definitions and migrations. The schema is centralized in packages/db/src/schema.ts, while the database client is exposed from packages/db/src/client.ts. The client uses a PostgreSQL connection pool and exposes a typed Drizzle instance to the rest of the backend.

This gives the project a clean database boundary:

apps/api        -> uses @basango/db
apps/crawler    -> forwards articles to API
packages/db     -> owns schema, queries, migrations
packages/domain -> owns shared models and validation schemas

The API does not redefine the database model. The dashboard does not manually recreate backend types. The crawler does not write directly to PostgreSQL. That separation keeps the system maintainable.

Core Data Model

At the center of the database is the article table.

An article is not only stored as text. It includes source information, publication date, categories, sentiment, credibility metadata, token statistics, reading time, deduplication hash, search index, and crawler timestamps.

The schema also includes related tables for sources, categories, users, bookmarks, followed sources, comments, login history, verification tokens, and refresh tokens.

Conceptually, the model looks like this:

Source
  └── Article
        ├── Category
        ├── Metadata
        ├── Credibility
        ├── Sentiment
        ├── Search vector
        └── Token statistics

User
  ├── Bookmarks
  ├── Followed sources
  ├── Comments
  ├── Login history
  └── Verification tokens

This structure supports both the current product and future extensions.

For example, bookmarks and followed_sources make the system ready for personalization. credibility creates room for source quality scoring. token_statistics prepares the dataset for NLP and LLM processing. categories and category_id support both source-provided labels and normalized platform-level classification.

Article as a Canonical Document

The most important database object is the article.

The crawler receives article data from different websites, but the database stores it through one canonical model:

id
sourceId
categoryId
title
body
link
hash
publishedAt
crawledAt
categories
metadata
credibility
sentiment
readingTime
tokenStatistics
clustered
tsv

This is where normalization becomes real.

A publisher may classify articles using labels like:

Politique
Actualité
Sécurité
Nord-Kivu
RDC

But Basango needs a cleaner product-level taxonomy:

Politics
Security
Economy
Society
Environment
Culture
International affairs

Basango handles this by preserving raw article categories while also assigning a normalized canonical category.

That is a strong design decision. Raw categories preserve how the original publisher described the article. Canonical categories make the product consistent across different sources.

Without this split, the application would either lose valuable source metadata or expose users to a messy category system.

Deduplication and Idempotency

News crawling is repetitive by nature.

A crawler may fetch the same article multiple times because of pagination changes, retries, update crawls, category pages, or source-specific duplication. The database therefore needs idempotency.

Basango generates a stable hash from the article link and uses it to detect duplicates before insertion. In the article creation query, the system first checks whether an article with the same hash already exists. If it does, it returns the existing article reference instead of inserting a duplicate.

That decision makes the crawler safe to rerun.

A crawler should be able to fail, retry, restart, and reprocess pages without corrupting the dataset. Idempotency is what makes that possible.

Classification Strategy

The first version of classification is deliberately pragmatic.

Instead of starting with an expensive LLM-based classifier, Basango uses a deterministic category classifier based on canonical categories, candidates, normalization, and weighted matching. The classifier normalizes source-provided category labels, compares them against known candidates, scores matches, and assigns the best canonical category.

This is the right engineering move for the current phase.

LLMs can be useful later, especially for ambiguous articles or richer topic modeling. But for a continuous ingestion system, every automated decision has a cost. Rule-based classification is cheaper, predictable, debuggable, and good enough as a first layer.

The classification strategy is layered:

1. Preserve raw source categories
2. Normalize category labels
3. Match against canonical candidates
4. Assign a platform-level category
5. Keep room for future embeddings or LLM classification

This keeps the door open for AI without making the whole system dependent on AI from day one.

Search and Retrieval

Basango uses PostgreSQL full-text search as the first search layer.

PostgreSQL full-text search identifies natural-language documents that match a query and can rank them by relevance. It preprocesses text into lexemes, stores searchable vectors, and supports indexed search through tsvector and tsquery.

That makes PostgreSQL a strong fit for the first version of Basango search.

Instead of immediately introducing Elasticsearch, Meilisearch, or a vector database, Basango starts with what PostgreSQL already provides:

GIN index on search vector
trigram indexes on title and link
source/date/id index for feed pagination
category index for filtering
hash uniqueness for deduplication

The schema includes indexes for article categories, title, link, search vector, source, publication date, and article ID.

This matches the main read paths:

Latest articles
Articles by source
Articles by category
Articles by sentiment
Search by keyword
Publication graph
Source distribution

The database is optimized for the workflows Basango actually needs instead of becoming a generic dump of scraped records.

Pagination Strategy

Feeds need pagination, but offset pagination becomes expensive and unstable when data changes frequently.

Basango uses a keyset-style pagination strategy based on publication date and article ID. The article query builds pagination state, applies filters, orders by publishedAt and id, and fetches one extra record to determine whether another page exists.

This is a strong backend decision for a news system.

In a constantly growing dataset, offset pagination can skip or duplicate records when new articles arrive. Keyset pagination is better because it uses a cursor based on stable ordered fields.

The query model supports filters such as:

source
sentiment
category
search query
cursor
limit

That gives the API a flexible foundation for the dashboard, mobile feed, and future recommendation surfaces.

Analytics and Reporting

Basango is also designed for analysis.

The database query layer exposes reporting functions for:

article publication graph
article source distribution
source publication graph
source category shares
dashboard overview

These queries are not UI logic. They belong in the database layer because they are data access and aggregation problems. The API simply exposes them through typed procedures.

The reporting layer computes metrics such as total articles, total users, total sources, active sources in a period, publication count over time, and percentage distribution by source or category.

This is important because Basango is not only a consumer news app. It is also a monitoring and research tool.

The dashboard needs to answer operational questions:

Are crawlers producing data?
Which sources are active?
Which topics dominate a source?
How much content was collected this week?
Are some sources overrepresented?

Those questions require aggregation, not just CRUD.

Database Query Layer

The database package separates schema definitions from query functions.

That gives the rest of the project a clean interface:

createArticle()
getArticles()
getArticleById()
getArticlesPublicationGraph()
getArticlesSourceDistribution()

createSource()
updateSource()
getSources()
getSourceById()
getSourcePublicationGraph()
getSourceCategoryShares()

getCategories()
getDashboardOverview()
getUserByEmail()
getUserById()

This is better than placing SQL directly inside API route handlers.

The API should orchestrate requests. The database package should own persistence and query behavior. This keeps the backend modular and easier to test.

It also means the same query functions can be reused by different entry points: REST ingestion endpoints, tRPC procedures, background jobs, CLI scripts, or future workers.

Database Flow

Why the Database Design Matters

The database design makes Basango resilient.

The crawler can be noisy, websites can change, and article categories can be inconsistent, but the database provides a stable model for the rest of the system.

The main engineering decisions are:

• use PostgreSQL as the core persistence and search layer;
• use Drizzle to keep schema and queries typed;
• store articles as structured documents, not text blobs;
• preserve raw source metadata while adding canonical platform metadata;
• use hashes for idempotent ingestion;
• use indexed search before adding more expensive infrastructure;
• keep analytics queries close to the database;
• model user-facing features early enough to support personalization later.

The result is a database layer that acts as more than storage. It is the foundation for search, analytics, source monitoring, topic classification, and future personalization.

API Layer

After the crawler collects articles and the database turns them into structured knowledge objects, the API becomes the coordination layer that controls how data enters, leaves, and moves through the system.

The API receives normalized articles from the crawler, validates inputs, exposes authenticated data access to first-party clients, and provides reporting endpoints for the dashboard.

It is not just a set of CRUD routes. It is the boundary that protects the database and organizes product capabilities into typed procedures.

Basango uses Hono with secure headers, logging, CORS configuration, REST routers, and a tRPC server mounted under /trpc/*.

That gives the API two roles:

REST boundary  -> ingestion and machine-facing endpoints
tRPC boundary  -> typed first-party application API

This is a practical split. The crawler can forward articles through a simple HTTP endpoint, while the dashboard and mobile app can use tRPC for a type-safe developer experience.

Design Goal

The API was designed around one core idea:

Keep the product API type-safe, but keep ingestion simple and explicit.

Basango has different consumers:

Crawler
Dashboard
Mobile app
Future external tools

The crawler does not need the same API experience as the dashboard. It needs a stable ingestion endpoint. The dashboard and mobile app need typed queries, authentication, pagination, filtering, analytics, and safe mutations.

Using both REST and tRPC lets each consumer use the right interface.

Why tRPC

tRPC fits Basango because the project is a TypeScript monorepo.

tRPC is an implementation of Remote Procedure Call for TypeScript applications. Instead of calling URLs directly and manually maintaining DTOs, clients call typed functions exposed by the server.

For Basango, that matters because the same API is consumed by first-party clients.

The dashboard and mobile app should not manually duplicate backend DTOs. They should consume the server contract directly.

With tRPC, a backend router like this:

articles.list
articles.create
articles.getPublications
sources.list
sources.update
reports.getDashboardOverview
auth.session

becomes a typed client API.

This reduces integration bugs. If the backend input changes, the client breaks at compile time instead of failing at runtime.

Router Structure

The tRPC API is organized into domain routers.

The application router combines:

articles
auth
categories
reports
sources

These routers expose the major product capabilities: article ingestion and listing, authentication, category retrieval, dashboard reporting, and source management. The API is organized around the capabilities the system exposes, not only around technical folders like controllers and handlers.

Context, Middleware, and Authentication

Every serious API needs a context model.

Basango’s tRPC context includes:

database connection
session
geolocation context

The API extracts an access token from the Authorization header, resolves the session, attaches the database instance, and derives geo context from the request. The tRPC initialization also defines public and protected procedures. Protected procedures require both database access and authentication.

That gives the API a consistent security model.

Instead of checking authentication manually in every procedure, Basango centralizes it through procedure composition:

publicProcedure
protectedProcedure

Authentication is exposed through login, refresh, and session procedures. Login validates user credentials, rejects locked accounts, verifies the password, and returns session tokens. Refresh validates a refresh token and issues a new session. Session returns the authenticated user for protected clients.

This matters because Basango is not only a public article feed. It also has user-specific features such as bookmarks, followed sources, comments, personalized feeds, dashboard access, and source administration.

Those features require a reliable identity layer.

API and Database Separation

The API delegates database work to query functions from @basango/db.

A procedure should usually be thin:

validate input
check authorization
call database query
return typed result

The article router calls functions such as:

createArticle()
getArticles()
getArticlesPublicationGraph()
getArticlesSourceDistribution()

The source router calls functions such as:

createSource()
updateSource()
getSources()
getSourceById()
getSourcePublicationGraph()
getSourceCategoryShares()

This separation keeps business data access logic outside the transport layer.

The API should not care about SQL details. The database package should not care whether the caller is tRPC, REST, a worker, or a CLI command.

That boundary keeps the codebase professional as it grows.

Validation Strategy

The API uses shared domain schemas from @basango/domain.

Validation belongs at the boundary. The crawler may send malformed article data. The dashboard may send invalid filters. A mobile client may call a procedure with an expired token. Every input crossing into the backend needs validation.

Basango avoids duplicating validation logic by keeping domain models and schemas in a shared package. The API imports those schemas and applies them at procedure boundaries.

The pattern is:

domain schema -> API input validation -> database query

This keeps the system consistent from ingestion to consumption.

REST Boundary for Ingestion

The crawler forwards normalized articles to the backend API after local persistence.

That is a deliberate design decision.

The crawler does not write directly to the database. It treats the API as the ingestion boundary.

This gives the system several advantages:

• the database stays private;
• ingestion can be authenticated;
• validation happens in one place;
• crawler failures are isolated from database access;
• future ingestion clients can reuse the same boundary.

Direct database writes from every internal component are tempting, but they create tight coupling and inconsistent validation. Basango avoids that by making the API the controlled entry point.

API Runtime Flow

Why the API Design Matters

The API gives Basango a professional integration boundary.

The crawler can evolve independently. The database schema can evolve behind query functions. The dashboard and mobile app can consume strongly typed procedures. Authentication, session resolution, CORS, secure headers, and request logging are centralized.

The main engineering decisions are:

• use REST where ingestion simplicity matters;
• use tRPC where first-party type safety matters;
• centralize authentication through protected procedures;
• keep database queries outside API route handlers;
• reuse domain schemas for input validation;
• expose analytical queries as product capabilities;
• keep the API thin, typed, and composable.

The result is an API that is not just a CRUD layer. It is a backend boundary for ingestion, reporting, authentication, search, and future personalization.

Consumers: Web Dashboard and Mobile App

Once the crawler, database, and API are in place, Basango needs product surfaces that make curated information usable.

The web and mobile applications are the consumption layer of Basango. They are not the core engineering problem, but they are essential because a curation system only matters if people can actually use the curated information.

Basango has two different consumer surfaces:

Dashboard -> operational monitoring and source management
Mobile    -> focused news consumption

That distinction matters.

A dashboard is for understanding the system. A mobile app is for benefiting from the system.

Open Basango dashboard sources viewOpen Basango dashboard articles viewOpen Basango dashboard overviewOpen Basango dashboard detail view

Web Dashboard

The web dashboard is the internal and operational surface of Basango.

It exposes capabilities such as:

total articles
total sources
active sources
publication trends
source distribution
category shares
article lists
source management
crawler outputs

The dashboard is built as a Next.js application and uses tRPC, TanStack Query, React Table, Recharts, Shadcn UI components, and shared domain packages.

The important engineering decision is that the dashboard is not just a UI over raw database tables. It consumes the same API contract as other clients.

It uses reporting procedures such as:

reports.getDashboardOverview
articles.getPublications
articles.getSourceDistribution
sources.getPublications
sources.getCategoryShares

That makes the dashboard a real product surface over backend capabilities.

Dashboard as an Observability Tool

For a crawler-based system, an admin dashboard is not optional.

Crawlers fail silently if nobody monitors them. A source can change its HTML structure. A WordPress API can stop responding. A date format can break. A source can suddenly produce far fewer articles than expected.

The dashboard helps answer operational questions:

Did the crawler collect articles today?
Which sources are active?
Which sources are overrepresented?
Are categories being assigned correctly?
Is a source producing mostly one kind of content?
Did publication volume change compared to the previous period?

This makes the dashboard part of the engineering system, not just a frontend layer.

A strong backend project needs feedback loops. The dashboard is one of those loops.

Mobile Application

The mobile app is the user-facing consumption surface.

Its role is different from the dashboard. The dashboard helps operate the system. The mobile app helps users stay informed without overconsuming information.

The mobile app is implemented as an Expo / React Native application using Expo Router and React Navigation.

The mobile product direction is:

personalized news feed
topic-specific reading
source following
saved articles
article details
search
recommendations
notifications

The database already contains structures that support this direction, including users, bookmarks, followed sources, comments, and authentication-related tables.

That is important because the data model anticipates the product direction instead of treating mobile as an afterthought.

Shared API Contract

Both web and mobile are designed to consume Basango through the API layer.

For first-party clients, tRPC provides a strong developer experience because the client can call backend procedures using shared TypeScript types. tRPC is designed for TypeScript applications and lets clients focus on calling procedures instead of manually maintaining HTTP route contracts.

This gives the clients a cleaner integration model:

api.articles.list()
api.sources.list()
api.reports.getDashboardOverview()
api.auth.login()
api.auth.session()

Instead of duplicating request and response types in every client, the server router becomes the contract.

That reduces integration bugs and makes refactoring safer. If an API input changes, the dashboard and mobile app receive type errors during development.

That is the correct failure mode.

Consumer Architecture

The consumer layer is organized around shared contracts:

packages/domain -> shared models and validation schemas
packages/ui     -> shared UI primitives
apps/api        -> typed backend contract
apps/dashboard  -> operational web client
apps/mobile     -> mobile news client

The web dashboard can focus on tables, charts, filters, and source administration.

The mobile app can focus on reading, saved articles, preferences, and notifications.

Both clients depend on the same backend concepts:

Article
Source
Category
User
Session
Bookmark
Report

That gives Basango product consistency across platforms.

Web vs Mobile Responsibilities

The dashboard should not try to be the mobile app. The mobile app should not try to be the dashboard.

That separation keeps the product clean.

The dashboard answers:

What is the system collecting?
Which sources are active?
How is the dataset evolving?
Are categories and sources healthy?

The mobile app answers:

What should I read now?
What happened about the topics I follow?
Which articles are relevant to me?
What should I save or revisit later?

This distinction prevents the product from becoming one overloaded interface.

The result is a product architecture where each layer has a clear responsibility. The clients do not own business logic; they consume capabilities exposed by the API. The API does not own query complexity; it delegates to the database package. The database does not know whether data is being used by the dashboard, mobile app, crawler, or future research tooling.

Key Engineering Decisions

Start with the real workflow, not the technology

The project started from a real information problem: staying accurately informed about DRC news, especially the eastern DRC conflict, without manually checking dozens of sources.

That problem shaped the architecture.

The goal was not “build a crawler” or “use an LLM.” The goal was to reduce the cost of staying informed.

That is why the system includes crawling, normalization, search, classification, dashboard reporting, and mobile consumption.

Use configuration for source variability

News websites change. HTML structures differ. Some sites expose WordPress APIs, others do not.

A configuration-driven crawler makes source onboarding and maintenance easier.

This was the correct abstraction because the variability lives in the sources, not in the core crawling workflow.

Keep ingestion idempotent

The crawler will retry. Pages will overlap. Sources will duplicate articles.

A hash-based deduplication strategy allows the crawler to run repeatedly without corrupting the dataset.

This is one of those small backend decisions that separates a prototype from a real system.

Use deterministic classification before LLM classification

The first classification layer is rule-based and candidate-driven.

That is not less sophisticated. It is more responsible.

LLMs can improve ambiguous classification later, but they should not be the first dependency in a system that needs to run continuously and cheaply.

Use PostgreSQL before adding search infrastructure

PostgreSQL already provides relational modeling, indexes, aggregation, and full-text search.

Adding Elasticsearch, Meilisearch, or a vector database too early would increase operational cost and complexity.

The better decision was to exhaust PostgreSQL first, then add specialized systems only when the product proves they are needed.

Keep API transport separate from database logic

The API exposes product capabilities.

The database package owns queries.

This keeps route handlers small and protects the system from turning into a pile of controller-level SQL.

Use tRPC for first-party clients

The web dashboard and mobile app are first-party TypeScript clients.

tRPC makes sense because it gives them a typed contract without code generation or duplicated DTOs.

For external integrations or ingestion, REST still makes sense.

Using both is not inconsistency. It is using the right interface for the right consumer.

Build the dashboard as an engineering feedback loop

The dashboard is not only for users. It helps operate the crawler and evaluate data quality.

For a data ingestion system, this is essential.

A crawler without monitoring is a system that can fail quietly.

What I Learned

The biggest lesson from Basango is that a serious aggregation system is not about downloading pages.

The hard part is building a reliable path from fragmented public information to structured, trustworthy, searchable, and usable knowledge.

Crawling must be treated as an unreliable process. Websites change, metadata is inconsistent, pagination is not always clean, and source categories cannot be trusted as a product taxonomy.

I also learned that AI should be introduced carefully. It is tempting to solve classification, recommendation, and semantic search with LLM calls immediately, but that can make the system expensive and hard to debug. A better approach is to build deterministic layers first, preserve metadata, collect enough data, and then introduce AI where it adds measurable value.

The database became more important than expected. Once articles were collected, the key challenge became retrieval: deduplication, indexing, filtering, category normalization, reporting, and future personalization. That pushed the project from crawler to information system.

The API layer also became a major design boundary. By separating REST ingestion from tRPC client access, Basango supports both machine workflows and first-party application development without forcing one interface style everywhere.

Finally, dashboarding is not a cosmetic layer. For systems that depend on background jobs and external websites, a dashboard is part of reliability engineering. It shows whether the pipeline is healthy, whether data is fresh, and whether the source strategy is working.

Conclusion

Basango is a scalable news curation system for Congolese media.

Its main engineering value is not that it crawls news websites. The value is the architecture around the crawling: source abstraction, idempotent ingestion, typed data modeling, low-cost classification, indexed retrieval, API boundaries, analytics, and a product path toward semantic search and recommendation.

The system turns a fragmented media ecosystem into a structured pipeline where articles can be collected, normalized, deduplicated, classified, searched, analyzed, and eventually recommended.

That is the core idea behind Basango: make critical Congolese news easier to monitor, evaluate, and consume.