Designing Realtime Abuse Master

The whiteboard for this problem is in excalidraw/4. Designing Realtime Abuse Master.excalidraw. This file is the full write up of that board.

1. Problem Statement

One video live stream is powered by one server. Every participant of that stream, up to a maximum of 100, is connected to the same server over a WebSocket. Participants can send text messages that are broadcast to everyone in the stream.

We want abusive words masked before the broadcast goes out, so shit becomes ****. The creator's video comes in over RTMP, but video is not what we are filtering. We only care about the chat messages, because that is where abuse happens.

2. Requirements

2.1 Functional Requirements

2.2 Non Functional Requirements

1. Very low latency. Masking must add well under 10 ms. Chat that lags behind the video feels broken.
2. No extra network hop per message. At 100 participants each sending messages, a remote call per message is not affordable.
3. Available. If the dictionary cannot be fetched, the server must still start and still serve chat.
4. Scales by stream. Thousands of concurrent streams, each on its own server process, with no shared bottleneck on the hot path.
5. Fast dictionary updates. A newly banned word should be live everywhere within a minute or two.

2.3 Out of Scope

Image and audio moderation, video frame analysis, machine learning based intent detection. Those are follow ups in section 11.

3. Understanding the Architecture

• The creator's stream is captured over RTMP.
• Participants connect to the same server over a WebSocket.
• A participant can send a message to everyone.

Because every message in one stream passes through the same server, that server is the natural place to filter. There is no need to coordinate with anything else to moderate a single stream.

3.1 How Socket.IO Fits

• Socket.IO gives real time client to server communication.
• Clients connect over WebSocket, with a fallback transport if needed.
• Server and clients talk in named events, so chat:send and chat:message are separate channels of meaning.
• Rooms group clients. One stream is one room.
• The server can send to one client, to everyone, or to one room.

Rooms are what make broadcast cheap. io.to(streamId).emit(...) reaches exactly the participants of that stream.

4. The Abuse Dictionary

Assume the list of banned words lives in a text file in blob storage such as S3, at a known path. When a live stream server starts up it:

1. Downloads the abuse file.
2. Loads it into memory.

The question is which data structure.

4.1 Approach 1: Tokenize and Look Up

1. Load all words into a hash set.
2. Tokenize the incoming message on whitespace and punctuation.
3. For each token, check membership in the set.
4. On a hit, replace the token with asterisks. On a miss, copy it through.

Cost is O(n) in message length with O(1) average lookups, and it is about ten lines of code. Good enough to ship.

Where it falls short:

• It only catches exact whole tokens. shitty is missed because the token does not equal shit.
• Substring search fixes that but then you are running the whole dictionary against every message, which is dictionary size times message length.
• It cannot catch a phrase spread across tokens.

4.2 Approach 2: Trie

Build a trie of the banned words once at startup, then walk the message a single character at a time and follow trie edges.

        root
       /    \
      s      f
      |      |
      h      u
      |      |
      i      c
      |      |
      t      k
    (end)  (end)

Properties:

• One pass over the message, and the cost per character does not depend on how many words are in the dictionary.
• Prefix matching comes free, so shitty is caught as soon as the walk reaches an end node.
• Multi word phrases work by keeping spaces as edges.
• An Aho-Corasick automaton, which is a trie plus failure links, finds every match in one pass with no backtracking at all. That is the production version of this idea.

The trie is what we put in the live stream server.

4.3 Handling Evasion

Before the lookup, normalize:

Normalization runs on a copy. We mask by index range in the original string, so the user still sees their own punctuation and capitalization everywhere the text was clean.

5. Why Not a Separate Moderation Service

The tempting design is a POST /clean-text microservice that the stream server calls per message. Think about it realistically.

At 100 participants per stream and thousands of streams, that is a huge amount of avoidable traffic to answer a question that a local trie answers in microseconds. Creating a service to clean the text is a bad idea.

The better approach: the live stream server itself downloads the abuse dictionary from S3, builds the trie in memory, and broadcasts the masked message.

The dictionary is small, a few thousand words is well under a megabyte, and every server holding a full copy is completely fine. This is a read mostly, tiny, shared configuration, which is exactly the kind of data you replicate everywhere instead of centralizing.

6. Configuring the Abuse List

We need a way for an internal team to maintain the list.

• A UI where the current words are listed.
• The ability to add or remove words and publish them to S3.

That is the abuse admin service.

The database is the source of truth for editing, with history and who changed what. The S3 file is the compiled snapshot that servers actually read. Editing and serving are separated on purpose: a thousand servers pulling one immutable S3 object is trivial, a thousand servers querying one database is not.

Two ways for a running server to learn about an update:

1. Poll. Fetch the version marker every 60 seconds, and re download only if it changed. Simple, no extra infrastructure, worst case one minute stale.
2. Push. The admin service publishes a message on a pub sub channel that every server subscribes to, and servers re download immediately. Faster, needs a broker.

Poll is the sane default, and push is worth adding once minutes of staleness starts mattering.

Rebuilding the trie happens off the request path. The server builds a new trie into a separate variable and swaps the reference when it is ready, so no message ever waits on a rebuild.

7. API Design

7.1 Admin REST API

Add a word:

{
  "word": "badword",
  "language": "en",
  "severity": "HIGH",
  "action": "MASK",
  "matchType": "SUBSTRING"
}

Publish response:

{
  "version": 42,
  "wordCount": 3184,
  "s3Key": "abuse/lists/v42.txt",
  "checksum": "sha256:9f2a...",
  "publishedAt": "2026-08-19T10:31:00Z"
}

7.2 Server to S3 Contract

The pointer file is tiny, so polling it every minute costs nothing. The list files are immutable and versioned, which means a rollback is just pointing current.json back at v41.

7.3 WebSocket Events

The sender gets chat:message with their own masked text too. If we showed the sender the raw version and everyone else the masked version, the sender would never learn the word is banned and would keep trying.

8. Data Model

Two very different shapes here: a small relational set of rules, and a high volume append only log of what was flagged.

8.1 Relational Tables

abuse_words

Severity drives what happens: MASK replaces the word, BLOCK drops the whole message, FLAG_ONLY broadcasts it unchanged but records it for review.

abuse_list_versions

allowlist_terms

The allowlist exists because substring matching produces embarrassing false positives. Place names and ordinary words that contain a banned substring go here.

streams

user_strikes

Strikes are scoped per stream, so one bad night does not follow a user forever. The live counters sit in Redis for speed and get flushed here for durability.

8.2 Append Only Log

flagged_messages (Cassandra, DynamoDB or Elasticsearch)

Storing list_version is what makes tuning possible later. When someone asks why a message was masked, the answer is reproducible because we know exactly which dictionary was live.

8.3 Relationships

In words:

• One internal user maintains many words and publishes many versions.
• One published version explains many flagged messages, each flagged message points at exactly one version.
• One user creates many streams, one stream has one creator.
• One stream contains many flagged messages and many strike rows.
• One user accumulates one strike row per stream they participate in.

Note that abuse_words and flagged_messages are not directly joined. The snapshot version sits between them, because the words table changes and the log must stay explainable.

9. Key Flows

9.1 Server Startup

If the S3 fetch fails, the server loads the last good copy from local disk. If there is no local copy either, it loads a small dictionary bundled with the build and logs loudly. The server always starts. Chat with a stale filter is much better than no chat.

9.2 A Message Being Broadcast

The log write is asynchronous and fire and forget. Nothing about broadcasting a message should wait on an audit write.

9.3 Publishing a New List

Order matters. The word list file is uploaded before the pointer file. A server polling in between sees the old pointer and the old list, which is consistent. The reverse order would briefly point at a file that does not exist yet.

10. Failure Handling

The pattern across this table: masking depends only on process local state, so almost nothing outside the process can stop chat from being filtered.

11. Tradeoffs

12. Exercise: Build a Small Version

From the board, a scoped down project you can actually write:

Moderated real time chat

1. Socket.IO server. A Node.js server handling many concurrent connections and broadcasting messages.
2. Dynamic data fetching. On startup, fetch a list of prohibited words from a URL using an HTTP client.
3. Optimal storage. Store the words in a hash set for constant average time lookup, then upgrade to a trie.
4. Linear time masking. Walk the message once, check candidates against the set, replace matches with asterisks.
5. Event cycle. Client sends a raw message, the server intercepts and filters it, the server broadcasts the sanitized message to all clients.
6. Error handling. If the fetch fails, fall back to a bundled list so the server still starts.

13. Future Work

• Aho-Corasick instead of a plain trie, for guaranteed single pass matching of all patterns.
• Per language dictionaries selected by the stream's locale.
• Context aware classification. A small model scoring intent, since word lists cannot tell insult from quotation.
• Image and audio moderation for shared media and creator speech, which is asynchronous work and belongs in a queue and worker pipeline, not in the hot path.
• Creator controls. Per stream strictness, and creator managed extra words.
• Feedback loop. Mine flagged_messages for near misses and evasion patterns, and feed them back into the dictionary.

14. Interview Talking Points

• Say early that every message in one stream already passes through one server. That is what justifies local filtering.
• Reject the moderation microservice explicitly, with the handshake and teardown cost as the reason. This is the main insight in the problem.
• Move from hash set to trie by naming the concrete case the hash set misses, which is shitty versus shit.
• The dictionary is small, read mostly configuration, so replicating it to every server is correct rather than wasteful.
• Upload the list before the pointer. Ordering details like this are what separate a design that works from one that mostly works.
• Fail open on the dictionary and fail safe on the trie swap. Chat must never go down because a word list could not be fetched.