JavaScript: a naive Bayes text classifier

Classify short messages as spam or ham from word counts with Laplace smoothing.

const training = [
  ["spam", "win money now"],
  ["spam", "win a prize now"],
  ["ham", "meeting at noon"],
  ["ham", "lunch at noon now"],
];

const counts = { spam: new Map(), ham: new Map() };
const docs = { spam: 0, ham: 0 };
const vocabulary = new Set();
for (const [label, text] of training) {
  docs[label]++;
  for (const word of text.split(" ")) {
    vocabulary.add(word);
    counts[label].set(word, (counts[label].get(word) ?? 0) + 1);
  }
}

function score(label, text) {
  const total = [...counts[label].values()].reduce((a, b) => a + b, 0);
  let logProbability = Math.log(docs[label] / training.length);
  for (const word of text.split(" ")) {
    const seen = counts[label].get(word) ?? 0;
    logProbability += Math.log((seen + 1) / (total + vocabulary.size));
  }
  return logProbability;
}

for (const message of ["win prize now", "lunch meeting"]) {
  const spam = score("spam", message);
  const ham = score("ham", message);
  console.log(message + " -> " + (spam > ham ? "spam" : "ham") + " (spam " + spam.toFixed(2) + ", ham " + ham.toFixed(2) + ")");
}

win prize now -> spam (spam -6.12, ham -8.32)
lunch meeting -> ham (spam -6.24, ham -4.85)

Explanation

Naive Bayes classifies by Bayes' rule. The posterior probability of a class is proportional to its prior times the probability of the observed words given the class. The naive assumption is that words are independent given the class. It is false for real language, but the classifier often works well anyway.

Probabilities are multiplied for every word, so to avoid underflow the program adds their logarithms. Laplace smoothing adds 1 to each word count, so a word never seen in a class does not make its whole probability zero. The class with the highest log score wins.

—

—

—

win prize now -> spam (spam -6.12, ham -8.32)
lunch meeting -> ham (spam -6.24, ham -4.85)

Cost and common mistakes

Training is O(total words); scoring a message is O(its words).

• Without smoothing, one unseen word gives a probability of 0 and the logarithm of 0 is -Infinity.
• The scores are log probabilities, so they are negative. A higher (less negative) score means a more likely class.
• Four training messages are far too few for real use. Real filters train on thousands of messages and handle punctuation and case.

Try it yourself

What would happen to the prior if the training set had three spam messages and one ham?

Further reading

• MDN JavaScript guide (external)
• Node.js API documentation (external)
• Mathematics for Machine Learning (external)