Papa Labs

Evaluating email security without being led by the vendor: a complete POV methodology

2 min read #email#security#poc#methodology 中文

We evaluated adding a layer of integrated cloud email security (KnowBe4 Defend — the API-into-M365 kind) on top of the existing gateway. Vendor demos are always beautiful; this post records how to run the POV (Proof of Value) as your own evaluation rather than the vendor’s roadshow.

Benchmark with real attack profiles

The attack profile the vendor dissected in the POV sessions matched what actually lands in our mailboxes — the BEC/impersonation class that traditional gateways struggle with most:

  1. Sent from a personal mailbox (gmail and friends: domain reputation fine, SPF/DKIM all pass);
  2. Impersonated signature (body styled as a supplier or colleague);
  3. Channel switching (“let’s continue on this address” / “our bank account changed”) — phishing the process trust, not the technology.

The product’s recommended action for these: straight to M365 quarantine, denying users any chance to interact with malicious content. Build your test set from these three profiles — not EICAR files the old gateway already catches. When evaluating a new layer, feed it what the old layer misses.

Layered email security and the POV checkpoints

The new product patches the gap behind the gateway — every POV checkpoint should aim at that gap

Three steps worth stealing

  1. Banner pilot: warning banners on suspicious mail go to pilot users first — banners cause cry-wolf fatigue when noisy, so gather the false-positive rate in the pilot before going wide;
  2. Quarantine-and-restore drill: deliberately experience the false-quarantine recovery flow (remediation / restore) — how many admin steps, can users self-appeal, how long until restored. This rehearses the ongoing operational cost, which shapes long-term experience more than detection rate does;
  3. A defined conclusion (Conclude POV): the POV must end with outputs — detection data, false-positive data, operational cost assessment, then go/no-go. A POC without exit criteria becomes an indefinite free trial with indefinite sales follow-up.

Layering vs replacing

Integrated (API into M365) and gateway (MX redirect) are different architectures: the former sees internal mail and post-delivery behavior; the latter filters before delivery. The core POV question isn’t “which is better” but:

Of what the new product catches, how much did the existing gateway miss? (incremental value) Do the combined false positives and operational load justify that increment?

Pull both layers’ catch lists for the POV period and the math answers itself.

Lessons

  1. Whoever controls the test samples controls the POV: test with attacks that actually got through your environment — vendor demo samples only prove the product catches its own targets;
  2. False-positive handling (the restore flow) is the systematically underrated dimension — detection rate decides whether you can buy it; false-positive operations decide whether you’ll regret it;
  3. Put a deadline and exit criteria for every POC in the kickoff email — it’s self-defense for your calendar.
← All posts