How we measure

We publish a number only when it traces to a measurement we can reproduce — stated with its baseline, caveated where it's preliminary, and never rounded in our favour. Everything below is from our own profiling on a single test machine. It's labelled as such, it's a conservative floor rather than a best case, and where we haven't earned a number yet we say so instead of guessing.

The agent, measured

We profile Edgepacer the way you'd audit any process: a generator streams synthetic log lines across many concurrent sources, the agent collects and ships them, and once a second we record the delivered line rate, the bytes on the wire, process CPU (kernel clock ticks) and resident memory (kernel smaps). At idle it sits at roughly 10 MB of memory and under 0.1% CPU — effectively zero measurable CPU over a 60-second window. Under a stress load it sustained about 500,000 lines per second (~100 MB/s on the wire) on under one CPU core; that 500k is the generator's ceiling, not the agent's — it never touched more than one of the machine's twelve cores — so read it as a floor. These runs predate later optimization work, which makes them conservative, not flattering. They're internal measurements, not production-at-scale; when we've measured Edgepacer in production we'll publish those numbers and exactly how we took them.

Compression, on the hardest public data we could find

We measure compression against the standard public log-parsing benchmark corpora (LogHub / LogPAI) on purpose: those datasets are curated for format diversity — thousands of distinct templates — which is close to a worst case for compression. Real production logs are the opposite, a handful of templates repeated endlessly, so they compress further. Publishing the benchmark floor keeps us honest.

Across all fourteen full corpora — 6.68 GB of raw logs — we land at 15.8× smaller (down to 424 MB), measured the hard way: your original raw bytes in, to fully structured and every-field-searchable bytes out, index included. Per corpus it ranges from about 6.3× on the most irregular data to about 21.7× on the most uniform; Thunderbird's 2.5 GB compresses 18.6×. Counted the way the industry usually reports compression — from already-structured data — the same result is 31.4×; we lead with the raw number because it's the disk you actually save. Every figure here re-derives from the stored shards on demand.

How far does "further" go? One specific real-world IIS access-log set — 736,058 lines, 214 MB — compresses 112× raw (202× counted the industry way) on the same engine, because real logs repeat far more than the benchmark's deliberately-diverse mix. That's a single labelled datapoint, not part of the 14-corpus aggregate above. We still headline the benchmark floor; your ratio tracks how repetitive your data is.

Search, honestly

Logs are structured records — time, host, service, severity and code are columns, not text you grep — so the normal query is a structured-field filter, and that's where it flies. On a 16.6-million-event corpus, equality filters over structured fields return in 15–17 ms median server time through the query daemon. Counting by a one-hour time window comes back in about 138 ms. Broader tail/message text scans — the fallback, not the normal path — run longer, from a few hundred milliseconds into the low seconds. We don't claim blanket "millisecond search"; we claim fast structured filters, measured server-side over 30 runs against raw-corpus oracle counts, on a single test machine.

Structuring, measured

The analyzer structures formats it has never seen — no regex, no grok, no templates — and re-derives the structure when a format drifts. Run across the sixteen public benchmark corpora it structured 99.99% of entries (three parse failures in ~31,900), deterministically; on 12 of the 16 it recovered the exact field breakdown of the hand-curated ground truth those datasets ship with, the other four off by a single reasonable boundary. It reassembles multi-line events into single records, too — on one corpus it folds 142 continuation lines back into their parent entries. To be precise about what this is: it measures structuring — recovering each line's fields — not a parsing-accuracy score against a template library; and it re-runs on demand from the same binary.

Why the agent is light and the data is small

The agent ships raw data and nothing else; the heavy work — structuring every field, compressing, indexing — happens downstream, where those three are co-designed to feed one another. The analyzer's structure is what lets the store compress and index every field at once, instead of forcing the index-everything-versus-keep-it-small trade-off you hit when those layers are bolted together from separate parts. That co-design is the mechanism behind the numbers above.