CouchDB3.4.3versus MongoDB8.3.8

# CouchDB vs MongoDB

Six experiments on one FAO crop-production dataset: bulk and single-document upload, queries and a
whole-collection update, document validation, users and access, change feeds, and tooling.
A Big Data 5410H course project (April 2024) rebuilt in 2026 so every number is reproducible and the
results page plays them back as timed races instead of bar charts.

Everything needed to run it lives in this folder: the code, the five CSVs, the database files, and a
tarball of the two Docker images. Nothing is fetched from the network at run time; the site loads its
font and Chart.js from its own files.

## Requirements

- Docker Desktop (Docker 27 or newer with `docker compose`)
- Python 3.12 or newer
- Node 22 or newer, only for the page's unit tests

## Run it

```
cp .env.example .env                       # CouchDB admin password; also read by bench.py
docker load -i docker/images.tar.gz        # optional: restores mongo:8 and couchdb:3.4 offline
docker compose up -d                       # mongo:8 (single-node replica set) + couchdb:3.4
python -m venv .venv && .venv/Scripts/pip install -r requirements.txt   # Linux/macOS: .venv/bin/pip
.venv/Scripts/python bench.py all          # about 30 minutes: every experiment x every continent
python -m http.server -d site              # open http://localhost:8000
```

Skip `docker load` when the machine is online; `docker compose up` pulls the same two images.
Both databases listen on localhost only.

Single pieces:

```
python bench.py load   --db couch --continent africa --runs 5
python bench.py query  --db mongo
python bench.py watch
python bench.py explore          # site/explore_<continent>.json for the data explorer
python bench.py selftest         # no Docker needed
node --test site/lib.test.mjs    # the page logic, 8 tests
```

Every subcommand merges into `results.json` and copies it to `site/`, so partial runs show partial
races.

## What is measured

- Wall-clock time from the client, median of `--runs` after `--warmup` runs; the first cold run is
  kept as `cold_ms`.
- Server CPU (% of one core) and memory (MB) read from the container's cgroup before and after each
  operation: the database process only, not the Python client.
- On-disk size after the bulk load with caller-supplied `_id`.
- Change-feed delivery: documents received of 100 inserted, and median insert-to-listener latency.
- Validation: did the rule reject the bad document and accept the good one.

The committed `results.json` was measured on 2026-09-07 with the database files in `dbdata/`.
MongoDB's memory column grows through a run because WiredTiger keeps its cache; CouchDB's does not.

## Layout

```
bench.py             the whole benchmark, one file
docker-compose.yml   the two databases, data bind-mounted from dbdata/
results.json         latest run (committed); site/results.json is a copy
site/                static page: index.html, style.css, app.js, lib.js + lib.test.mjs,
                     chart.umd.min.js (Chart.js 4.5.0), fonts/, results and explorer JSON
embed.py             pastes bench.py, compose, requirements and .env.example into the page's
                     Reproduce section; run it after editing any of them
data/                the five FAO CSVs, 38,146 rows (gitignored, see data/README.md)
dbdata/              MongoDB and CouchDB data directories (gitignored, ~750 MB)
docker/              images.tar.gz, both images for offline use (gitignored, ~450 MB)
.venv/               Python environment (gitignored)
.claude/skills/      mongodb (official), mongodb-expert, couchdb, hunt-dom + hunt-xss
```

`git clone` gives the code and results only. Copy `data/`, `dbdata/` and `docker/` alongside it for
the full offline package.

## Deploy

`site/` is static with no build step and no external requests. Upload the folder to any static host.

## Credits

Prerna Gurbaxani and Kharanshu Naghera, original project and report. Data from FAO via
[data.world/agriculture/crop-production](https://data.world/agriculture/crop-production).

# Data

FAO crop production statistics, 1961–2019, one CSV per continent:

```
production_crops_e_africa.csv
production_crops_e_americas.csv
production_crops_e_asia.csv
production_crops_e_europe.csv
production_crops_e_oceania.csv
```

Source: FAOSTAT "Crops and livestock products" as packaged on data.world
(https://data.world/agriculture/crop-production). Drop the CSVs in this folder
(gitignored, ~28 MB, 38,146 rows). `bench.py` reads them directly; no conversion step.

Columns: `area_code, area, item_code, item, element_code, element, unit`, then
`y1961 … y2019` values and `y1961f … y2019f` FAO flags (`F` estimate, `Fc` calculated,
`Im` imputed, `M` missing, blank = official).

# copy to .env — read by docker compose (CouchDB admin) and by bench.py (connection URLs)
COUCHDB_USER=admin
COUCHDB_PASSWORD=password
COUCH_URL=http://admin:password@localhost:5984
MONGO_URL=mongodb://localhost:27017/?directConnection=true

pymongo>=4.10
requests>=2.32
psutil>=6

services:
  mongo:
    image: mongo:8
    container_name: cvm-mongo
    ports: ["127.0.0.1:27017:27017"]
    # single-node replica set: change streams (experiment 5) need one.
    # No auth: local benchmark box only. bench.py runs rs.initiate() on first use.
    command: ["--replSet", "rs0", "--bind_ip_all"]
    volumes: ["./dbdata/mongo:/data/db"]   # data lives in the repo folder, not in a Docker volume
    healthcheck:   # so `docker compose up -d --wait` returns only once the DB answers
      test: ["CMD", "mongosh", "--quiet", "--eval", "db.adminCommand('ping').ok"]
      interval: 5s
      timeout: 3s
      retries: 20

  couchdb:
    image: couchdb:3.4
    container_name: cvm-couch
    ports: ["127.0.0.1:5984:5984"]
    env_file: .env   # COUCHDB_USER / COUCHDB_PASSWORD (CouchDB 3 refuses to start without an admin)
    volumes: ["./dbdata/couch:/opt/couchdb/data"]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5984/_up"]
      interval: 5s
      timeout: 3s
      retries: 20

#!/usr/bin/env python3
"""CouchDB vs MongoDB on the FAO crop-production CSVs. One file.

    docker compose up -d
    pip install -r requirements.txt
    python bench.py all                # every experiment, every continent -> results.json + site/
    python bench.py load --db couch --continent africa --runs 5
    python bench.py explore            # site/explore_<continent>.json for the data explorer
    python bench.py selftest
"""
import argparse, csv, json, os, shutil, statistics, subprocess, threading, time
from pathlib import Path

import psutil, requests
from pymongo import MongoClient
from pymongo.errors import WriteError

ROOT = Path(__file__).resolve().parent
DATA, SITE, RESULTS = ROOT / "data", ROOT / "site", ROOT / "results.json"
CONTINENTS = ["africa", "americas", "asia", "europe", "oceania"]
CONTAINER = {"mongo": "cvm-mongo", "couch": "cvm-couch"}
ONE_LIMIT = 1000   # one-by-one inserts stop here: 38K single HTTP posts is a lunch break, not a benchmark
CHUNK = 2000       # docs per CouchDB _bulk_docs request
ELEMENTS = ["Area harvested", "Production", "Yield"]
GOOD, BAD = {"area": "Test", "item": "Test", "element": "Yield"}, {"area": "Test", "item": "Test", "element": "Bogus"}
RUNS, WARMUP = 3, 1

for line in (ROOT / ".env").read_text().splitlines() if (ROOT / ".env").exists() else []:
    if "=" in line and not line.startswith("#"):
        k, v = line.split("=", 1); os.environ.setdefault(k.strip(), v.strip())
MONGO_URL = os.environ.get("MONGO_URL", "mongodb://localhost:27017/?directConnection=true")
COUCH_URL = os.environ.get("COUCH_URL", "http://admin:password@localhost:5984").rstrip("/")

# ---------------------------------------------------------------- data
def rows(continent):
    with open(DATA / f"production_crops_e_{continent}.csv", encoding="utf-8", newline="") as f:
        return [dict(r, continent=continent) for r in csv.DictReader(f)]   # values stay strings, like the 2024 run

def with_ids(docs, continent):
    return [dict(d, _id=f"{continent}_{i}") for i, d in enumerate(docs)]

def pick(base):
    """(area, [item, item]) for the query experiments: first area with two crops that have production data."""
    for area in dict.fromkeys(r["area"] for r in base):
        items = list(dict.fromkeys(r["item"] for r in base if r["area"] == area and r["element"] == "Production" and r["y2005"]))
        if len(items) >= 2:
            return area, items[:2]
    return base[0]["area"], []   # nothing usable: the queries would run empty, but do not crash the run

# ---------------------------------------------------------------- clients
mongo = MongoClient(MONGO_URL, serverSelectionTimeoutMS=5000)
couch = requests.Session()   # keep-alive, so HTTP gets the same connection reuse pymongo's pool gets

def mcol(c): return mongo["cvm"][c]
def cdb(c): return f"{COUCH_URL}/cvm_{c}"

def ensure_replset():
    try:
        mongo.admin.command("replSetGetStatus")
    except Exception:
        mongo.admin.command("replSetInitiate", {"_id": "rs0", "members": [{"_id": 0, "host": "localhost:27017"}]})
        deadline = time.time() + 60
        while not mongo.is_primary:
            if time.time() > deadline:
                raise RuntimeError("mongo did not become primary within 60s after replSetInitiate")
            time.sleep(0.5)

def reset(db, c):
    if db == "mongo":
        mcol(c).drop()
    else:
        couch.delete(cdb(c)); couch.put(cdb(c)).raise_for_status()

def versions():
    def v(fn):
        try:
            return fn()
        except Exception:
            return None   # a DB being down must not cost us the results file
    return {"mongo": v(lambda: mongo.server_info()["version"]), "couch": v(lambda: couch.get(COUCH_URL).json()["version"])}

# ---------------------------------------------------------------- measurement
def parse_cgroup(words):
    """cgroup v2 (`cpu.stat` + `memory.current`) or v1 (`cpuacct.usage` ns + `memory.usage_in_bytes`) -> (cpu_s, mem_mb)."""
    if "usage_usec" in words:
        return int(words[words.index("usage_usec") + 1]) / 1e6, int(words[-1]) / 2**20
    return int(words[0]) / 1e9, int(words[1]) / 2**20

def cgroup(db):
    """Cumulative CPU seconds and current memory of the DB container; (None, None) when docker is not around."""
    sh = ("cat /sys/fs/cgroup/cpu.stat /sys/fs/cgroup/memory.current 2>/dev/null"
          " || cat /sys/fs/cgroup/cpuacct/cpuacct.usage /sys/fs/cgroup/memory/memory.usage_in_bytes")
    try:
        out = subprocess.run(["docker", "exec", CONTAINER[db], "sh", "-c", sh], capture_output=True, text=True, timeout=10)
        return parse_cgroup(out.stdout.split())
    except Exception:
        return None, None

IDLE = {}          # CPU seconds one cgroup sample pair costs the container by itself (sh + cat), per db
CPU_MIN_S = 0.25   # below this the sampling overhead swamps the signal; cpu_pct is left out

def calibrate(db):
    if db not in IDLE and cgroup(db)[0] is not None:
        pairs = [(cgroup(db)[0], cgroup(db)[0]) for _ in range(3)]
        IDLE[db] = statistics.median(b - a for a, b in pairs)

def measure(db, fn, setup=None):
    """Median wall-clock ms of fn over RUNS after WARMUP; server CPU (% of one core) and RSS from the container cgroup."""
    ms, cpu, mem = [], [], []
    calibrate(db)
    for _ in range(WARMUP + RUNS):
        if setup: setup()
        c0, _ = cgroup(db)
        t0 = time.perf_counter(); fn(); t = time.perf_counter() - t0
        c1, m = cgroup(db)
        ms.append(t * 1000)
        if c0 is not None and c1 is not None:
            cpu.append(max(0.0, c1 - c0 - IDLE.get(db, 0.0)) / t * 100); mem.append(m)
        else:
            cpu.append(None); mem.append(None)   # keep the lists aligned with ms, the WARMUP slice depends on it
    r = {"ms": round(statistics.median(ms[WARMUP:]), 1), "cold_ms": round(ms[0], 1)}
    mem_ok, cpu_ok = [x for x in mem if x is not None], [x for x in cpu[WARMUP:] if x is not None]
    if mem_ok:
        r["mem_mb"] = round(max(mem_ok), 1)
        if cpu_ok and statistics.median(ms[WARMUP:]) >= CPU_MIN_S * 1000:
            r["cpu_pct"] = round(statistics.median(cpu_ok), 1)
    return r

# ---------------------------------------------------------------- experiments
def load(db, c, docs, mode="bulk"):
    if db == "mongo":
        if mode == "bulk":
            mcol(c).insert_many(docs, ordered=False)
        else:
            for d in docs: mcol(c).insert_one(d)
    elif mode == "bulk":
        for i in range(0, len(docs), CHUNK):
            couch.post(f"{cdb(c)}/_bulk_docs", json={"docs": docs[i:i + CHUNK]}).raise_for_status()
    else:
        for d in docs: couch.post(cdb(c), json=d).raise_for_status()

def size_mb(db, c):
    if db == "mongo":
        mongo.admin.command("fsync", lock=False)   # checkpoint, else storageSize is whatever the last one saw
        s = mongo["cvm"].command("collstats", c)
        return round((s["storageSize"] + s["totalIndexSize"]) / 2**20, 2)
    return round(couch.get(cdb(c)).json()["sizes"]["file"] / 2**20, 2)

def exp_load(db, c, base):
    out, batch = {}, []
    # order matters: the with-_id bulk load runs last and stays in place for the other experiments
    for key, docs, mode in [("upload_bulk", base, "bulk"), (f"upload_one_{ONE_LIMIT}", base[:ONE_LIMIT], "one"),
                            ("upload_bulk_id", with_ids(base, c), "bulk")]:
        def setup():
            reset(db, c); batch[:] = [dict(d) for d in docs]      # fresh copies: insert_many stamps _id into dicts
        out[key] = measure(db, lambda: load(db, c, batch, mode), setup)
    out["size_mb"] = size_mb(db, c)
    return out

def exp_query(db, c, base):
    area, items = pick(base)
    simple, out = {"area": area}, {}
    if db == "mongo":
        col = mcol(c)
        try:
            col.drop_index("area_1")   # a previous run's index would make the "unindexed" number a lie
        except Exception:
            pass
        out["query_simple"] = measure(db, lambda: list(col.find(simple)))
        col.create_index("area")
        out["query_simple_indexed"] = measure(db, lambda: list(col.find(simple)))
        pipe = [{"$match": {"area": area, "item": {"$in": items}, "element": "Production"}},
                {"$group": {"_id": None, "total": {"$sum": {"$toDouble": {"$cond": [{"$eq": ["$y2005", ""]}, "0", "$y2005"]}}}}}]
        out["query_aggregate"] = measure(db, lambda: list(col.aggregate(pipe)))
    else:
        find = lambda **kw: couch.post(f"{cdb(c)}/_find", json={"selector": simple, "limit": 10**6, **kw}).json()["docs"]
        for ix in couch.get(f"{cdb(c)}/_index").json()["indexes"]:   # same reason: leftover index from a previous run
            if ix["name"] == "area" and ix.get("ddoc"):
                couch.delete(f"{cdb(c)}/_index/{ix['ddoc']}/json/area")
        out["query_simple"] = measure(db, find)
        couch.post(f"{cdb(c)}/_index", json={"index": {"fields": ["area"]}, "name": "area"}).raise_for_status()
        out["query_simple_indexed"] = measure(db, lambda: find(use_index="area"))
        view = {"views": {"total": {"reduce": "_sum", "map":
                f"function(d){{ if(d.area==={json.dumps(area)} && d.element==='Production' && {json.dumps(items)}.indexOf(d.item)>=0)"
                f" emit('total', parseFloat(d.y2005)||0); }}"}}}
        couch.put(f"{cdb(c)}/_design/agg", json=view).raise_for_status()
        out["query_aggregate"] = measure(db, lambda: couch.get(f"{cdb(c)}/_design/agg/_view/total").json())
    return out

def exp_update(db, c, base):
    """Experiment 2b: y2005 string -> number, blanks -> 0, across the whole collection."""
    def setup():
        reset(db, c); load(db, c, with_ids(base, c))
    if db == "mongo":
        pipe = [{"$set": {"y2005": {"$toDouble": {"$cond": [{"$eq": ["$y2005", ""]}, "0", "$y2005"]}}}}]
        fn = lambda: mcol(c).update_many({}, pipe)
    else:
        def fn():   # CouchDB has no server-side update: fetch everything, change it, write it back
            docs = [r["doc"] for r in couch.get(f"{cdb(c)}/_all_docs", params={"include_docs": "true"}).json()["rows"]]
            for d in docs: d["y2005"] = float(d["y2005"] or 0)
            for i in range(0, len(docs), CHUNK):
                couch.post(f"{cdb(c)}/_bulk_docs", json={"docs": docs[i:i + CHUNK]}).raise_for_status()
    return {"update_y2005": measure(db, fn, setup)}

def exp_validate(db, c):
    t0 = time.perf_counter()
    if db == "mongo":
        mongo["cvm"].command("collMod", c, validator={"$jsonSchema": {"required": ["element"], "properties": {"element": {"enum": ELEMENTS}}}},
                             validationLevel="strict", validationAction="error")
        def put(d):
            try:
                mcol(c).insert_one(dict(d)); return True
            except WriteError:   # only the validator saying no; connection errors must still blow up
                return False
    else:
        fn = ("function(newDoc, oldDoc, userCtx){ if(newDoc._deleted) return;"
              f" if({json.dumps(ELEMENTS)}.indexOf(newDoc.element) < 0) throw({{forbidden: 'element must be one of {', '.join(ELEMENTS)}'}}); }}")
        couch.put(f"{cdb(c)}/_design/validate", json={"validate_doc_update": fn}).raise_for_status()
        put = lambda d: couch.post(cdb(c), json=d).status_code == 201
    return {"validation": {"accepted_good": put(GOOD), "rejected_bad": not put(BAD), "ms": round((time.perf_counter() - t0) * 1000, 1)}}

def exp_watch(db, c, n=100):
    """Experiment 5: insert n docs while a listener tails the change feed; report delivery count and median latency."""
    seen, stop = {}, threading.Event()
    def listen():
        if db == "mongo":
            with mcol(c).watch([{"$match": {"operationType": "insert"}}], max_await_time_ms=200) as stream:
                while not stop.is_set():
                    ch = stream.try_next()
                    if ch: seen[ch["fullDocument"]["_id"]] = time.perf_counter()
        else:
            since = couch.get(cdb(c)).json()["update_seq"]
            with requests.get(f"{cdb(c)}/_changes", params={"feed": "continuous", "since": since, "heartbeat": 500}, stream=True) as r:
                for line in r.iter_lines():
                    if stop.is_set(): break
                    if not line: continue
                    j = json.loads(line)
                    if "id" in j: seen[j["id"]] = time.perf_counter()   # the trailing last_seq line has none
    threading.Thread(target=listen, daemon=True).start(); time.sleep(1.5)   # let the listener attach
    sent = {}
    for i in range(n):
        d = dict(GOOD, _id=f"watch_{i}"); sent[d["_id"]] = time.perf_counter()
        mcol(c).insert_one(d) if db == "mongo" else couch.post(cdb(c), json=d)
    time.sleep(2); stop.set()
    lat = [(seen[k] - sent[k]) * 1000 for k in sent if k in seen]
    return {"watch": {"sent": n, "received": len(lat), "median_latency_ms": round(statistics.median(lat), 1) if lat else None}}

def explore(continent):
    """Production series per area x crop for the site's data explorer (all units are tonnes)."""
    years, areas = list(range(1961, 2020)), {}
    for r in rows(continent):
        if r["element"] == "Production":
            areas.setdefault(r["area"], {})[r["item"]] = [round(float(r[f"y{y}"])) if r[f"y{y}"] else None for y in years]
    return {"years": years, "unit": "tonnes", "areas": areas}

# ---------------------------------------------------------------- driver
EXPERIMENTS = {"load": exp_load, "query": exp_query, "update": exp_update, "validate": exp_validate, "watch": exp_watch}

def save(update):
    r = json.loads(RESULTS.read_text()) if RESULTS.exists() else {"meta": {}, "results": {}}
    for c, exps in update.items():
        for k, v in exps.items():
            r["results"].setdefault(c, {}).setdefault(k, {}).update(v)
    r["meta"] |= {"date": time.strftime("%Y-%m-%d"), "runs": RUNS, "warmup": WARMUP, "one_limit": ONE_LIMIT,
                  "rows": {c: len(rows(c)) for c in CONTINENTS}, "versions": versions(),
                  "host": {"cpu": os.cpu_count(), "ram_gb": round(psutil.virtual_memory().total / 2**30)}}
    RESULTS.write_text(json.dumps(r, indent=1)); SITE.mkdir(exist_ok=True); shutil.copy(RESULTS, SITE / "results.json")
    print(json.dumps(update, indent=1))

def run(names, dbs, continents):
    ensure_replset()
    for c in continents:
        base, out = rows(c), {}
        for name in names:
            for db in dbs:
                print(f"[{c}] {name} {db} ...", flush=True)
                res = EXPERIMENTS[name](db, c, base) if name in ("load", "query", "update") else EXPERIMENTS[name](db, c)
                for k, v in res.items():
                    out.setdefault(k, {})[db] = v
        save({c: out})

def selftest():
    global rows
    sample = [dict({f"y{y}": "" for y in range(1961, 2020)}, area="A", item="X", element="Production", unit="tonnes", y2005="12.7")]
    real, rows = rows, lambda c: sample
    try:
        e = explore("x"); assert e["areas"]["A"]["X"][2005 - 1961] == 13 and e["areas"]["A"]["X"][0] is None
    finally:
        rows = real
    assert with_ids([{"a": 1}], "z")[0]["_id"] == "z_0"
    assert parse_cgroup("usage_usec 1500000 user_usec 1 system_usec 2 104857600".split()) == (1.5, 100.0)
    assert parse_cgroup(["2000000000", "52428800"]) == (2.0, 50.0)
    assert pick([{"area": "A", "item": "X", "element": "Production", "y2005": "1"}, {"area": "A", "item": "X", "element": "Yield", "y2005": "1"},
                 {"area": "A", "item": "Y", "element": "Production", "y2005": ""}, {"area": "A", "item": "Z", "element": "Production", "y2005": "2"}]) == ("A", ["X", "Z"])
    print("selftest ok")

if __name__ == "__main__":
    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    p.add_argument("cmd", choices=["all", *EXPERIMENTS, "explore", "selftest"])
    p.add_argument("--db", choices=["mongo", "couch", "both"], default="both")
    p.add_argument("--continent", choices=[*CONTINENTS, "all"], default="all")
    p.add_argument("--runs", type=int, default=RUNS); p.add_argument("--warmup", type=int, default=WARMUP)
    a = p.parse_args()
    RUNS, WARMUP = a.runs, a.warmup
    continents = CONTINENTS if a.continent == "all" else [a.continent]
    if a.cmd == "selftest":
        selftest()
    elif a.cmd == "explore":
        for c in continents:
            (SITE / f"explore_{c}.json").write_text(json.dumps(explore(c), separators=(",", ":")))
            print(c, (SITE / f"explore_{c}.json").stat().st_size // 1024, "KB")
    else:
        run(list(EXPERIMENTS) if a.cmd == "all" else [a.cmd], ["mongo", "couch"] if a.db == "both" else [a.db], continents)
        if a.cmd == "all":
            for c in continents:
                (SITE / f"explore_{c}.json").write_text(json.dumps(explore(c), separators=(",", ":")))