Most of the risk in a modern web app doesn't live in the code you wrote, it lives in the hundreds of transitive dependencies you pulled in to ship faster. My bachelor thesis, named "Comparative Analysis of Dependency Management Tools for Automated Vulnerability Remediation in Web Applications", compared the two dependency management tools that dominate in practice: GitHub's Dependabot and Mend's Renovate. This is what I measured, and what I'd take from it.
Problem Statement
Reusing third-party libraries means silently inheriting their vulnerabilities, and the npm ecosystem shows it clearly. One study of over 610,000 npm packages found that more than 133,000 of them directly depended on a package with a known vulnerability [1]. Fixes also arrive slowly: for vulnerabilities that are eventually fixed upstream, it takes 3 to 5 months on average until dependent projects pick the fix up [2]. Many maintainers simply don't know they're affected [3], and updates that might break the build get deprioritized [4].
React2Shell (CVE-2025-55182) is a recent example. Applications were exposed just because a vulnerable version sat somewhere in their dependency tree, even if nobody had chosen that package.
Bots like Dependabot and Renovate exist to close that gap, but there are few empirical comparisons of how well they work day to day. The thesis asked three questions:
- Detection accuracy: how well does each bot find known vulnerabilities?
- Automation quality and noise: how many of the proposed updates end up merged, and how many are wasted effort?
- Usability and integration complexity: how much developer effort and transparency does each tool involve?
How I tested it
Each bot ran on its own identical copy of the same recipe-management app, with mostly default configuration and only the minimal alignment needed for a fair comparison. OWASP Dependency-Track provided the ground truth and initially detected 38 known vulnerabilities in the baseline. I then observed PRs, CI runs and vulnerability state daily for two weeks in December 2025.
What I found
Detection was not the differentiator
| Tool | True positives | False positives | False negatives | F1 |
|---|---|---|---|---|
| Dependabot | 41 | 1 | 1 | 97.62% |
| Renovate | 39 | 0 | 0 | 100% |
Both tools surfaced essentially everything. The true positives add up to more than the 38-vulnerability baseline because GitHub recorded one next-auth advisory twice, so both bots start with 39 alerts, and Dependabot picked up two more next advisories that were published mid-experiment. The single Dependabot miss was an advisory that never showed up in GitHub's alert UI, and its false positive was an alert for an @trpc/server advisory that wasn't in the baseline and was created and fixed within minutes on 2025-12-16. One more alert, for brace-expansion, was auto-dismissed under Dependabot's defaults. It still counts as detected, but the vulnerable version was never actually fixed, while Renovate fixed it.
The more interesting result is that "how many vulnerabilities are there?" depends on how you count. GitHub listed 39 alerts, which collapsed to 36 distinct advisory IDs, while counting per affected package version gave 38. One advisory was even listed twice. A dashboard number is not ground truth, and software composition analysis tools are known to disagree with each other [5].
Remediation is where they diverged
Both bots cut the initial backlog quickly after activation. Renovate got the total down to 5 remaining vulnerabilities, while Dependabot dropped to 25 on the first day and then drifted back up to 28 of the 38 as new advisories came in. The dashed line is the 38-vulnerability baseline.
| Dependabot | Renovate | |
|---|---|---|
| PRs opened | 70 | 86 |
| PRs merged | 20 | 45 |
| Closed without merge | 20 | 1 |
| Update setup | one burst of 41 PRs on day one | security PRs first, routine updates the next day |
The charts below put PR activity next to the number of vulnerabilities fixed per day. Both show the same front-loaded burst, but only Renovate keeps fixing vulnerabilities after it.
Dependabot's churn came from superseding, as it closed older PRs in favor of newer ones as versions advanced [6], so a maintainer had to process more PR events to reach the same progress. Renovate separated security from routine updates, opening [security] PRs immediately and deferring the rest.
CI mattered as much as the bot. Failures clustered in the end-to-end tests, where Playwright failed 37.7% of Dependabot-triggered runs and 30.5% of Renovate-triggered runs. In this setup the verification layer, more than patch generation, often decided whether an update was mergeable. Lockfile conflicts in pnpm-lock.yaml also blocked PRs, which fits how differently package managers design their lockfiles [7], and Dependabot struggled more with transitive resolution, for example a vite update it could not land cleanly, which left a vulnerable esbuild version in place.
Usability and integration
Renovate's PRs carried structured vulnerability details and a clear justification, and my ratings for trust and noise were more favorable and more consistent. Dependabot's ratings dropped when CI or dependency resolution failed, and an "unknown" compatibility score raised the review burden instead of being neutral. That score can come back unknown when its underlying computation fails [8].
Integration effort differed too. Renovate could express the policy I wanted (scheduling plus automerge of low-risk updates) entirely in its own config [9]. Dependabot needed a separate GitHub Actions workflow for automerge [10], with its own permission handling and a fallback credential when the default token failed. That spreads one policy across bot settings, workflow code and repository permissions, which is more to maintain and more likely to drift.
What I take from it
- Don't choose on detection. Both tools find vulnerabilities reliably. The difference is in what happens after.
- Judge a bot by yield, not alerts. What matters is how many of its PRs stay stable long enough to merge, and how much context the PR gives you to decide quickly.
- Fix your CI before turning up the autonomy. Flaky E2E tests raise the cost of every merge, so unreliable pipelines are also a supply-chain risk because fixes pile up unmerged [11].
- Treat a dismissed alert as a decision to review. A clean dashboard doesn't prove the vulnerable version is gone. Cross-check with an SBOM scan or a second tool.
- Mind your package manager. Lockfile semantics decided whether updates applied cleanly, so test the bot against your actual setup.
What I reach for
On a small GitHub project where I just want security PRs to start arriving, Dependabot's near-zero setup is still a reasonable default. If I want scheduling, security-first ordering and automerge in one place, Renovate performed better in this experiment and is where I'd start.
The thesis doesn't compare either bot against manual updates or against doing nothing, so I can't claim how big the gap to no automation is. What it does show is that the two tools differ far more in remediation behavior than in detection. Supply-chain security stays a process you keep running, with committed lockfiles, reliable tests and a habit of triaging what automation can't fix.