The Lifetime of an URL

# The Lifetime of an URL Alexandre Dulaunoy **version 1 - 2016-06-19** While digging my bookshelf for some references, I found this old book **The Invisible Web: Uncovering Information Sources Search Engines Can’t See by Chris Sherman and Gary Price** published in 2001. [](https://www.flickr.com/photos/adulau/14776685031/) Practical software and hardware vulnerability management is inherently difficult and painful. But even if you have a great vulnerability management procedure in place, _we make a huge assumption which is the ability to know about those vulnerabilities_. While developing [cve-search](https://github.com/cve-search/cve-search) confirmed my fear that we have a really narrow view on the real vulnerability landscape of the software, hardware and embedded devices. Just to expose my point regarding our very limited view on the software vulnerability, we can simply focus on free software (which is often seen as a commodity software included in many proprietary or embedded devices). Software vulnerability types in the free software community could be summarised in four categories: - (1) Software where security vulnerabilities are fixed and CVEs are attributed. - (2) Software where vulnerabilities are silently fixed (e.g. bugs versus security bugs). - (3) Software where security vulnerabilities are never fixed (e.g. unmaintained software). - (4) Software not very popular and vulnerabilities are rarely discovered (e.g. software used by a very limited set of users). While trying to do some statistics about the popularity of free software packages using the [popularity-contest](http://popcon.ubuntu.com) database, there is a fundamental problem in the inventory of the software. Vulnerable software, with a CVE (category (1) in the list above) assigned, usually have one or more CPE (Common Platform Enumeration[^1]) associated. A CPE helps to know which software version or packages are affected by the vulnerability with the assigned CVE. CPE are often used to find back which of your software are impacted by a certain vulnerability. In cve-search, we have an interface to search per CPE in order to find the vulnerable packages (check the [cve-search presentation at BruCON 2015](https://www.circl.lu/assets/files/brucon2015-cve-search.pdf) for more details). In the category 1 which is the best case scenario for vulnerability management, we should be able to find easily vulnerable version? But the mapping between CPE and software is not so obvious. In the top software being used in the popularity contest, we have "ncurses" which is listed under the following package name in Ubuntu: - ncurses-base - ncurses-bin - libncurses5 What are the CPE names for ncurses? CPE are formated in the following format: ~~~~ cpe:/{part}:{vendor}:{product}:{version}:{update}:{edition}:{language} ~~~~ Assuming the vendor is the core developer team of the package, for ncurses it should be the GNU project (discarding the cases where there is maintained version by Debian or Ubuntu with backports). So we should easily find the related vulnerabilities to ncurses if we look at the [GNU project vendor](https://cve.circl.lu/browse/gnu) in the CPE... there are many of the software developed by the GNU project but there is no ncurses. But if you do a full-text search on all CVE assigned you'll find the following CVE: [CVE-2000-0963](https://cve.circl.lu/cve/CVE-2000-0963) - [CVE-2005-1796](https://cve.circl.lu/cve/CVE-2005-1796) - [CVE-2002-0062](https://cve.circl.lu/cve/CVE-2002-0062). The CPE listed only some of the vulnerable distributions including the vulnerable ncurses version but not the ncurses library itself as being vulnerable. As there is no parsable automatic cross-references between the vulnerabilities and the software name used in their deployment, it's very hard to build automatic tools which follow vulnerability management procedure. This example covers the case where CVEs are properly assigned and vulnerabilities are properly documented. __To summarize, if you solely rely on automatic tools to find your exposure to software vulnerabilities, your view is obviously narrowed. So please careful and don't underestimate the attack surface that you are not aware of__. ## Recommendations or some random ideas for improvement Linux distributions (like Debian or Ubuntu) should introduce the possibility to add the CPE names for their software packages. The mapping should include a proper mapping of the package names with version number and the respective CPE version (e.g. libglib:2.0 instead of libglib2.0-0). An automatic generation of the CPE name could be done to help the package maintainer to refine or keep the generated CPE name. CPE references in vulnerable_configuration (in CVEs) tend to include the vulnerable operating system but not the vulnerable software itself. cpe:/a: is not always defined and only the vulnerable operating systems are mentioned. There are many potential improvements but the CPE management could be slightly improved with a collaborative approach to add or updates CPE entries. Maybe software vendors (including free software authors and proprietary vendors) should include a list of CPE describing the software included in their distributions, appliance, IoT, mobile phone... to support the work of people and organization who try to do vulnerability management. ## References - [cve-search project](https://github.com/cve-search/) - [Debian Popularity Contest](http://popcon.debian.org/) - [Add Common Platform Enumerator information to package meta information](https://wiki.debian.org/CPEtagPackagesDep) - time to restart this proposal... [^1]: CPE is a structured naming scheme for information technology systems, software, and packages. Based upon the generic syntax for Uniform Resource Identifiers (URI), CPE includes a formal name format, a method for checking names against a system, and a description format for binding text and tests to a name. [https://nvd.nist.gov/cpe.cfm](https://nvd.nist.gov/cpe.cfm)