Overview

Recog may be one of the most underrated open source security projects of all time. Recog started off in the early 2000s as the fingerprinting backend for Rapid7’s Nexpose (aka InsightVM) vulnerability scanner. It was released as open source in 2014 and integrated into the Metasploit Framework and Metasploit Pro products. The fingerprint coverage continues to grow through analysis of the Project Sonar data and contributions by our team as part of Rumble development.

At its core, Recog is a collection of XML files, each containing a list of fingerprints, with each fingerprint consisting of a regular expression and a series of values to assert on match. These XML files each correlate to a specific protocol response or field. For example, the http_servers.xml database matches the normalized value of the HTTP Server header. To use this database, the fingerprints are processed sequentially, stopping after the first match, and recording the asserted values in the fingerprint. Regular expression capture parameters can be used to extract subfields and assert these in the match values. These capture parameters can be combined with static strings to build up more complicated match values.

The following fingerprint matches the HTTP Server value of Microsoft-IIS/7.5:

  <fingerprint pattern="^Microsoft-IIS/7.5$">
    <description>Microsoft IIS 7.5 runs on Windows Server 2008 R2 (and Windows 7)</description>
    <example>Microsoft-IIS/7.5</example>
    <param pos="0" name="service.vendor" value="Microsoft"/>
    <param pos="0" name="service.product" value="IIS"/>
    <param pos="0" name="service.family" value="IIS"/>
    <param pos="0" name="service.version" value="7.5"/>
    <param pos="0" name="service.cpe23" value="cpe:/a:microsoft:iis:7.5"/>
    <param pos="0" name="os.vendor" value="Microsoft"/>
    <param pos="0" name="os.family" value="Windows"/>
    <param pos="0" name="os.product" value="Windows Server 2008 R2"/>
    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:-"/>
  </fingerprint>

Those <param> items result in those specific keys and values being asserted if this match is successful.

This process applies for every other supported match type. New databases can be created for almost anything and can assert one of the standard values, or a custom match value (often in OID format).

Recog Development with Rumble

If you would like to get started with Recog development, the Rumble Scanner (available in our free tier) is a quick way to get rolling. The --fingerprints (shorthand: -f) option can be used to specify an alternate fingerprint database and the --fingerprints-debug option can by used to write scan log entries for sucessful and missing matches.

Install the Rumble Scanner into your path and clone a copy of the Recog repository:

$ git clone https://github.com/rapid7/recog.git

To test the HTTP fingerprints, we can run a new Rumble scan, limiting it to just the SYN probe and two HTTP ports:

 $ sudo rumble-scanner --text -o test.log --overwrite --probes syn -p 80,443 -f ./recog/xml/ --fingerprints-debug 192.168.0.0/24

Running this scan will produce a bunch of FP-MATCH and FP-FAIL output from the fingerprinting engine:

Aug  5 14:05:01.100 [INFO] loading alternate fingerprints from ./recog/xml/
Aug  5 14:05:01.311 [INFO] writing results to C:\Users\Developer\go\recog-test\test.log
Aug  5 14:10:24.281 [INFO] [recog] html_title.xml FP-FAIL "index"
Aug  5 14:10:24.281 [INFO] [recog] http_servers.xml FP-FAIL "App-webs/"
Aug  5 14:10:24.285 [INFO] [recog] favicons.xml FP-MATCH "7de37347ff2f9277824b3e3cfe4a8ada" to "^7de37347ff2f9277824b3e3cfe4a8ada$" (TRENDnet IP Camera)
Aug  5 14:10:24.285 [INFO] [recog] http_servers.xml FP-MATCH "Apache/2.4.29 (Ubuntu)" to "(?i)^Apache(?:-AdvancedExtranetServer)?(?:/([012][\\d.]*)\\s*(.*))?$" (Apache)
Aug  5 14:10:24.286 [INFO] [recog] apache_os.xml FP-MATCH "Apache/2.4.29 (Ubuntu)" to ".*\\(Ubuntu\\).*" (Ubuntu)
Aug  5 14:10:24.295 [INFO] [recog] html_title.xml FP-FAIL "Panopticon Human Observer POHO-1984"
Aug  5 14:05:40.744 [INFO] scan completed in 39.4086113s
Aug  5 14:05:41.139 [INFO] identified 11 unique assets through correlation
Aug  5 14:05:41.330 [INFO] artifact generation completed

A list of failed matches can be pulled from the scan log using grep:

$ grep FP-FAIL test.log/scan.log

If the HTML title of Panopticon Human Observer was a reliable match and the device model was POHO-1984 we could create a new fingerprint at the end of html_title.xml:

  <fingerprint pattern="^Panopticon Human Observer (POHO-\d+)$">
    <description>Panopticon Human Observer IP Camera</description>
    <example>Panopticon Human Observer</example>
    <param pos="0" name="hw.vendor" value="Panopticon"/>
    <param pos="0" name="hw.device" value="Web cam"/>
    <param pos="1" name="hw.product"/>
  </fingerprint>

This fingerprint captures the model name in the regular expression and then asserts that value as the hardware product.

Saving this XML file and rerunning the scan should now show:

Aug  5 14:25:19.185 [INFO] [recog] html_title.xml FP-MATCH "Panopticon Human Observer POHO-1984" (Panopticon Human Observer IP Camera)

To verify that the XML fingerprints are well-formed, there are two tools available, recog_verify and rspec.

First, make sure a recent version of Ruby is installed and configure the dependencies:

$ cd recog
$ gem install bundler
$ bundle install

To use recog_verify, specify the XML file as the argument:

$ ruby bin/recog_verify xml/html_title.xml

The full test suite can run be via rspec:

$ rspec

To normalize the vendor, hardware, operating system, and service names, you can use recog_normalize (also in bin).

The CPE mappings can be regenerated via the update_cpe.py script.

Once you are happy with your changes, you can submit this to the upstream Recog project by following the contribution guide.

Acknowledgements

Recog is an open source project, but didn’t start off that way, and many of the contributors from the early days deserve credit for what the project became. The Rapid7 Nexpose team did a phenomonal job of building a future-proof fingerprinting system. The late Jon Hart was critical in shepherding Recog from a Nexpose asset to a successful open source project. The entire Rapid7 team has been instrumental in keeping this project alive and fed with the latest Sonar data. The folks behind the NICER report did an incredible amount of work assessing Recog’s coverage and filling the gaps where needed.

Outside of the Rapid7 team, the Metasploit & Rumble communities have been amazing to work with and helpful in identifying bugs and missing fingerprints. Recog has become a great example of how open source can bring value to both commercial and community projects through global collaboration.

Similar Content

Overview Rumble 1.10 is live with continuous scanning, user interface updates, an event log, updates to the scan engine, additional fingerprints, and a new way to keep recurring scans in sync with their sites! Continuous Scanning All paid plans now support a new Continuous scanning option. This will run scans back-to-back, pausing only to apply agent updates. For folks who want to keep a close eye on their networks, continuous scans bring you fresher data, faster.
Overview Rumble 1.9.0 is out with major updates to the scan engine, reports, fingerprinting, user interface, documentation, and much more! Scan Engine Folks who scan external assets using their hostnames will now see asset correlation occur using the DNS name itself. For environments where IP addresses are constantly changing (load balancers, CDNs, etc) this leads to less churn and a more accurate inventory. The Rumble Agent and Rumble Scanner now detect and automatically filter out invalid services caused by intercepting middle devices such as Fortigate firewalls and Cisco ASAs.
Overview Version 1.7.0 of Rumble Network Discovery is live with big updates to reporting. The Analysis Reports introduced in version 1.6.2 are now joined by a new Subnet Grid Report, linked off the main Subnets Report under the Explore menu. The Query Library has been updated with small tweaks and new built-in query for finding expired TLS certificates, supported by improvements to the scan engine. The Rumble backend has been upgraded to support our larger customers as well as all of our new Starter Edition users.