Recog Development with Rumble
Recog may be one of the most underrated open source security projects of all time. Recog started off in the early 2000s as the fingerprinting backend for Rapid7’s Nexpose (aka InsightVM) vulnerability scanner. It was released as open source in 2014 and integrated into the Metasploit Framework and Metasploit Pro products. The fingerprint coverage continues to grow through analysis of the Project Sonar data and contributions by our team as part of Rumble development.
At its core, Recog is a collection of XML files, each containing a list of fingerprints, with each fingerprint consisting of a regular expression and a series of values to assert on match. These XML files each correlate to a specific protocol response or field. For example, the
http_servers.xml database matches the normalized value of the HTTP Server header. To use this database, the fingerprints are processed sequentially, stopping after the first match, and recording the asserted values in the fingerprint. Regular expression capture parameters can be used to extract subfields and assert these in the match values. These capture parameters can be combined with static strings to build up more complicated match values.
The following fingerprint matches the HTTP Server value of
<fingerprint pattern="^Microsoft-IIS/7.5$"> <description>Microsoft IIS 7.5 runs on Windows Server 2008 R2 (and Windows 7)</description> <example>Microsoft-IIS/7.5</example> <param pos="0" name="service.vendor" value="Microsoft"/> <param pos="0" name="service.product" value="IIS"/> <param pos="0" name="service.family" value="IIS"/> <param pos="0" name="service.version" value="7.5"/> <param pos="0" name="service.cpe23" value="cpe:/a:microsoft:iis:7.5"/> <param pos="0" name="os.vendor" value="Microsoft"/> <param pos="0" name="os.family" value="Windows"/> <param pos="0" name="os.product" value="Windows Server 2008 R2"/> <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:-"/> </fingerprint>
<param> items result in those specific keys and values being asserted if this match is successful.
This process applies for every other supported match type. New databases can be created for almost anything and can assert one of the standard values, or a custom match value (often in OID format).
If you would like to get started with Recog development, the Rumble Scanner (available in our free tier) is a quick way to get rolling. The
-f) option can be used to specify an alternate fingerprint database and the
--fingerprints-debug option can by used to write scan log entries for sucessful and missing matches.
Install the Rumble Scanner into your path and clone a copy of the Recog repository:
$ git clone https://github.com/rapid7/recog.git
To test the HTTP fingerprints, we can run a new Rumble scan, limiting it to just the SYN probe and two HTTP ports:
$ sudo rumble-scanner --text -o test.log --overwrite --probes syn -p 80,443 -f ./recog/xml/ --fingerprints-debug 192.168.0.0/24
Running this scan will produce a bunch of
FP-FAIL output from the fingerprinting engine:
Aug 5 14:05:01.100 [INFO] loading alternate fingerprints from ./recog/xml/ Aug 5 14:05:01.311 [INFO] writing results to C:\Users\Developer\go\recog-test\test.log Aug 5 14:10:24.281 [INFO] [recog] html_title.xml FP-FAIL "index" Aug 5 14:10:24.281 [INFO] [recog] http_servers.xml FP-FAIL "App-webs/" Aug 5 14:10:24.285 [INFO] [recog] favicons.xml FP-MATCH "7de37347ff2f9277824b3e3cfe4a8ada" to "^7de37347ff2f9277824b3e3cfe4a8ada$" (TRENDnet IP Camera) Aug 5 14:10:24.285 [INFO] [recog] http_servers.xml FP-MATCH "Apache/2.4.29 (Ubuntu)" to "(?i)^Apache(?:-AdvancedExtranetServer)?(?:/([\\d.]*)\\s*(.*))?$" (Apache) Aug 5 14:10:24.286 [INFO] [recog] apache_os.xml FP-MATCH "Apache/2.4.29 (Ubuntu)" to ".*\\(Ubuntu\\).*" (Ubuntu) Aug 5 14:10:24.295 [INFO] [recog] html_title.xml FP-FAIL "Panopticon Human Observer POHO-1984" Aug 5 14:05:40.744 [INFO] scan completed in 39.4086113s Aug 5 14:05:41.139 [INFO] identified 11 unique assets through correlation Aug 5 14:05:41.330 [INFO] artifact generation completed
A list of failed matches can be pulled from the scan log using grep:
$ grep FP-FAIL test.log/scan.log
If the HTML title of
Panopticon Human Observer was a reliable match and the device model was
POHO-1984 we could create a new fingerprint at the end of
<fingerprint pattern="^Panopticon Human Observer (POHO-\d+)$"> <description>Panopticon Human Observer IP Camera</description> <example>Panopticon Human Observer</example> <param pos="0" name="hw.vendor" value="Panopticon"/> <param pos="0" name="hw.device" value="Web cam"/> <param pos="1" name="hw.product"/> </fingerprint>
This fingerprint captures the model name in the regular expression and then asserts that value as the hardware product.
Saving this XML file and rerunning the scan should now show:
Aug 5 14:25:19.185 [INFO] [recog] html_title.xml FP-MATCH "Panopticon Human Observer POHO-1984" (Panopticon Human Observer IP Camera)
To verify that the XML fingerprints are well-formed, there are two tools available,
First, make sure a recent version of Ruby is installed and configure the dependencies:
$ cd recog $ gem install bundler $ bundle install
recog_verify, specify the XML file as the argument:
$ ruby bin/recog_verify xml/html_title.xml
The full test suite can run be via
To normalize the vendor, hardware, operating system, and service names, you can use
recog_normalize (also in bin).
The CPE mappings can be regenerated via the
Once you are happy with your changes, you can submit this to the upstream Recog project by following the contribution guide.
Recog is an open source project, but didn’t start off that way, and many of the contributors from the early days deserve credit for what the project became. The Rapid7 Nexpose team did a phenomonal job of building a future-proof fingerprinting system. The late Jon Hart was critical in shepherding Recog from a Nexpose asset to a successful open source project. The entire Rapid7 team has been instrumental in keeping this project alive and fed with the latest Sonar data. The folks behind the NICER report did an incredible amount of work assessing Recog’s coverage and filling the gaps where needed.
Outside of the Rapid7 team, the Metasploit & Rumble communities have been amazing to work with and helpful in identifying bugs and missing fingerprints. Recog has become a great example of how open source can bring value to both commercial and community projects through global collaboration.
August 4, 2020
Rumble 1.10: Continuous Scans, Site Defaults, and More!
Overview Rumble 1.10 is live with continuous scanning, user interface updates, an event log, updates to the scan engine, additional fingerprints, and a new way to keep recurring scans in sync with their sites! Continuous Scanning All paid plans now support a new Continuous …Read More
July 7, 2020
Rumble 1.9.0: Scan Engine Updates, Reports, and More!
Overview Rumble 1.9.0 is out with major updates to the scan engine, reports, fingerprinting, user interface, documentation, and much more! Scan Engine Folks who scan external assets using their hostnames will now see asset correlation occur using the DNS name itself. For …Read More
May 4, 2020
Rumble 1.7.0: Reporting, Fingerprints, and More!
Overview Version 1.7.0 of Rumble Network Discovery is live with big updates to reporting. The Analysis Reports introduced in version 1.6.2 are now joined by a new Subnet Grid Report, linked off the main Subnets Report under the Explore menu. The Query Library has been …Read More