module Mdm::Host::OperatingSystemNormalization
Rules for operating system fingerprinting in Metasploit
The `os.product` key identifies the common-name of a specific operating system Examples include: Linux, Windows XP, Mac OS X, IOS, AIX, HP-UX, VxWorks
The `os.version` key identifies the service pack or version of the operating system Sometimes this means a kernel or firmware version when the distribution or OS version is not available. Examples include: SP2, 10.04, 2.6.47, 10.6.1
The `os.vendor` key identifies the manufacturer of the operating system Examples include: Microsoft, Ubuntu, Cisco, HP, IBM, Wind River
The `os.family` key identifies the group of the operating system. This is often a duplicate of os.product, unless a more specific product name is available. Examples include: Windows, Linux, IOS, HP-UX, AIX
The `os.edition` key identifies the specific variant of the operating system Examples include: Enterprise, Professional, Starter, Evaluation, Home, Datacenter
An example breakdown of a common operating system is shown below
* Microsoft Windows XP Professional Service Pack 3 English (x86) - os.product = 'Windows XP' - os.edition = 'Professional' - os.vendor = 'Microsoft' - os.version = 'SP3' - os.language = 'English' - os.arch = 'x86'
These rules are then mapped to the {Mdm::Host} attributes below:
* os_name - Maps to a normalized os.product key * os_flavor - Maps to a normalized os.edition key * os_sp - Maps to a normalized os.version key (soon os_version) * os_lang - Maps to a normalized os.language key * arch - Maps to a normalized os.arch key
Additional rules include the following mappings:
* name - Maps to the host.name key * mac - Maps to the host.mac key
The following keys are not mapped to {Mdm::Host} at this time (but should be):
* os.vendor
In order to execute these rules, this module is responsible for mapping various fingerprint sources to {Mdm::Host} values. This requires some ugly glue code to account for differences between each supported input (external scanners), the Recog gem and associated databases, and how Metasploit itself likes to handle these values. Getting a mapping wrong is often harmless, but can impact the automatic targetting capabilities of certain exploit modules.
In other words, this is a best-effort attempt to rationalize multiple competing sources of information about a host and come up with the values representing a normalized assessment of the system. The use of `Recog` and multiple scanner fingerprints can result in a comprehensive (and confident) identification of the remote operating system and associated services.
Historically, there are direct conflicts between certain Metasploit modules, certain scanners, and external fingerprint databases in terms of how a particular OS and patch level is represented. This module attempts to fix what it can and serve as documentation and live workarounds for the rest.
Examples of known conflicts that are still in progress:
-
Metasploit defines an OS constant of 'win'/'windows' as Microsoft Windows
-
Scanner modules report a mix of 'Microsoft Windows' and 'Windows'
-
Nearly all exploit modules reference 'Windows <Release> SP<Version>'
-
Nmap (and other scanners) also prefix the vendor before Windows
-
-
Windows service packs represented as 'Service Pack X' or 'SPX'
-
The preferred form is to set os.version to 'SPX'
-
Many external scanners & Recog prefer 'Service Pack X'
-
-
Apple Mac OS X, Cisco IOS, IBM AIX, Ubuntu Linux, all reported with vendor prefix
-
The preferred form is to remove the vendor from os.product
-
{Mdm::Host} currently has no vendor field, so this information is lost today
-
Many scanners report leading vendor strings and require normalization
-
-
The os_flavor field is used in contradictory ways across Metasploit
-
The preferred form is to be a 'display only' field
-
Some Recog fingerprints still append the edition to os.product
-
Many scanners report the edition as a trailing suffix to os.product
Maintenance:
-
Ensure that the latest Recog gem is present and installed
-
For new operating system releases, update relevant sections a) Windows releases will require updates to a few methods
1) parse_windows_os_str() 2) normalize_nmap_fingerprint() 3) normalize_nexpose_fingerprint() 4) Other scanner normalizers
b) Mobile operating systems are minimally recognized
@todo Handle OS icon incompatiblities with new fingerprint names
Note that VMWare ESX(i) was special cased before as well, make sure it still works 1) Cisco IOS -> IOS breaks the icon mapping in MSP/MSCE of /cisco/ 2) Ubuntu Linux -> Linux breaks the distro selection The real solution is to add os_vendor and take this into account for icons
@todo Implement rspec coverage for #normalize_os() @todo Implement smb.generic fingerprint database (replace {#parse_windows_os_str}?) @todo Implement Samba version matching for specific distributions and OS versions @todo Implement DD-WRT and various embedded device signatures currently missing @todo Correct inconsistencies in os_name use by removing the vendor string (Microsoft Windows -> Windows)
This applies to MSF core and a handful of modules, not to mention some Recog fingerprints.
@todo Rename host.os_sp to host.os_version @todo Add host.os_vendor @todo Add host.os_confidence @todo Add host.domain
Constants
- MAX_NMAP_CERTAINTY
Cap nmap certainty at 0.84 until we update it more frequently XXX: Without this, Nmap will beat the default certainty of recog
matches and its less-confident guesses will take precedence over service-based fingerprints.
Public Instance Methods
Examine the assertations of the merged best match and map these back to fields of {Mdm::Host}. Take particular care not to leave related fields (os_*) in a conflicting state, leverage existing values where possible, and use the most confident values we have.
# File lib/mdm/host/operating_system_normalization.rb, line 367 def apply_match_to_host(match) host = self # These values in a match always override the current value unless # the host attribute has been explicitly locked by the user if match['host.mac'] && !host.attribute_locked?(:mac) host.mac = sanitize(match['host.mac']) end if match['host.name'] && !host.attribute_locked?(:name) host.name = sanitize(match['host.name']) end # Select the os architecture if available if match['os.arch'] && !host.attribute_locked?(:arch) host.arch = sanitize(match['os.arch']) end # Guess the purpose using some basic heuristics if ! host.attribute_locked?(:purpose) host.purpose = guess_purpose_from_match(match) end # # Map match fields from Recog fingerprint style to Metasploit style # # os.build: Examples: 9001, 2600, 7602 # os.device: Examples: General, ADSL Modem, Broadband router, Cable Modem, Camera, Copier, CSU/DSU # os.edition: Examples: Web, Storage, HPC, MultiPoint, Enterprise, Home, Starter, Professional # os.family: Examples: Windows, Linux, Solaris, NetWare, ProCurve, Mac OS X, HP-UX, AIX # os.product: Examples: Windows, Linux, Windows Server 2008 R2, Windows XP, Enterprise Linux, NEO Tape Library # os.vendor: Examples: Microsoft, HP, IBM, Sun, 3Com, Ricoh, Novell, Ubuntu, Apple, Cisco, Xerox # os.version: Examples: SP1, SP2, 6.5 SP3 CPR, 10.04, 8.04, 12.10, 4.0, 6.1, 8.5 # os.language: Examples: English, Arabic, German # linux.kernel.version: Examples: 2.6.32 # Metasploit currently ignores os.build, os.device, and os.vendor as separate fields. # Select the OS name from os.name, fall back to os.family if ! host.attribute_locked?(:os_name) # Try to fill this value from os.product first if it exists if match.has_key?('os.product') host.os_name = sanitize(match['os.product']) else # Fall back to os.family otherwise, if available if match.has_key?('os.family') host.os_name = sanitize(match['os.family']) end end end # Select the flavor from os.edition if available if match.has_key?('os.edition') and ! host.attribute_locked?(:os_flavor) host.os_flavor = sanitize(match['os.edition']) end # Select an OS version as os.version, fall back to linux.kernel.version if ! host.attribute_locked?(:os_sp) if match['os.version'] host.os_sp = sanitize(match['os.version']) else if match['linux.kernel.version'] host.os_sp = sanitize(match['linux.kernel.version']) end end end # Select the os language if available if match.has_key?('os.language') and ! host.attribute_locked?(:os_lang) host.os_lang = sanitize(match['os.language']) end # Normalize MAC addresses to lower-case colon-delimited format if host.mac and ! host.attribute_locked?(:mac) host.mac = host.mac.downcase if host.mac =~ /^[a-f0-9]{12}$/ host.mac = host.mac.scan(/../).join(':') end end end
Return a normalized architecture based on patterns in the input string. This will identify things like sparc, powerpc, x86_x64, and i686
# File lib/mdm/host/operating_system_normalization.rb, line 892 def get_arch_from_string(str) res = Recog::Nizer.match("architecture", str) return unless (res and res['os.arch']) res['os.arch'] end
Loosely guess the purpose of a device based on available match values. In the future, also take into account the exposed services and rename to guess_purpose_with_match()
# File lib/mdm/host/operating_system_normalization.rb, line 456 def guess_purpose_from_match(match) # Create a string based on all match values pstr = match.values.join(' ').downcase # Loosely map keywords to specific purposes case pstr when /windows server|windows (nt|20)/ 'server' when /windows (xp|vista|[78])/ 'client' when /printer|print server/ 'printer' when /router/ 'router' when /firewall/ 'firewall' when /linux/ 'server' else 'device' end end
Normalize data from FusionVM fingerprints
# File lib/mdm/host/operating_system_normalization.rb, line 756 def normalize_fusionvm_fingerprint(data) ret = {} case data[:os] when /Windows/ ret.update(parse_windows_os_str(data[:os])) when /Linux ([^[:space:]]*) ([^[:space:]]*) .* (\(.*\))/ ret['os.product'] = "Linux" ret['host.name'] = $1 ret['os.version'] = $2 ret['os.arch'] = get_arch_from_string($3) else ret['os.product'] = data[:os] end ret['os.arch'] = data[:arch] if data[:arch] ret['host.name'] = data[:name] if data[:name] [ ret ] end
Normalize data from generic fingerprints
# File lib/mdm/host/operating_system_normalization.rb, line 777 def normalize_generic_fingerprint(data) ret = {} ret['os.product'] = data[:os_name] || data[:os] || data[:os_fingerprint] || "Unknown" ret['os.arch'] = data[:os_arch] if data[:os_arch] ret['os.certainty'] = data[:os_certainty] || 0.5 [ ret ] end
Normalize matches in order to handle inconsistencies between fingerprint sources and our desired usage in Metasploit. This amounts to yet more duct tape, but the situation should improve as the fingerprint sources are updated and enhanced. In the future, this method will no longer be needed (or at least, doing less and less work)
# File lib/mdm/host/operating_system_normalization.rb, line 315 def normalize_match(m) # Normalize os.version strings containing 'Service Pack X' to just 'SPX' if m['os.version'] and m['os.version'].index('Service Pack ') == 0 m['os.version'] = m['os.version'].gsub(/Service Pack /, 'SP') end if m['os.product'] # Normalize Apple Mac OS X to just Mac OS X if m['os.product'] =~ /^Apple Mac/ m['os.product'] = m['os.product'].gsub(/Apple Mac/, 'Mac') m['os.vendor'] ||= 'Apple' end # Normalize Sun Solaris/Sun SunOS to just Solaris/SunOS if m['os.product'] =~ /^Sun (Solaris|SunOS)/ m['os.product'] = m['os.product'].gsub(/^Sun /, '') m['os.vendor'] ||= 'Oracle' end # Normalize Microsoft Windows to just Windows to catch any stragglers if m['os.product'] =~ /^Microsoft Windows/ m['os.product'] = m['os.product'].gsub(/Microsoft Windows/, 'Windows') m['os.vendor'] ||= 'Microsoft' end # Normalize Windows Server to just Windows to match Metasploit target names if m['os.product'] =~ /^Windows Server/ m['os.product'] = m['os.product'].gsub(/Windows Server/, 'Windows') end end m end
Normalize data from MBSA fingerprints
# File lib/mdm/host/operating_system_normalization.rb, line 551 def normalize_mbsa_fingerprint(data) ret = {} # :os_match=>"Microsoft Windows Vista SP0 or SP1, Server 2008, or Windows 7 Ultimate (build 7000)" # :os_vendor=>"Microsoft" :os_family=>"Windows" :os_version=>"7" :os_accuracy=>"100" ret['os.certainty'] = ( data[:os_accuracy].to_f / 100.0 ).to_s if data[:os_accuracy] ret['os.family'] = data[:os_family] if data[:os_family] ret['os.vendor'] = data[:os_vendor] if data[:os_vendor] if data[:os_family] and data[:os_version] ret['os.product'] = data[:os_family] + " " + data[:os_version] end ret['host.name'] = data[:hostname] if data[:hostname] [ ret ] end
Normalize data from Nessus fingerprints
# File lib/mdm/host/operating_system_normalization.rb, line 656 def normalize_nessus_fingerprint(data) ret = {} # :os=>"Microsoft Windows 2000 Advanced Server (English)" # :os=>"Microsoft Windows 2000\nMicrosoft Windows XP" # :os=>"Linux Kernel 2.6" # :os=>"Sun Solaris 8" # :os=>"IRIX 6.5" # Nessus sometimes jams multiple OS names together with a newline. oses = data[:os].split(/\n/) if oses.length > 1 # Multiple fingerprints means Nessus wasn't really sure, reduce # the certainty accordingly ret['os.certainty'] = 0.5 else ret['os.certainty'] = 0.8 end # Since there is no confidence associated with them, the best we # can do is just take the first one. case oses.first when /^(Microsoft |)Windows/ ret.update(parse_windows_os_str(data[:os])) when /(2\.[46]\.\d+[-a-zA-Z0-9]+)/ # Look for older Linux kernel versions ret['os.product'] = "Linux" ret['os.version'] = $1 when /^Linux Kernel ([\d\.]+)(.*)/ # Look for strings like "Linux Kernel 2.6 on Ubuntu 9.10 (karmic)" # Ex: Linux Kernel 2.2 on Red Hat Linux release 6.2 (Zoot) # Ex: Linux Kernel 2.6 on Ubuntu Linux 8.04 (hardy) ret['os.product'] = "Linux" ret['os.version'] = $1 vendor = $2.to_s # Try to snag the vendor name as well if vendor =~ /on (\w+|\w+ \w+|\w+ \w+ \w+) (Linux|\d)/ ret['os.vendor'] = $1 end when /(.*) ([0-9\.]+)$/ # Then we don't necessarily know what the os is, but this fingerprint has # some version information at the end, pull it off, treat the first part # as the OS, and the rest as the version. ret['os.product'] = $1.gsub("Kernel", '').strip ret['os.version'] = $2 else # TODO: Return each OS guess as a separate match ret['os.product'] = oses.first end ret['host.name'] = data[:hname] if data[:hname] [ ret ] end
Normalize data from Nexpose fingerprints
# File lib/mdm/host/operating_system_normalization.rb, line 572 def normalize_nexpose_fingerprint(data) ret = {} # :family=>"Windows" :certainty=>"0.85" :vendor=>"Microsoft" :product=>"Windows 7 Ultimate Edition" # :family=>"Windows" :certainty=>"0.67" :vendor=>"Microsoft" :arch=>"x86" :product=>'Windows 7' :version=>'SP1' # :family=>"Linux" :certainty=>"0.64" :vendor=>"Linux" :product=>"Linux" # :family=>"Linux" :certainty=>"0.80" :vendor=>"Ubuntu" :product=>"Linux" # :family=>"IOS" :certainty=>"0.80" :vendor=>"Cisco" :product=>"IOS" # :family=>"embedded" :certainty=>"0.61" :vendor=>"Linksys" :product=>"embedded" ret['os.certainty'] = data[:certainty] if data[:certainty] ret['os.family'] = data[:family] if data[:family] ret['os.vendor'] = data[:vendor] if data[:vendor] case data[:product] when /^Windows/ # TODO: Verify Windows CE and Windows 8 RT fingerprints # Translate the version into the representation we want case data[:version].to_s # These variants are normalized to just 'Windows <Version>' when "NT", "2000", "95", "ME", "XP", "Vista", "7", "8", "8.1" ret['os.product'] = "Windows #{data[:version]}" # Service pack in the version field should be recognized when /^SP\d+/, /^Service Pack \d+/ ret['os.product'] = data[:product] ret['os.version'] = data[:version] # No version means the version is part of the product already when nil, '' # Trim any 'Server' suffix and use as it is ret['os.product'] = data[:product].sub(/ Server$/, '') # Otherwise, we assume a Server version of Windows else ret['os.product'] = "Windows Server #{data[:version]}" end # Extract the edition string if it is present if data[:product] =~ /(XP|Vista|\d+(?:\.\d+)) (\w+|\w+ \w+|\w+ \w+ \w+) Edition/ ret['os.edition'] = $2 end when nil, 'embedded' # Use the family or vendor name when the product is empty or 'embedded' ret['os.product'] = data[:family] unless data[:family] == 'embedded' ret['os.product'] ||= data[:vendor] ret['os.version'] = data[:version] if data[:version] else # Default to using the product name reported by Nexpose ret['os.product'] = data[:product] if data[:product] end ret['os.arch'] = get_arch_from_string(data[:arch]) if data[:arch] ret['os.arch'] ||= get_arch_from_string(data[:desc]) if data[:desc] [ ret ] end
Normalize data from Nmap fingerprints
# File lib/mdm/host/operating_system_normalization.rb, line 520 def normalize_nmap_fingerprint(data) ret = {} # :os_vendor=>"Microsoft" :os_family=>"Windows" :os_version=>"2000" :os_accuracy=>"94" ret['os.certainty'] = ( data[:os_accuracy].to_f / 100.0 ).to_s if data[:os_accuracy] if (data[:os_vendor] == data[:os_family]) ret['os.product'] = data[:os_family] else ret['os.product'] = data[:os_family] ret['os.vendor'] = data[:os_vendor] end # Nmap places the type of Windows (XP, 7, etc) into the version field if ret['os.product'] == 'Windows' and data[:os_version] ret['os.product'] = ret['os.product'] + ' ' + data[:os_version].to_s else ret['os.version'] = data[:os_version] end ret['host.name'] = data[:hostname] if data[:hostname] if ret['os.certainty'] ret['os.certainty'] = [ ret['os.certainty'].to_f, MAX_NMAP_CERTAINTY ].min.to_s end [ ret ] end
Normalize the operating system fingerprints provided by various scanners (nmap, nexpose, retina, nessus, metasploit modules, and more!)
These are stored as {Mdm::Note notes} (instead of directly in the os_* fields) specifically for this purpose.
The goal is to infer as much as we can about the OS of the device and the various {Mdm::Service services} offered using the Recog gem and some glue logic to determine the best weights. This method can result in changes to the recorded {#os_name}, {#os_flavor}, {#os_sp}, {#os_lang}, {#purpose}, {#name}, {#arch}, and the {Mdm::Service service details}.
# File lib/mdm/host/operating_system_normalization.rb, line 151 def normalize_os host = self matches = [] # Note that we're already restricting the query to this host by using # host.notes instead of Note, so don't need a host_id in the # conditions. fingerprintable_notes = self.notes.where("ntype like '%%fingerprint'") fingerprintable_notes.each do |fp_note| matches += recog_matches_for_note(fp_note) end # XXX: This hack solves the memory leak generated by self.services.each {} fingerprintable_services = self.services.where("name is not null and name != '' and info is not null and info != ''") fingerprintable_services.each do |s| matches += recog_matches_for_service(s) end # # Look for generic fingerprint.match notes that generate a match hash from modules # This handles ad-hoc os.language, host.name, etc identifications # generated_matches = self.notes.where(ntype: 'fingerprint.match') generated_matches.each do |m| next unless (m.data and m.data.kind_of?(::Hash)) matches << m.data.dup end # Normalize matches for consistency during the ranking phase matches = matches.map{ |m| normalize_match(m) } # Calculate the best OS match based on fingerprint hits match = Recog::Nizer.best_os_match(matches) # Merge and normalize the best match to the host object apply_match_to_host(match) if match # Set some sane defaults if needed host.os_name ||= 'Unknown' host.purpose ||= 'device' host.save if host.changed? end
Normalize data from Qualys fingerprints
# File lib/mdm/host/operating_system_normalization.rb, line 717 def normalize_qualys_fingerprint(data) ret = {} # :os=>"Microsoft Windows 2000" # :os=>"Windows 2003" # :os=>"Microsoft Windows XP Professional SP3" # :os=>"Ubuntu Linux" # :os=>"Cisco IOS 12.0(3)T3" # :os=>"Red-Hat Linux 6.0" case data[:os] when /Windows/ ret.update(parse_windows_os_str(data[:os])) when /^(Cisco) (IOS) (\d+[^\s]+)/ ret['os.product'] = $2 ret['os.vendor'] = $1 ret['os.version'] = $3 when /^([^\s]+) (Linux)(.*)/ ret['os.product'] = $2 ret['os.vendor'] = $1 ver = $3.to_s.strip.split(/\s+/).first if ver =~ /^\d+\./ ret['os.version'] = ver end else parts = data[:os].split(/\s+/, 3) ret['os.product'] = "Unknown" ret['os.product'] = parts[0] if parts[0] ret['os.product'] << " " + parts[1] if parts[1] ret['os.version'] = parts[2] if parts[2] end [ ret ] end
Normalize data from Retina fingerprints
# File lib/mdm/host/operating_system_normalization.rb, line 637 def normalize_retina_fingerprint(data) ret = {} # :os=>"Windows Server 2003 (X64), Service Pack 2" case data[:os] when /Windows/ ret.update(parse_windows_os_str(data[:os])) else # No idea what this looks like if it isn't windows. Just store # the whole thing and hope for the best. # TODO: Add examples of non-Windows results ret['os.product'] = data[:os] if data[:os] end [ ret ] end
Convert a host.os.*_fingerprint Note into a hash containing 'os.*' and 'host.*' fields
Also includes a os.certainty which is a float from 0 - 1.00 indicating the scanner's confidence in its fingerprint. If the particular scanner does not provide such information, default to 0.80.
# File lib/mdm/host/operating_system_normalization.rb, line 792 def normalize_scanner_fp(fp) hits = [] return hits if not validate_fingerprint_data(fp) case fp.ntype when /^host\.os\.(.*_fingerprint)$/ pname = $1 pmeth = 'normalize_' + pname if self.respond_to?(pmeth) hits = self.send(pmeth, fp.data) else hits = normalize_generic_fingerprint(fp.data) end end hits.each {|hit| hit['os.certainty'] ||= 0.80} hits end
Normalize data from Meterpreter's client.sys.config.sysinfo()
# File lib/mdm/host/operating_system_normalization.rb, line 488 def normalize_session_fingerprint(data) ret = {} case data[:os] when /Windows/ ret.update(parse_windows_os_str(data[:os])) # Switch to this code block once the multi-meterpreter code review is complete when /^(Windows \w+)\s*\(Build (\d+)(.*)\)/ ret['os.product'] = $1 ret['os.build'] = $2 ret['os.vendor'] = 'Microsoft' possible_sp = $3 if possible_sp =~ /Service Pack (\d+)/ ret['os.version'] = 'SP' + $1 end when /Linux (\d+\.\d+\.\d+\S*)\s* \((\w*)\)/ ret['os.product'] = "Linux" ret['os.version'] = $1 ret['os.arch'] = get_arch_from_string($2) else ret['os.product'] = data[:os] end ret['os.arch'] = data[:arch] if data[:arch] ret['host.name'] = data[:name] if data[:name] [ ret ] end
Take a windows version string and return a hash with fields suitable for Host this object's version fields. This is used as a fall-back to parse external fingerprints and should eventually be replaced by per-source mappings.
A few example strings that this will have to parse: sessions
Windows XP (Build 2600, Service Pack 3). Windows .NET Server (Build 3790). Windows 2008 (Build 6001, Service Pack 1).
retina
Windows Server 2003 (X64), Service Pack 2
nessus
Microsoft Windows 2000 Advanced Server (English)
qualys
Microsoft Windows XP Professional SP3 Windows 2003
Note that this list doesn't include nexpose or nmap, since they are both kind enough to give us the various strings in seperate pieces that we don't have to parse out manually.
# File lib/mdm/host/operating_system_normalization.rb, line 834 def parse_windows_os_str(str) ret = {} # Set some reasonable defaults for Windows ret['os.vendor'] = 'Microsoft' ret['os.product'] = 'Windows' # Determine the actual Windows product name case str when /\.NET Server/ ret['os.product'] << ' Server 2003' when / (2000|2003|2008|2012)/ ret['os.product'] << ' Server ' + $1 when / (NT (?:3\.51|4\.0))/ ret['os.product'] << ' ' + $1 when /Windows (95|98|ME|XP|Vista|[\d\.]+)/ ret['os.product'] << ' ' + $1 else # If we couldn't pull out anything specific for the flavor, just cut # off the stuff we know for sure isn't it and hope for the best ret['os.product'] = (ret['os.product'] + ' ' + str.gsub(/(Microsoft )|(Windows )|(Service Pack|SP) ?(\d+)/i, '').strip).strip # Make sure the product name doesn't include any non-alphanumeric stuff # This fixes cases where the above code leaves 'Windows XX (Build 3333,)...' ret['os.product'] = ret['os.product'].split(/[^a-zA-Z0-9 ]/).first.strip end # Take a guess at the architecture arch = get_arch_from_string(str) ret['os.arch'] = arch if arch # Extract any service pack value in the string if str =~ /(Service Pack|SP) ?(\d+)/i ret['os.version'] = "SP#{$2}" end # Extract any build ID found in the string if str =~ /build (\d+)/i ret['os.build'] = $1 end # Extract the OS edition if available if str =~ /(\d+|\d+\.\d+) (\w+|\w+ \w+|\w+ \w+ \w+) Edition/ ret['os.edition'] = $2 else if str =~ /(Professional|Enterprise|Pro|Home|Start|Datacenter|Web|Storage|MultiPoint)/ ret['os.edition'] = $1 end end ret end
Recog matches for the fingerprint in `note`.
@return [Array<Hash>] Keys will be host, service, and os attributes
# File lib/mdm/host/operating_system_normalization.rb, line 244 def recog_matches_for_note(note) # Skip notes that are missing the correct structure or have been blacklisted return [] if not validate_fingerprint_data(note) # # These rules define the relationship between fingerprint note keys # and specific Recog databases for detailed matching. Notes that do # not match a rule are passed to the generic matcher. # fingerprint_note_match_keys = { 'smb.fingerprint' => { :native_os => [ 'smb.native_os' ], }, 'http.fingerprint' => { :header_server => [ 'http_header.server', 'apache_os' ], :header_set_cookie => [ 'http_header.cookie' ], :header_www_authenticate => [ 'http_header.wwwauth' ], # TODO: Candidates for future Recog support # :content => 'http_body' # :code => 'http_response_code' # :message => 'http_response_message' } } matches = [] # Look for a specific Recog database for this type and data key if fingerprint_note_match_keys.has_key?( note.ntype ) fingerprint_note_match_keys[ note.ntype ].each_pair do |k,rdbs| if note.data.has_key?(k) rdbs.each do |rdb| res = Recog::Nizer.match(rdb, note.data[k]) matches << res if res end end end else # Add all generic match results to the overall match array normalize_scanner_fp(note).each do |m| next unless m matches << m end end matches end
Recog matches for the `s` service.
@param s [Mdm::Service] @return [Array<Hash>] Keys will be host, service, and os attributes
# File lib/mdm/host/operating_system_normalization.rb, line 199 def recog_matches_for_service(s) # # We assume that the service.info field contains certain types of probe # replies and associate these with one or more Recog databases. The mapping # of service.name to a specific database only fits into so many places and # Mdm currently serves that role. # service_match_keys = { # TODO: Implement smb.generic fingerprint database # 'smb' => [ 'smb.generic' ], # Distinct from smb.fingerprint, use os.certainty to choose best match # 'netbios' => [ 'smb.generic' ], # Distinct from smb.fingerprint, use os.certainty to choose best match 'ssh' => [ 'ssh.banner' ], # Recog expects just the vendor string, not the protocol version 'http' => [ 'http_header.server', 'apache_os'], # The 'Apache' fingerprints try to infer OS/distribution from the extra information in the Server header 'https' => [ 'http_header.server', 'apache_os'], # XXX: verify vmware esx(i) case on https (TODO: normalize https to http, track SSL elsewhere, such as a new set of fields) 'snmp' => [ 'snmp.sys_description' ], 'telnet' => [ 'telnet.banner' ], 'smtp' => [ 'smtp.banner' ], 'imap' => [ 'imap4.banner' ], # Metasploit reports 143/993 as imap (TODO: normalize imap to imap4) 'pop3' => [ 'pop3.banner' ], # Metasploit reports 110/995 as pop3 'nntp' => [ 'nntp.banner' ], 'ftp' => [ 'ftp.banner' ], 'ssdp' => [ 'ssdp_header.server' ] } matches = [] return matches unless service_match_keys.has_key?(s.name) service_match_keys[s.name].each do |rdb| banner = s.info if self.respond_to?("service_banner_recog_filter_#{s.name}") banner = self.send("service_banner_recog_filter_#{s.name}", banner) end res = Recog::Nizer.match(rdb, banner) matches << res if res end matches end
Ensure that the host attribute is using ascii safe text and escapes any other byte value.
# File lib/mdm/host/operating_system_normalization.rb, line 481 def sanitize(text) Rex::Text.ascii_safe_hex(text) end
Determine if the fingerprint data is readable. If not, it nearly always means that there was a problem with the YAML or the Marshal'ed data, so let's log that for later investigation.
# File lib/mdm/host/operating_system_normalization.rb, line 294 def validate_fingerprint_data(fp) if fp.data.kind_of?(Hash) and !fp.data.empty? return true elsif fp.ntype == "postgresql.fingerprint" # Special case postgresql.fingerprint; it's always a string, # and should not be used for OS fingerprinting (yet), so # don't bother logging it. TODO: fix os fingerprint finding, this # name collision seems silly. return false else return false end end