module Mdm::Host::OperatingSystemNormalization

Rules for operating system fingerprinting in Metasploit

The `os.product` key identifies the common-name of a specific operating system Examples include: Linux, Windows XP, Mac OS X, IOS, AIX, HP-UX, VxWorks

The `os.version` key identifies the service pack or version of the operating system Sometimes this means a kernel or firmware version when the distribution or OS version is not available. Examples include: SP2, 10.04, 2.6.47, 10.6.1

The `os.vendor` key identifies the manufacturer of the operating system Examples include: Microsoft, Ubuntu, Cisco, HP, IBM, Wind River

The `os.family` key identifies the group of the operating system. This is often a duplicate of os.product, unless a more specific product name is available. Examples include: Windows, Linux, IOS, HP-UX, AIX

The `os.edition` key identifies the specific variant of the operating system Examples include: Enterprise, Professional, Starter, Evaluation, Home, Datacenter

An example breakdown of a common operating system is shown below

* Microsoft Windows XP Professional Service Pack 3 English (x86)
   - os.product  = 'Windows XP'
   - os.edition  = 'Professional'
   - os.vendor   = 'Microsoft'
   - os.version  = 'SP3'
   - os.language = 'English'
   - os.arch     = 'x86'

These rules are then mapped to the {Mdm::Host} attributes below:

* os_name     - Maps to a normalized os.product key
* os_flavor   - Maps to a normalized os.edition key
* os_sp       - Maps to a normalized os.version key (soon os_version)
* os_lang     - Maps to a normalized os.language key
* arch        - Maps to a normalized os.arch key

Additional rules include the following mappings:

* name        - Maps to the host.name key
* mac         - Maps to the host.mac key

The following keys are not mapped to {Mdm::Host} at this time (but should be):

* os.vendor

In order to execute these rules, this module is responsible for mapping various fingerprint sources to {Mdm::Host} values. This requires some ugly glue code to account for differences between each supported input (external scanners), the Recog gem and associated databases, and how Metasploit itself likes to handle these values. Getting a mapping wrong is often harmless, but can impact the automatic targetting capabilities of certain exploit modules.

In other words, this is a best-effort attempt to rationalize multiple competing sources of information about a host and come up with the values representing a normalized assessment of the system. The use of `Recog` and multiple scanner fingerprints can result in a comprehensive (and confident) identification of the remote operating system and associated services.

Historically, there are direct conflicts between certain Metasploit modules, certain scanners, and external fingerprint databases in terms of how a particular OS and patch level is represented. This module attempts to fix what it can and serve as documentation and live workarounds for the rest.

Examples of known conflicts that are still in progress:

Maintenance:

  1. Ensure that the latest Recog gem is present and installed

  2. For new operating system releases, update relevant sections a) Windows releases will require updates to a few methods

    1) parse_windows_os_str()
    2) normalize_nmap_fingerprint()
    3) normalize_nexpose_fingerprint()
    4) Other scanner normalizers

    b) Mobile operating systems are minimally recognized

@todo Handle OS icon incompatiblities with new fingerprint names

Note that VMWare ESX(i) was special cased before as well, make sure it still works
1) Cisco IOS -> IOS breaks the icon mapping in MSP/MSCE of /cisco/
2) Ubuntu Linux -> Linux breaks the distro selection
The real solution is to add os_vendor and take this into account for icons

@todo Implement rspec coverage for #normalize_os() @todo Implement smb.generic fingerprint database (replace {#parse_windows_os_str}?) @todo Implement Samba version matching for specific distributions and OS versions @todo Implement DD-WRT and various embedded device signatures currently missing @todo Correct inconsistencies in os_name use by removing the vendor string (Microsoft Windows -> Windows)

This applies to MSF core and a handful of modules, not to mention some Recog fingerprints.

@todo Rename host.os_sp to host.os_version @todo Add host.os_vendor @todo Add host.os_confidence @todo Add host.domain

Constants

MAX_NMAP_CERTAINTY

Cap nmap certainty at 0.84 until we update it more frequently XXX: Without this, Nmap will beat the default certainty of recog

matches and its less-confident guesses will take precedence
over service-based fingerprints.

Public Instance Methods

apply_match_to_host(match) click to toggle source

Examine the assertations of the merged best match and map these back to fields of {Mdm::Host}. Take particular care not to leave related fields (os_*) in a conflicting state, leverage existing values where possible, and use the most confident values we have.

# File lib/mdm/host/operating_system_normalization.rb, line 367
def apply_match_to_host(match)
  host = self

  # These values in a match always override the current value unless
  # the host attribute has been explicitly locked by the user

  if match['host.mac'] && !host.attribute_locked?(:mac)
    host.mac = sanitize(match['host.mac'])
  end

  if match['host.name'] && !host.attribute_locked?(:name)
    host.name = sanitize(match['host.name'])
  end

  # Select the os architecture if available
  if match['os.arch'] && !host.attribute_locked?(:arch)
    host.arch = sanitize(match['os.arch'])
  end

  # Guess the purpose using some basic heuristics
  if ! host.attribute_locked?(:purpose)
    host.purpose = guess_purpose_from_match(match)
  end

  #
  # Map match fields from Recog fingerprint style to Metasploit style
  #

  # os.build:                 Examples: 9001, 2600, 7602
  # os.device:                Examples: General, ADSL Modem, Broadband router, Cable Modem, Camera, Copier, CSU/DSU
  # os.edition:               Examples: Web, Storage, HPC, MultiPoint, Enterprise, Home, Starter, Professional
  # os.family:                Examples: Windows, Linux, Solaris, NetWare, ProCurve, Mac OS X, HP-UX, AIX
  # os.product:               Examples: Windows, Linux, Windows Server 2008 R2, Windows XP, Enterprise Linux, NEO Tape Library
  # os.vendor:                Examples: Microsoft, HP, IBM, Sun, 3Com, Ricoh, Novell, Ubuntu, Apple, Cisco, Xerox
  # os.version:               Examples: SP1, SP2, 6.5 SP3 CPR, 10.04, 8.04, 12.10, 4.0, 6.1, 8.5
  # os.language:              Examples: English, Arabic, German
  # linux.kernel.version:     Examples: 2.6.32

  # Metasploit currently ignores os.build, os.device, and os.vendor as separate fields.

  # Select the OS name from os.name, fall back to os.family
  if ! host.attribute_locked?(:os_name)
    # Try to fill this value from os.product first if it exists
    if match.has_key?('os.product')
      host.os_name = sanitize(match['os.product'])
    else
      # Fall back to os.family otherwise, if available
      if match.has_key?('os.family')
        host.os_name = sanitize(match['os.family'])
      end
    end
  end

  # Select the flavor from os.edition if available
  if match.has_key?('os.edition') and ! host.attribute_locked?(:os_flavor)
    host.os_flavor = sanitize(match['os.edition'])
  end

  # Select an OS version as os.version, fall back to linux.kernel.version
  if ! host.attribute_locked?(:os_sp)
    if match['os.version']
      host.os_sp = sanitize(match['os.version'])
    else
      if match['linux.kernel.version']
        host.os_sp = sanitize(match['linux.kernel.version'])
      end
    end
  end

  # Select the os language if available
  if match.has_key?('os.language') and ! host.attribute_locked?(:os_lang)
    host.os_lang = sanitize(match['os.language'])
  end

  # Normalize MAC addresses to lower-case colon-delimited format
  if host.mac and ! host.attribute_locked?(:mac)
    host.mac = host.mac.downcase
    if host.mac =~ /^[a-f0-9]{12}$/
      host.mac = host.mac.scan(/../).join(':')
    end
  end

end
get_arch_from_string(str) click to toggle source

Return a normalized architecture based on patterns in the input string. This will identify things like sparc, powerpc, x86_x64, and i686

# File lib/mdm/host/operating_system_normalization.rb, line 892
def get_arch_from_string(str)
  res = Recog::Nizer.match("architecture", str)
  return unless (res and res['os.arch'])
  res['os.arch']
end
guess_purpose_from_match(match) click to toggle source

Loosely guess the purpose of a device based on available match values. In the future, also take into account the exposed services and rename to guess_purpose_with_match()

# File lib/mdm/host/operating_system_normalization.rb, line 456
def guess_purpose_from_match(match)
  # Create a string based on all match values
  pstr = match.values.join(' ').downcase

  # Loosely map keywords to specific purposes
  case pstr
  when /windows server|windows (nt|20)/
    'server'
  when /windows (xp|vista|[78])/
    'client'
  when /printer|print server/
    'printer'
  when /router/
    'router'
  when /firewall/
    'firewall'
  when /linux/
    'server'
  else
    'device'
  end
end
normalize_fusionvm_fingerprint(data) click to toggle source

Normalize data from FusionVM fingerprints

# File lib/mdm/host/operating_system_normalization.rb, line 756
def normalize_fusionvm_fingerprint(data)
  ret = {}
  case data[:os]
    when /Windows/
      ret.update(parse_windows_os_str(data[:os]))
    when /Linux ([^[:space:]]*) ([^[:space:]]*) .* (\(.*\))/
      ret['os.product'] = "Linux"
      ret['host.name']  = $1
      ret['os.version'] = $2
      ret['os.arch']    = get_arch_from_string($3)
    else
      ret['os.product'] = data[:os]
  end
  ret['os.arch'] = data[:arch] if data[:arch]
  ret['host.name'] = data[:name] if data[:name]
  [ ret ]
end
normalize_generic_fingerprint(data) click to toggle source

Normalize data from generic fingerprints

# File lib/mdm/host/operating_system_normalization.rb, line 777
def normalize_generic_fingerprint(data)
  ret = {}
  ret['os.product'] = data[:os_name] || data[:os] || data[:os_fingerprint] || "Unknown"
  ret['os.arch'] = data[:os_arch] if data[:os_arch]
  ret['os.certainty'] = data[:os_certainty] || 0.5
  [ ret ]
end
normalize_match(m) click to toggle source

Normalize matches in order to handle inconsistencies between fingerprint sources and our desired usage in Metasploit. This amounts to yet more duct tape, but the situation should improve as the fingerprint sources are updated and enhanced. In the future, this method will no longer be needed (or at least, doing less and less work)

# File lib/mdm/host/operating_system_normalization.rb, line 315
def normalize_match(m)
  # Normalize os.version strings containing 'Service Pack X' to just 'SPX'
  if m['os.version'] and m['os.version'].index('Service Pack ') == 0
    m['os.version'] = m['os.version'].gsub(/Service Pack /, 'SP')
  end

  if m['os.product']

    # Normalize Apple Mac OS X to just Mac OS X
    if m['os.product'] =~ /^Apple Mac/
      m['os.product']  = m['os.product'].gsub(/Apple Mac/, 'Mac')
      m['os.vendor'] ||= 'Apple'
    end

    # Normalize Sun Solaris/Sun SunOS to just Solaris/SunOS
    if m['os.product'] =~ /^Sun (Solaris|SunOS)/
      m['os.product']  = m['os.product'].gsub(/^Sun /, '')
      m['os.vendor'] ||= 'Oracle'
    end

    # Normalize Microsoft Windows to just Windows to catch any stragglers
    if m['os.product'] =~ /^Microsoft Windows/
      m['os.product']  = m['os.product'].gsub(/Microsoft Windows/, 'Windows')
      m['os.vendor'] ||= 'Microsoft'
    end

    # Normalize Windows Server to just Windows to match Metasploit target names
    if m['os.product'] =~ /^Windows Server/
      m['os.product'] = m['os.product'].gsub(/Windows Server/, 'Windows')
    end
  end

  m
end
normalize_mbsa_fingerprint(data) click to toggle source

Normalize data from MBSA fingerprints

# File lib/mdm/host/operating_system_normalization.rb, line 551
def normalize_mbsa_fingerprint(data)
  ret = {}
  # :os_match=>"Microsoft Windows Vista SP0 or SP1, Server 2008, or Windows 7 Ultimate (build 7000)"
  #    :os_vendor=>"Microsoft" :os_family=>"Windows" :os_version=>"7" :os_accuracy=>"100"
  ret['os.certainty'] = ( data[:os_accuracy].to_f / 100.0 ).to_s if data[:os_accuracy]
  ret['os.family']    = data[:os_family] if data[:os_family]
  ret['os.vendor']    = data[:os_vendor] if data[:os_vendor]

  if data[:os_family] and data[:os_version]
    ret['os.product'] = data[:os_family] + " " + data[:os_version]
  end

  ret['host.name'] = data[:hostname] if data[:hostname]

  [ ret ]
end
normalize_nessus_fingerprint(data) click to toggle source

Normalize data from Nessus fingerprints

# File lib/mdm/host/operating_system_normalization.rb, line 656
def normalize_nessus_fingerprint(data)
  ret = {}
  # :os=>"Microsoft Windows 2000 Advanced Server (English)"
  # :os=>"Microsoft Windows 2000\nMicrosoft Windows XP"
  # :os=>"Linux Kernel 2.6"
  # :os=>"Sun Solaris 8"
  # :os=>"IRIX 6.5"

  # Nessus sometimes jams multiple OS names together with a newline.
  oses = data[:os].split(/\n/)
  if oses.length > 1
    # Multiple fingerprints means Nessus wasn't really sure, reduce
    # the certainty accordingly
    ret['os.certainty'] = 0.5
  else
    ret['os.certainty'] = 0.8
  end

  # Since there is no confidence associated with them, the best we
  # can do is just take the first one.
  case oses.first
    when /^(Microsoft |)Windows/
      ret.update(parse_windows_os_str(data[:os]))

    when /(2\.[46]\.\d+[-a-zA-Z0-9]+)/
      # Look for older Linux kernel versions
      ret['os.product'] = "Linux"
      ret['os.version'] = $1

    when /^Linux Kernel ([\d\.]+)(.*)/
      # Look for strings like "Linux Kernel 2.6 on Ubuntu 9.10 (karmic)"
      # Ex: Linux Kernel 2.2 on Red Hat Linux release 6.2 (Zoot)
      # Ex: Linux Kernel 2.6 on Ubuntu Linux 8.04 (hardy)
      ret['os.product'] = "Linux"
      ret['os.version'] = $1

      vendor = $2.to_s

      # Try to snag the vendor name as well
      if vendor =~ /on (\w+|\w+ \w+|\w+ \w+ \w+) (Linux|\d)/
        ret['os.vendor'] = $1
      end

    when /(.*) ([0-9\.]+)$/
      # Then we don't necessarily know what the os is, but this fingerprint has
      # some version information at the end, pull it off, treat the first part
      # as the OS, and the rest as the version.
      ret['os.product'] = $1.gsub("Kernel", '').strip
      ret['os.version'] = $2
    else
      # TODO: Return each OS guess as a separate match
      ret['os.product'] = oses.first
  end

  ret['host.name'] = data[:hname] if data[:hname]
  [ ret ]
end
normalize_nexpose_fingerprint(data) click to toggle source

Normalize data from Nexpose fingerprints

# File lib/mdm/host/operating_system_normalization.rb, line 572
def normalize_nexpose_fingerprint(data)
  ret = {}
  # :family=>"Windows" :certainty=>"0.85" :vendor=>"Microsoft" :product=>"Windows 7 Ultimate Edition"
  # :family=>"Windows" :certainty=>"0.67" :vendor=>"Microsoft" :arch=>"x86" :product=>'Windows 7' :version=>'SP1'
  # :family=>"Linux" :certainty=>"0.64" :vendor=>"Linux" :product=>"Linux"
  # :family=>"Linux" :certainty=>"0.80" :vendor=>"Ubuntu" :product=>"Linux"
  # :family=>"IOS" :certainty=>"0.80" :vendor=>"Cisco" :product=>"IOS"
  # :family=>"embedded" :certainty=>"0.61" :vendor=>"Linksys" :product=>"embedded"

  ret['os.certainty'] = data[:certainty] if data[:certainty]
  ret['os.family']    = data[:family]    if data[:family]
  ret['os.vendor']    = data[:vendor]    if data[:vendor]

  case data[:product]
  when /^Windows/

    # TODO: Verify Windows CE and Windows 8 RT fingerprints
    # Translate the version into the representation we want

    case data[:version].to_s

    # These variants are normalized to just 'Windows <Version>'
    when "NT", "2000", "95", "ME", "XP", "Vista", "7", "8", "8.1"
      ret['os.product'] = "Windows #{data[:version]}"

    # Service pack in the version field should be recognized
    when /^SP\d+/, /^Service Pack \d+/
      ret['os.product'] = data[:product]
      ret['os.version'] = data[:version]

    # No version means the version is part of the product already
    when nil, ''
      # Trim any 'Server' suffix and use as it is
      ret['os.product'] = data[:product].sub(/ Server$/, '')

    # Otherwise, we assume a Server version of Windows
    else
      ret['os.product'] = "Windows Server #{data[:version]}"
    end

    # Extract the edition string if it is present
    if data[:product] =~ /(XP|Vista|\d+(?:\.\d+)) (\w+|\w+ \w+|\w+ \w+ \w+) Edition/
      ret['os.edition'] = $2
    end

  when nil, 'embedded'
    # Use the family or vendor name when the product is empty or 'embedded'
    ret['os.product']   = data[:family] unless data[:family] == 'embedded'
    ret['os.product'] ||= data[:vendor]
    ret['os.version']   = data[:version] if data[:version]
  else
    # Default to using the product name reported by Nexpose
    ret['os.product'] = data[:product] if data[:product]
  end

  ret['os.arch'] = get_arch_from_string(data[:arch]) if data[:arch]
  ret['os.arch'] ||= get_arch_from_string(data[:desc]) if data[:desc]

  [ ret ]
end
normalize_nmap_fingerprint(data) click to toggle source

Normalize data from Nmap fingerprints

# File lib/mdm/host/operating_system_normalization.rb, line 520
def normalize_nmap_fingerprint(data)
  ret = {}

  # :os_vendor=>"Microsoft" :os_family=>"Windows" :os_version=>"2000" :os_accuracy=>"94"
  ret['os.certainty'] = ( data[:os_accuracy].to_f / 100.0 ).to_s if data[:os_accuracy]
  if (data[:os_vendor] == data[:os_family])
    ret['os.product'] = data[:os_family]
  else
    ret['os.product'] = data[:os_family]
    ret['os.vendor'] = data[:os_vendor]
  end

  # Nmap places the type of Windows (XP, 7, etc) into the version field
  if ret['os.product'] == 'Windows' and data[:os_version]
    ret['os.product'] = ret['os.product'] + ' ' + data[:os_version].to_s
  else
    ret['os.version'] = data[:os_version]
  end

  ret['host.name'] = data[:hostname] if data[:hostname]

  if ret['os.certainty']
    ret['os.certainty'] = [ ret['os.certainty'].to_f, MAX_NMAP_CERTAINTY ].min.to_s
  end

  [ ret ]
end
normalize_os() click to toggle source

Normalize the operating system fingerprints provided by various scanners (nmap, nexpose, retina, nessus, metasploit modules, and more!)

These are stored as {Mdm::Note notes} (instead of directly in the os_* fields) specifically for this purpose.

The goal is to infer as much as we can about the OS of the device and the various {Mdm::Service services} offered using the Recog gem and some glue logic to determine the best weights. This method can result in changes to the recorded {#os_name}, {#os_flavor}, {#os_sp}, {#os_lang}, {#purpose}, {#name}, {#arch}, and the {Mdm::Service service details}.

# File lib/mdm/host/operating_system_normalization.rb, line 151
def normalize_os
  host   = self
  matches = []

  # Note that we're already restricting the query to this host by using
  # host.notes instead of Note, so don't need a host_id in the
  # conditions.
  fingerprintable_notes = self.notes.where("ntype like '%%fingerprint'")
  fingerprintable_notes.each do |fp_note|
    matches += recog_matches_for_note(fp_note)
  end

  # XXX: This hack solves the memory leak generated by self.services.each {}
  fingerprintable_services = self.services.where("name is not null and name != '' and info is not null and info != ''")
  fingerprintable_services.each do |s|
    matches += recog_matches_for_service(s)
  end

  #
  # Look for generic fingerprint.match notes that generate a match hash from modules
  # This handles ad-hoc os.language, host.name, etc identifications
  #
  generated_matches = self.notes.where(ntype: 'fingerprint.match')
  generated_matches.each do |m|
    next unless (m.data and m.data.kind_of?(::Hash))
    matches << m.data.dup
  end

  # Normalize matches for consistency during the ranking phase
  matches = matches.map{ |m| normalize_match(m) }

  # Calculate the best OS match based on fingerprint hits
  match = Recog::Nizer.best_os_match(matches)

  # Merge and normalize the best match to the host object
  apply_match_to_host(match) if match

  # Set some sane defaults if needed
  host.os_name ||= 'Unknown'
  host.purpose ||= 'device'

  host.save if host.changed?
end
normalize_qualys_fingerprint(data) click to toggle source

Normalize data from Qualys fingerprints

# File lib/mdm/host/operating_system_normalization.rb, line 717
def normalize_qualys_fingerprint(data)
  ret = {}
  # :os=>"Microsoft Windows 2000"
  # :os=>"Windows 2003"
  # :os=>"Microsoft Windows XP Professional SP3"
  # :os=>"Ubuntu Linux"
  # :os=>"Cisco IOS 12.0(3)T3"
  # :os=>"Red-Hat Linux 6.0"
  case data[:os]
    when /Windows/
      ret.update(parse_windows_os_str(data[:os]))

    when /^(Cisco) (IOS) (\d+[^\s]+)/
      ret['os.product'] = $2
      ret['os.vendor']  = $1
      ret['os.version'] = $3

    when /^([^\s]+) (Linux)(.*)/
      ret['os.product'] = $2
      ret['os.vendor'] = $1

      ver = $3.to_s.strip.split(/\s+/).first
      if ver =~ /^\d+\./
        ret['os.version'] = ver
      end

    else
      parts = data[:os].split(/\s+/, 3)
      ret['os.product'] = "Unknown"
      ret['os.product'] = parts[0] if parts[0]
      ret['os.product'] << " " + parts[1] if parts[1]
      ret['os.version'] = parts[2] if parts[2]
  end
  [ ret ]
end
normalize_retina_fingerprint(data) click to toggle source

Normalize data from Retina fingerprints

# File lib/mdm/host/operating_system_normalization.rb, line 637
def normalize_retina_fingerprint(data)
  ret = {}
  # :os=>"Windows Server 2003 (X64), Service Pack 2"
  case data[:os]
    when /Windows/
      ret.update(parse_windows_os_str(data[:os]))
    else
      # No idea what this looks like if it isn't windows.  Just store
      # the whole thing and hope for the best.
      # TODO: Add examples of non-Windows results
      ret['os.product'] = data[:os] if data[:os]
  end
  [ ret ]
end
normalize_scanner_fp(fp) click to toggle source

Convert a host.os.*_fingerprint Note into a hash containing 'os.*' and 'host.*' fields

Also includes a os.certainty which is a float from 0 - 1.00 indicating the scanner's confidence in its fingerprint. If the particular scanner does not provide such information, default to 0.80.

# File lib/mdm/host/operating_system_normalization.rb, line 792
def normalize_scanner_fp(fp)
  hits = []

  return hits if not validate_fingerprint_data(fp)

  case fp.ntype
  when /^host\.os\.(.*_fingerprint)$/
    pname = $1
    pmeth = 'normalize_' + pname
    if self.respond_to?(pmeth)
      hits = self.send(pmeth, fp.data)
    else
      hits = normalize_generic_fingerprint(fp.data)
    end
  end
  hits.each {|hit| hit['os.certainty'] ||= 0.80}
  hits
end
normalize_session_fingerprint(data) click to toggle source

Normalize data from Meterpreter's client.sys.config.sysinfo()

# File lib/mdm/host/operating_system_normalization.rb, line 488
  def normalize_session_fingerprint(data)
    ret = {}
    case data[:os]
      when /Windows/
        ret.update(parse_windows_os_str(data[:os]))
    # Switch to this code block once the multi-meterpreter code review is complete

      when /^(Windows \w+)\s*\(Build (\d+)(.*)\)/
        ret['os.product'] = $1
        ret['os.build'] = $2
        ret['os.vendor'] = 'Microsoft'
        possible_sp = $3
        if possible_sp =~ /Service Pack (\d+)/
          ret['os.version'] = 'SP' + $1
        end
      when /Linux (\d+\.\d+\.\d+\S*)\s* \((\w*)\)/
        ret['os.product'] = "Linux"
        ret['os.version'] = $1
        ret['os.arch']    = get_arch_from_string($2)
      else
        ret['os.product'] = data[:os]
    end
    ret['os.arch'] = data[:arch] if data[:arch]
    ret['host.name'] = data[:name] if data[:name]
    [ ret ]
  end
parse_windows_os_str(str) click to toggle source

Take a windows version string and return a hash with fields suitable for Host this object's version fields. This is used as a fall-back to parse external fingerprints and should eventually be replaced by per-source mappings.

A few example strings that this will have to parse: sessions

Windows XP (Build 2600, Service Pack 3).
Windows .NET Server (Build 3790).
Windows 2008 (Build 6001, Service Pack 1).

retina

Windows Server 2003 (X64), Service Pack 2

nessus

Microsoft Windows 2000 Advanced Server (English)

qualys

Microsoft Windows XP Professional SP3
Windows 2003

Note that this list doesn't include nexpose or nmap, since they are both kind enough to give us the various strings in seperate pieces that we don't have to parse out manually.

# File lib/mdm/host/operating_system_normalization.rb, line 834
def parse_windows_os_str(str)
  ret = {}

  # Set some reasonable defaults for Windows
  ret['os.vendor']  = 'Microsoft'
  ret['os.product'] = 'Windows'

  # Determine the actual Windows product name
  case str
    when /\.NET Server/
      ret['os.product'] << ' Server 2003'
    when / (2000|2003|2008|2012)/
      ret['os.product'] << ' Server ' + $1
    when / (NT (?:3\.51|4\.0))/
      ret['os.product'] << ' ' + $1
    when /Windows (95|98|ME|XP|Vista|[\d\.]+)/
      ret['os.product'] << ' ' + $1
    else
      # If we couldn't pull out anything specific for the flavor, just cut
      # off the stuff we know for sure isn't it and hope for the best
      ret['os.product'] = (ret['os.product'] + ' ' + str.gsub(/(Microsoft )|(Windows )|(Service Pack|SP) ?(\d+)/i, '').strip).strip

      # Make sure the product name doesn't include any non-alphanumeric stuff
      # This fixes cases where the above code leaves 'Windows XX (Build 3333,)...'
      ret['os.product'] = ret['os.product'].split(/[^a-zA-Z0-9 ]/).first.strip

  end

  # Take a guess at the architecture
  arch = get_arch_from_string(str)
  ret['os.arch'] = arch if arch

  # Extract any service pack value in the string
  if str =~ /(Service Pack|SP) ?(\d+)/i
    ret['os.version']  = "SP#{$2}"
  end

  # Extract any build ID found in the string
  if str =~ /build (\d+)/i
    ret['os.build'] = $1
  end

  # Extract the OS edition if available
  if str =~ /(\d+|\d+\.\d+) (\w+|\w+ \w+|\w+ \w+ \w+) Edition/
    ret['os.edition'] = $2
  else
    if str =~ /(Professional|Enterprise|Pro|Home|Start|Datacenter|Web|Storage|MultiPoint)/
      ret['os.edition'] = $1
    end
  end

  ret
end
recog_matches_for_note(note) click to toggle source

Recog matches for the fingerprint in `note`.

@return [Array<Hash>] Keys will be host, service, and os attributes

# File lib/mdm/host/operating_system_normalization.rb, line 244
def recog_matches_for_note(note)
  # Skip notes that are missing the correct structure or have been blacklisted
  return [] if not validate_fingerprint_data(note)

  #
  # These rules define the relationship between fingerprint note keys
  # and specific Recog databases for detailed matching. Notes that do
  # not match a rule are passed to the generic matcher.
  #
  fingerprint_note_match_keys = {
    'smb.fingerprint'  => {
      :native_os               => [ 'smb.native_os' ],
    },
    'http.fingerprint' => {
      :header_server           => [ 'http_header.server', 'apache_os' ],
      :header_set_cookie       => [ 'http_header.cookie' ],
      :header_www_authenticate => [ 'http_header.wwwauth' ],
    # TODO: Candidates for future Recog support
    # :content                 => 'http_body'
    # :code                    => 'http_response_code'
    # :message                 => 'http_response_message'
    }
  }

  matches = []

  # Look for a specific Recog database for this type and data key
  if fingerprint_note_match_keys.has_key?( note.ntype )
    fingerprint_note_match_keys[ note.ntype ].each_pair do |k,rdbs|
      if note.data.has_key?(k)
        rdbs.each do |rdb|
          res = Recog::Nizer.match(rdb, note.data[k])
          matches << res if res
        end
      end
    end
  else
    # Add all generic match results to the overall match array
    normalize_scanner_fp(note).each do |m|
      next unless m
      matches << m
    end
  end

  matches
end
recog_matches_for_service(s) click to toggle source

Recog matches for the `s` service.

@param s [Mdm::Service] @return [Array<Hash>] Keys will be host, service, and os attributes

# File lib/mdm/host/operating_system_normalization.rb, line 199
def recog_matches_for_service(s)
  #
  # We assume that the service.info field contains certain types of probe
  # replies and associate these with one or more Recog databases. The mapping
  # of service.name to a specific database only fits into so many places and
  # Mdm currently serves that role.
  #

  service_match_keys = {
    # TODO: Implement smb.generic fingerprint database
    # 'smb'     => [ 'smb.generic' ], # Distinct from smb.fingerprint, use os.certainty to choose best match
    # 'netbios' => [ 'smb.generic' ], # Distinct from smb.fingerprint, use os.certainty to choose best match

    'ssh'     => [ 'ssh.banner' ], # Recog expects just the vendor string, not the protocol version
    'http'    => [ 'http_header.server', 'apache_os'], # The 'Apache' fingerprints try to infer OS/distribution from the extra information in the Server header
    'https'   => [ 'http_header.server', 'apache_os'], # XXX: verify vmware esx(i) case on https (TODO: normalize https to http, track SSL elsewhere, such as a new set of fields)
    'snmp'    => [ 'snmp.sys_description' ],
    'telnet'  => [ 'telnet.banner' ],
    'smtp'    => [ 'smtp.banner' ],
    'imap'    => [ 'imap4.banner' ],  # Metasploit reports 143/993 as imap (TODO: normalize imap to imap4)
    'pop3'    => [ 'pop3.banner' ],   # Metasploit reports 110/995 as pop3
    'nntp'    => [ 'nntp.banner' ],
    'ftp'     => [ 'ftp.banner' ],
    'ssdp'    => [ 'ssdp_header.server' ]
  }

  matches = []

  return matches unless service_match_keys.has_key?(s.name)

  service_match_keys[s.name].each do |rdb|
    banner = s.info
    if self.respond_to?("service_banner_recog_filter_#{s.name}")
      banner = self.send("service_banner_recog_filter_#{s.name}", banner)
    end
    res = Recog::Nizer.match(rdb, banner)
    matches << res if res
  end

  matches
end
sanitize(text) click to toggle source

Ensure that the host attribute is using ascii safe text and escapes any other byte value.

# File lib/mdm/host/operating_system_normalization.rb, line 481
def sanitize(text)
  Rex::Text.ascii_safe_hex(text)
end
service_banner_recog_filter_ssh(banner) click to toggle source

Recog assumes that the protocol version of the SSH banner has been removed

# File lib/mdm/host/operating_system_normalization.rb, line 353
def service_banner_recog_filter_ssh(banner)
  if banner =~ /^SSH-\d+\.\d+-(.*)/
    $1
  else
    banner
  end
end
validate_fingerprint_data(fp) click to toggle source

Determine if the fingerprint data is readable. If not, it nearly always means that there was a problem with the YAML or the Marshal'ed data, so let's log that for later investigation.

# File lib/mdm/host/operating_system_normalization.rb, line 294
def validate_fingerprint_data(fp)
  if fp.data.kind_of?(Hash) and !fp.data.empty?
    return true
  elsif fp.ntype == "postgresql.fingerprint"
    # Special case postgresql.fingerprint; it's always a string,
    # and should not be used for OS fingerprinting (yet), so
    # don't bother logging it. TODO: fix os fingerprint finding, this
    # name collision seems silly.
    return false
  else
    return false
  end
end