Class BufferedTokenizer
In: lib/em/buftok.rb
Parent: Object

BufferedTokenizer takes a delimiter upon instantiation, or acts line-based by default. It allows input to be spoon-fed from some outside source which receives arbitrary length datagrams which may-or-may-not contain the token by which entities are delimited.

By default, new BufferedTokenizers will operate on lines delimited by "\n" by default or allow you to specify any delimiter token you so choose, which will then be used by String#split to tokenize the input data

@example Using BufferedTokernizer to parse lines out of incoming data

  module LineBufferedConnection
    def receive_data(data)
      (@buffer ||= BufferedTokenizer.new).extract(data).each do |line|
        receive_line(line)
      end
    end
  end

@author Tony Arcieri @author Martin Emde

Methods

empty?   extract   flush   new  

Public Class methods

@param [String] delimiter @param [Integer] size_limit

Public Instance methods

@return [Boolean]

Extract takes an arbitrary string of input data and returns an array of tokenized entities, provided there were any available to extract.

@example

  tokenizer.extract(data).
    map { |entity| Decode(entity) }.each { ... }

@param [String] data

Flush the contents of the input buffer, i.e. return the input buffer even though a token has not yet been encountered.

@return [String]

[Validate]