## net.sf.saxon.functions
Class Tokenize

    java.lang.Object
      net.sf.saxon.expr.Expression
          net.sf.saxon.expr.FunctionCall
              net.sf.saxon.functions.SystemFunction
                  net.sf.saxon.functions.Tokenize

All Implemented Interfaces:
  java.io.Serializable, javax.xml.transform.SourceLocator, LocationProvider, SaxonLocator, EvaluableItem, SequenceIterable, InstructionInfo, org.xml.sax.Locator

    public class Tokenize

extends SystemFunction

This class implements the tokenize() function for regular expression matching. This returns a sequence of strings representing the unmatched substrings: the separators which match the regular expression are not returned.

See Also:
  Serialized Form

Field Summary

Fields inherited from class net.sf.saxon.functions.SystemFunction
`operation`

Fields inherited from class net.sf.saxon.expr.FunctionCall
`argument`

Fields inherited from class net.sf.saxon.expr.Expression
`EVALUATE_METHOD, ITERATE_METHOD, locationId, PROCESS_METHOD, staticProperties`

Constructor Summary
`Tokenize()`

Method Summary
`RegularExpression` `getCompiledRegularExpression()` Get the compiled regular expression if available, otherwise return null
`SequenceIterator` `iterate(XPathContext c)` Iterate over the results of the function
`static void` `main(java.lang.String[] args)` Simple command-line interface for testing.
`Expression` `optimize(ExpressionVisitor visitor, ItemType contextItemType)` Perform optimisation of an expression and its subexpressions.
`Expression` `simplify(ExpressionVisitor visitor)` Simplify and validate.

Methods inherited from class net.sf.saxon.functions.SystemFunction
`addContextDocumentArgument, addDocToPathMap, checkArguments, computeCardinality, computeSpecialProperties, copy, getDetails, getErrorCodeForTypeErrors, getItemType, getRequiredType, makeSystemFunction, setDetails, useContextItemAsDefault`

Methods inherited from class net.sf.saxon.expr.FunctionCall
`addExternalFunctionCallToPathMap, checkArgumentCount, equals, explain, getArguments, getDisplayName, getFunctionName, getNumberOfArguments, hashCode, iterateSubExpressions, preEvaluate, promote, replaceSubExpression, setArguments, setFunctionName, simplifyArguments, toString, typeCheck`

Methods inherited from class net.sf.saxon.expr.Expression

`addToPathMap, adoptChildExpression, checkForUpdatingSubexpressions, checkPermittedContents, computeDependencies, computeStaticProperties, display, doPromotion, dynamicError, effectiveBooleanValue, evaluateAsString, evaluateItem, evaluatePendingUpdates, explain, findParentOf, getCardinality, getColumnNumber, getColumnNumber, getConstructType, getContainer, getDependencies, getExecutable, getHostLanguage, getImplementationMethod, getIntrinsicDependencies, getLineNumber, getLineNumber, getLocationId, getLocationProvider, getObjectName, getProperties, getProperty, getPublicId, getSlotsUsed, getSpecialProperties, getSystemId, getSystemId, hasLoopingSubexpression, implementsStaticTypeCheck, isUpdatingExpression, iterateEvents, markTailFunctionCalls, process, resetLocalStaticProperties, setContainer, setFiltered, setFlattened, setLocationId, staticTypeCheck, suppressValidation, typeError`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

Constructor Detail

### Tokenize

    public Tokenize()

Method Detail

### simplify

    public Expression simplify(ExpressionVisitor visitor)
                        throws XPathException

  Simplify and validate. This is a pure function so it can be simplified in advance if the arguments are known

  Overrides:
    `simplify` in class `FunctionCall`
  Parameters:
    `visitor` - an expression visitor
  Returns:
    the simplified expression
  Throws:
    `XPathException` - if an error is discovered during expression rewriting

### optimize

    public Expression optimize(ExpressionVisitor visitor,
                               ItemType contextItemType)
                        throws XPathException

  Perform optimisation of an expression and its subexpressions.

  This method is called after all references to functions and variables have been resolved to the declaration of the function or variable, and after all type checking has been done.

  Overrides:
    `optimize` in class `SystemFunction`
  Parameters:
    `visitor` - an expression visitor
    `contextItemType` - the static type of "." at the point where this expression is invoked. The parameter is set to null if it is known statically that the context item will be undefined. If the type of the context item is not known statically, the argument is set to `Type.ITEM_TYPE`
  Returns:
    the original expression, rewritten if appropriate to optimize execution
  Throws:
    `XPathException` - if an error is discovered during this phase (typically a type error)

### getCompiledRegularExpression

    public RegularExpression getCompiledRegularExpression()

  Get the compiled regular expression if available, otherwise return null

  Returns:
    the compiled regular expression, or null

### iterate

    public SequenceIterator iterate(XPathContext c)
                             throws XPathException

  Iterate over the results of the function

  Specified by:
    `iterate` in interface `SequenceIterable`
  Overrides:
    `iterate` in class `Expression`
  Parameters:
    `c` - supplies the context for evaluation
  Returns:
    a SequenceIterator that can be used to iterate over the result of the expression
  Throws:
    `XPathException` - if any dynamic error occurs evaluating the expression

### main

    public static void main(java.lang.String[] args)
                     throws java.lang.Exception

  Simple command-line interface for testing.

  Parameters:
    `args` - (1) the string to be tokenized (2) the regular expression
  Throws:
    `java.lang.Exception`

Copyright (c) Saxonica Limited. All rights reserved.