org.apache.hc.core5.util

## Class Tokenizer

- java.lang.Object
- - org.apache.hc.core5.util.Tokenizer

- Direct Known Subclasses:
    TokenParser

      @Contract(threading=IMMUTABLE)
      public class Tokenizer
      extends Object

  Tokenizer that can be used as a foundation for more complex parsing routines. Methods of this class are designed to produce near zero intermediate garbage and make no intermediate copies of input data.

  This class is immutable and thread safe.

  Since:
    5.1

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `Tokenizer.Cursor`
    `static interface` | `Tokenizer.Delimiter` Represents a predicate whether the given character is a delimiter.

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static int` | `CR`
    `static char` | `DQUOTE` Double quote
    `static char` | `ESCAPE` Backward slash / escape character
    `static int` | `HT`
    `static Tokenizer` | `INSTANCE`
    `static int` | `LF`
    `static int` | `SP`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `Tokenizer()`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `copyContent(CharSequence buf, Tokenizer.Cursor cursor, BitSet bitSet, StringBuilder dst)` Deprecated. Use `copyContent(CharSequence, Cursor, Delimiter, StringBuilder)`
    `void` | `copyContent(CharSequence buf, Tokenizer.Cursor cursor, Tokenizer.Delimiter delimiterPredicate, StringBuilder dst)` Transfers content into the destination buffer until a whitespace character or any of the given delimiters is encountered.
    `void` | `copyQuotedContent(CharSequence buf, Tokenizer.Cursor cursor, StringBuilder dst)` Transfers content enclosed with quote marks into the destination buffer.
    `void` | `copyUnquotedContent(CharSequence buf, Tokenizer.Cursor cursor, BitSet bitSet, StringBuilder dst)` Deprecated. Use `copyUnquotedContent(CharSequence, Cursor, Delimiter, StringBuilder)`
    `void` | `copyUnquotedContent(CharSequence buf, Tokenizer.Cursor cursor, Tokenizer.Delimiter delimiterPredicate, StringBuilder dst)` Transfers content into the destination buffer until a whitespace character, a quote, or any of the given delimiters is encountered.
    `static Tokenizer.Delimiter` | `delimiters(BitSet delimiters)`
    `static Tokenizer.Delimiter` | `delimiters(char... delimiters)`
    `static Tokenizer.Delimiter` | `delimiters(char delimiter)`
    `static Tokenizer.Delimiter` | `delimiters(char delimiter1, char delimiter2)`
    `static Tokenizer.Delimiter` | `delimiters(char delimiter1, char delimiter2, char delimiter3)`
    `static BitSet` | `INIT_BITSET(int... b)` Deprecated. Do not use.
    `static boolean` | `isWhitespace(char ch)`
    `String` | `parseContent(CharSequence buf, Tokenizer.Cursor cursor, BitSet bitSet)` Deprecated. use `parseContent(CharSequence, Cursor, Delimiter)`
    `String` | `parseContent(CharSequence buf, Tokenizer.Cursor cursor, Tokenizer.Delimiter delimiterPredicate)` Extracts from the sequence of chars a token terminated with any of the given delimiters or a whitespace characters.
    `String` | `parseToken(CharSequence buf, Tokenizer.Cursor cursor, BitSet bitSet)` Deprecated. use `parseToken(CharSequence, Cursor, Delimiter)`
    `String` | `parseToken(CharSequence buf, Tokenizer.Cursor cursor, Tokenizer.Delimiter delimiterPredicate)` Extracts from the sequence of chars a token terminated with any of the given delimiters discarding semantically insignificant whitespace characters.
    `String` | `parseValue(CharSequence buf, Tokenizer.Cursor cursor, BitSet bitSet)` Deprecated. use `parseValue(CharSequence, Cursor, Delimiter)`
    `String` | `parseValue(CharSequence buf, Tokenizer.Cursor cursor, Tokenizer.Delimiter delimiterPredicate)` Extracts from the sequence of chars a value which can be enclosed in quote marks and terminated with any of the given delimiters discarding semantically insignificant whitespace characters.
    `void` | `skipWhiteSpace(CharSequence buf, Tokenizer.Cursor cursor)` Skips semantically insignificant whitespace characters and moves the cursor to the closest non-whitespace character.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### DQUOTE

          public static final char DQUOTE

      Double quote
      See Also:
        Constant Field Values

    - #### ESCAPE

          public static final char ESCAPE

      Backward slash / escape character
      See Also:
        Constant Field Values

    - #### CR

          public static final int CR

      See Also:
        Constant Field Values

    - #### LF

          public static final int LF

      See Also:
        Constant Field Values

    - #### SP

          public static final int SP

      See Also:
        Constant Field Values

    - #### HT

          public static final int HT

      See Also:
        Constant Field Values

    - #### INSTANCE

          public static final Tokenizer INSTANCE

  - ### Constructor Detail

    - #### Tokenizer

          public Tokenizer()

  - ### Method Detail

    - #### INIT_BITSET

          @Deprecated
          public static BitSet INIT_BITSET(int... b)

      Deprecated. Do not use.

    - #### isWhitespace

          public static boolean isWhitespace(char ch)

    - #### delimiters

          public static Tokenizer.Delimiter delimiters(BitSet delimiters)

      Since:
        5.3

    - #### delimiters

          public static Tokenizer.Delimiter delimiters(char... delimiters)

      Since:
        5.3

    - #### delimiters

          public static Tokenizer.Delimiter delimiters(char delimiter)

      Since:
        5.3

    - #### delimiters

          public static Tokenizer.Delimiter delimiters(char delimiter1,
                                                       char delimiter2)

      Since:
        5.3

    - #### delimiters

          public static Tokenizer.Delimiter delimiters(char delimiter1,
                                                       char delimiter2,
                                                       char delimiter3)

      Since:
        5.3

    - #### parseContent

          public String parseContent(CharSequence buf,
                                     Tokenizer.Cursor cursor,
                                     Tokenizer.Delimiter delimiterPredicate)

      Extracts from the sequence of chars a token terminated with any of the given delimiters or a whitespace characters.
      Parameters:
        `buf` - buffer with the sequence of chars to be parsed
        `cursor` - defines the bounds and current position of the buffer
        `delimiterPredicate` - delimiter predicate. Can be `null` if the token is not delimited by any character.

    - #### parseContent

          @Deprecated
          public String parseContent(CharSequence buf,
                                                 Tokenizer.Cursor cursor,
                                                 BitSet bitSet)

      Deprecated. use `parseContent(CharSequence, Cursor, Delimiter)`

    - #### parseToken

          public String parseToken(CharSequence buf,
                                   Tokenizer.Cursor cursor,
                                   Tokenizer.Delimiter delimiterPredicate)

      Extracts from the sequence of chars a token terminated with any of the given delimiters discarding semantically insignificant whitespace characters.
      Parameters:
        `buf` - buffer with the sequence of chars to be parsed
        `cursor` - defines the bounds and current position of the buffer
        `delimiterPredicate` - delimiter predicate. Can be `null` if the token is not delimited by any character.

    - #### parseToken

          @Deprecated
          public String parseToken(CharSequence buf,
                                               Tokenizer.Cursor cursor,
                                               BitSet bitSet)

      Deprecated. use `parseToken(CharSequence, Cursor, Delimiter)`

    - #### parseValue

          public String parseValue(CharSequence buf,
                                   Tokenizer.Cursor cursor,
                                   Tokenizer.Delimiter delimiterPredicate)

      Extracts from the sequence of chars a value which can be enclosed in quote marks and terminated with any of the given delimiters discarding semantically insignificant whitespace characters.
      Parameters:
        `buf` - buffer with the sequence of chars to be parsed
        `cursor` - defines the bounds and current position of the buffer
        `delimiterPredicate` - delimiter predicate. Can be `null` if the token is not delimited by any character.

    - #### parseValue

          @Deprecated
          public String parseValue(CharSequence buf,
                                               Tokenizer.Cursor cursor,
                                               BitSet bitSet)

      Deprecated. use `parseValue(CharSequence, Cursor, Delimiter)`

    - #### skipWhiteSpace

          public void skipWhiteSpace(CharSequence buf,
                                     Tokenizer.Cursor cursor)

      Skips semantically insignificant whitespace characters and moves the cursor to the closest non-whitespace character.
      Parameters:
        `buf` - buffer with the sequence of chars to be parsed
        `cursor` - defines the bounds and current position of the buffer

    - #### copyContent

          public void copyContent(CharSequence buf,
                                  Tokenizer.Cursor cursor,
                                  Tokenizer.Delimiter delimiterPredicate,
                                  StringBuilder dst)

      Transfers content into the destination buffer until a whitespace character or any of the given delimiters is encountered.
      Parameters:
        `buf` - buffer with the sequence of chars to be parsed
        `cursor` - defines the bounds and current position of the buffer
        `delimiterPredicate` - delimiter predicate. Can be `null` if the token is delimited by a whitespace only.
        `dst` - destination buffer

    - #### copyContent

          @Deprecated
          public void copyContent(CharSequence buf,
                                              Tokenizer.Cursor cursor,
                                              BitSet bitSet,
                                              StringBuilder dst)

      Deprecated. Use `copyContent(CharSequence, Cursor, Delimiter, StringBuilder)`

    - #### copyUnquotedContent

          public void copyUnquotedContent(CharSequence buf,
                                          Tokenizer.Cursor cursor,
                                          Tokenizer.Delimiter delimiterPredicate,
                                          StringBuilder dst)

      Transfers content into the destination buffer until a whitespace character, a quote, or any of the given delimiters is encountered.
      Parameters:
        `buf` - buffer with the sequence of chars to be parsed
        `cursor` - defines the bounds and current position of the buffer
        `delimiterPredicate` - delimiter predicate. Can be `null` if the token is delimited by a whitespace or a quote only.
        `dst` - destination buffer

    - #### copyUnquotedContent

          @Deprecated
          public void copyUnquotedContent(CharSequence buf,
                                                      Tokenizer.Cursor cursor,
                                                      BitSet bitSet,
                                                      StringBuilder dst)

      Deprecated. Use `copyUnquotedContent(CharSequence, Cursor, Delimiter, StringBuilder)`

    - #### copyQuotedContent

          public void copyQuotedContent(CharSequence buf,
                                        Tokenizer.Cursor cursor,
                                        StringBuilder dst)

      Transfers content enclosed with quote marks into the destination buffer.
      Parameters:
        `buf` - buffer with the sequence of chars to be parsed
        `cursor` - defines the bounds and current position of the buffer
        `dst` - destination buffer

Copyright © 2005–2021 The Apache Software Foundation. All rights reserved.