Package org.apache.sysds.runtime.transform.tokenize

## Interface TokenizerPre

- All Superinterfaces:
    `Serializable`
  All Known Implementing Classes:
    `TokenizerPreNgram`, `TokenizerPreWhitespaceSplit`

      public interface TokenizerPre
      extends Serializable

- - ### Method Summary

    All Methods Instance Methods Abstract Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens>` | `tokenizePre​(FrameBlock in)` |

- - ### Method Detail

    - #### tokenizePre

          List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens> tokenizePre​(FrameBlock in)