org.apache.sysds.runtime.util

## Class DataConverter

- java.lang.Object
- - org.apache.sysds.runtime.util.DataConverter

-     public class DataConverter
      extends Object

  This class provides methods to read and write matrix blocks from to HDFS using different data formats. Those functionalities are used especially for CP read/write and exporting in-memory matrices to HDFS (before executing MR jobs).

- - ### Constructor Summary

    Constructors
    Constructor and Description
    `DataConverter()`

  - ### Method Summary

    All Methods Static Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static org.apache.commons.math3.linear.Array2DRowRealMatrix` | `convertToArray2DRowRealMatrix(MatrixBlock mb)` Helper method that converts SystemDS matrix variable (`varname`) into a Array2DRowRealMatrix format, which is useful in invoking Apache CommonsMath.
    `static org.apache.commons.math3.linear.BlockRealMatrix` | `convertToBlockRealMatrix(MatrixBlock mb)`
    `static boolean[]` | `convertToBooleanVector(MatrixBlock mb)`
    `static DenseBlock` | `convertToDenseBlock(MatrixBlock mb)`
    `static DenseBlock` | `convertToDenseBlock(MatrixBlock mb, boolean deep)`
    `static List<Double>` | `convertToDoubleList(MatrixBlock mb)`
    `static double[][]` | `convertToDoubleMatrix(MatrixBlock mb)` Creates a two-dimensional double matrix of the input matrix block.
    `static double[]` | `convertToDoubleVector(MatrixBlock mb)`
    `static double[]` | `convertToDoubleVector(MatrixBlock mb, boolean deep)`
    `static double[]` | `convertToDoubleVector(MatrixBlock mb, boolean deep, boolean allowNull)`
    `static FrameBlock` | `convertToFrameBlock(MatrixBlock mb)` Converts a matrix block into a frame block of value type double.
    `static FrameBlock` | `convertToFrameBlock(MatrixBlock mb, Types.ValueType vt)` Converts a matrix block into a frame block of a given value type.
    `static FrameBlock` | `convertToFrameBlock(MatrixBlock mb, Types.ValueType[] schema)`
    `static FrameBlock` | `convertToFrameBlock(String[][] data)` Converts a two dimensions string array into a frame block of value type string.
    `static FrameBlock` | `convertToFrameBlock(String[][] data, Types.ValueType[] schema)`
    `static FrameBlock` | `convertToFrameBlock(String[][] data, Types.ValueType[] schema, String[] colnames)`
    `static int[]` | `convertToIntVector(MatrixBlock mb)`
    `static long[]` | `convertToLongVector(MatrixBlock mb)`
    `static MatrixBlock` | `convertToMatrixBlock(CTableMap map)`
    `static MatrixBlock` | `convertToMatrixBlock(CTableMap map, int rlen, int clen)` NOTE: this method also ensures the specified matrix dimensions
    `static MatrixBlock` | `convertToMatrixBlock(double[][] data)` Creates a dense Matrix Block and copies the given double matrix into it.
    `static MatrixBlock` | `convertToMatrixBlock(double[] data, boolean columnVector)` Creates a dense Matrix Block and copies the given double vector into it.
    `static MatrixBlock` | `convertToMatrixBlock(FrameBlock frame)` Converts a frame block with arbitrary schema into a matrix block.
    `static MatrixBlock` | `convertToMatrixBlock(HashMap<MatrixIndexes,Double> map)`
    `static MatrixBlock` | `convertToMatrixBlock(HashMap<MatrixIndexes,Double> map, int rlen, int clen)` NOTE: this method also ensures the specified matrix dimensions
    `static MatrixBlock` | `convertToMatrixBlock(int[][] data)` Converts an Integer matrix to an MatrixBlock
    `static MatrixBlock` | `convertToMatrixBlock(org.apache.commons.math3.linear.RealMatrix rm)`
    `static MatrixBlock[]` | `convertToMatrixBlockPartitions(MatrixBlock mb, boolean colwise)`
    `static String[][]` | `convertToStringFrame(FrameBlock frame)` Converts a frame block with arbitrary schema into a two dimensional string array.
    `static TensorBlock` | `convertToTensorBlock(MatrixBlock mb, Types.ValueType vt, boolean toBasicTensor)`
    `static void` | `copyToDoubleVector(MatrixBlock mb, double[] dest, int destPos)`
    `static int[]` | `getTensorDimensions(ExecutionContext ec, CPOperand dims)`
    `static MatrixBlock` | `readMatrixFromHDFS(ReadProperties prop)` Core method for reading matrices in format textcell, matrixmarket, binarycell, or binaryblock from HDFS into main memory.
    `static MatrixBlock` | `readMatrixFromHDFS(String dir, Types.FileFormat fmt, long rlen, long clen, int blen)`
    `static MatrixBlock` | `readMatrixFromHDFS(String dir, Types.FileFormat fmt, long rlen, long clen, int blen, boolean localFS)`
    `static MatrixBlock` | `readMatrixFromHDFS(String dir, Types.FileFormat fmt, long rlen, long clen, int blen, long expectedNnz)`
    `static MatrixBlock` | `readMatrixFromHDFS(String dir, Types.FileFormat fmt, long rlen, long clen, int blen, long expectedNnz, boolean localFS)`
    `static MatrixBlock` | `readMatrixFromHDFS(String dir, Types.FileFormat fmt, long rlen, long clen, int blen, long expectedNnz, FileFormatProperties formatProperties)`
    `static TensorBlock` | `readTensorFromHDFS(String dir, Types.FileFormat fmt, long[] dims, int blen, Types.ValueType[] schema)`
    `static BitSet` | `toBitSet(double[] data)`
    `static double[]` | `toDouble(BitSet data, int len)`
    `static double[]` | `toDouble(float[] data)`
    `static double[]` | `toDouble(int[] data)`
    `static double[]` | `toDouble(long[] data)`
    `static double[]` | `toDouble(String[] data)`
    `static float[]` | `toFloat(double[] data)`
    `static int[]` | `toInt(double[] data)`
    `static long[]` | `toLong(double[] data)`
    `static String[]` | `toString(double[] data)`
    `static String` | `toString(FrameBlock fb)`
    `static String` | `toString(FrameBlock fb, boolean sparse, String separator, String lineseparator, int rowsToPrint, int colsToPrint, int decimal)`
    `static String` | `toString(ListObject list, int rows, int cols, boolean sparse, String separator, String lineSeparator, int rowsToPrint, int colsToPrint, int decimal)`
    `static String` | `toString(MatrixBlock mb)`
    `static String` | `toString(MatrixBlock mb, boolean sparse, String separator, String lineseparator, int rowsToPrint, int colsToPrint, int decimal)` Returns a string representation of a matrix
    `static String` | `toString(TensorBlock tb)`
    `static String` | `toString(TensorBlock tb, boolean sparse, String separator, String lineseparator, String leftBorder, String rightBorder, int rowsToPrint, int colsToPrint, int decimal)` Returns a string representation of a tensor
    `static void` | `writeMatrixToHDFS(MatrixBlock mat, String dir, Types.FileFormat fmt, DataCharacteristics dc)`
    `static void` | `writeMatrixToHDFS(MatrixBlock mat, String dir, Types.FileFormat fmt, DataCharacteristics dc, int replication, FileFormatProperties formatProperties)`
    `static void` | `writeMatrixToHDFS(MatrixBlock mat, String dir, Types.FileFormat fmt, DataCharacteristics dc, int replication, FileFormatProperties formatProperties, boolean diag)`
    `static void` | `writeTensorToHDFS(TensorBlock tensor, String dir, Types.FileFormat fmt, DataCharacteristics dc)`

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### DataConverter

          public DataConverter()

  - ### Method Detail

    - #### writeMatrixToHDFS

          public static void writeMatrixToHDFS(MatrixBlock mat,
                                               String dir,
                                               Types.FileFormat fmt,
                                               DataCharacteristics dc)
                                        throws IOException

      Throws:
        `IOException`

    - #### writeMatrixToHDFS

          public static void writeMatrixToHDFS(MatrixBlock mat,
                                               String dir,
                                               Types.FileFormat fmt,
                                               DataCharacteristics dc,
                                               int replication,
                                               FileFormatProperties formatProperties)
                                        throws IOException

      Throws:
        `IOException`

    - #### writeMatrixToHDFS

          public static void writeMatrixToHDFS(MatrixBlock mat,
                                               String dir,
                                               Types.FileFormat fmt,
                                               DataCharacteristics dc,
                                               int replication,
                                               FileFormatProperties formatProperties,
                                               boolean diag)
                                        throws IOException

      Throws:
        `IOException`

    - #### writeTensorToHDFS

          public static void writeTensorToHDFS(TensorBlock tensor,
                                               String dir,
                                               Types.FileFormat fmt,
                                               DataCharacteristics dc)
                                        throws IOException

      Throws:
        `IOException`

    - #### readMatrixFromHDFS

          public static MatrixBlock readMatrixFromHDFS(String dir,
                                                       Types.FileFormat fmt,
                                                       long rlen,
                                                       long clen,
                                                       int blen,
                                                       boolean localFS)
                                                throws IOException

      Throws:
        `IOException`

    - #### readMatrixFromHDFS

          public static MatrixBlock readMatrixFromHDFS(String dir,
                                                       Types.FileFormat fmt,
                                                       long rlen,
                                                       long clen,
                                                       int blen)
                                                throws IOException

      Throws:
        `IOException`

    - #### readMatrixFromHDFS

          public static MatrixBlock readMatrixFromHDFS(String dir,
                                                       Types.FileFormat fmt,
                                                       long rlen,
                                                       long clen,
                                                       int blen,
                                                       long expectedNnz)
                                                throws IOException

      Throws:
        `IOException`

    - #### readMatrixFromHDFS

          public static MatrixBlock readMatrixFromHDFS(String dir,
                                                       Types.FileFormat fmt,
                                                       long rlen,
                                                       long clen,
                                                       int blen,
                                                       long expectedNnz,
                                                       boolean localFS)
                                                throws IOException

      Throws:
        `IOException`

    - #### readMatrixFromHDFS

          public static MatrixBlock readMatrixFromHDFS(String dir,
                                                       Types.FileFormat fmt,
                                                       long rlen,
                                                       long clen,
                                                       int blen,
                                                       long expectedNnz,
                                                       FileFormatProperties formatProperties)
                                                throws IOException

      Throws:
        `IOException`

    - #### readTensorFromHDFS

          public static TensorBlock readTensorFromHDFS(String dir,
                                                       Types.FileFormat fmt,
                                                       long[] dims,
                                                       int blen,
                                                       Types.ValueType[] schema)
                                                throws IOException

      Throws:
        `IOException`

    - #### readMatrixFromHDFS

          public static MatrixBlock readMatrixFromHDFS(ReadProperties prop)
                                                throws IOException

      Core method for reading matrices in format textcell, matrixmarket, binarycell, or binaryblock from HDFS into main memory. For expected dense matrices we directly copy value- or block-at-a-time into the target matrix. In contrast, for sparse matrices, we append (column-value)-pairs and do a final sort if required in order to prevent large reorg overheads and increased memory consumption in case of unordered inputs. DENSE MxN input: * best/average/worst: O(M*N) SPARSE MxN input * best (ordered, or binary block w/ clen<=blen): O(M*N) * average (unordered): O(M*N*log(N)) * worst (descending order per row): O(M * N^2) NOTE: providing an exact estimate of 'expected sparsity' can prevent a full copy of the result matrix block (required for changing sparse->dense, or vice versa)
      Parameters:
        `prop` - read properties
      Returns:
        matrix block
      Throws:
        `IOException` - if IOException occurs

    - #### convertToDoubleMatrix

          public static double[][] convertToDoubleMatrix(MatrixBlock mb)

      Creates a two-dimensional double matrix of the input matrix block.
      Parameters:
        `mb` - matrix block
      Returns:
        2d double array

    - #### convertToBooleanVector

          public static boolean[] convertToBooleanVector(MatrixBlock mb)

    - #### convertToIntVector

          public static int[] convertToIntVector(MatrixBlock mb)

    - #### convertToLongVector

          public static long[] convertToLongVector(MatrixBlock mb)

    - #### convertToDenseBlock

          public static DenseBlock convertToDenseBlock(MatrixBlock mb)

    - #### convertToDenseBlock

          public static DenseBlock convertToDenseBlock(MatrixBlock mb,
                                                       boolean deep)

    - #### convertToDoubleVector

          public static double[] convertToDoubleVector(MatrixBlock mb)

    - #### convertToDoubleVector

          public static double[] convertToDoubleVector(MatrixBlock mb,
                                                       boolean deep)

    - #### convertToDoubleVector

          public static double[] convertToDoubleVector(MatrixBlock mb,
                                                       boolean deep,
                                                       boolean allowNull)

    - #### convertToDoubleList

          public static List<Double> convertToDoubleList(MatrixBlock mb)

    - #### convertToMatrixBlock

          public static MatrixBlock convertToMatrixBlock(double[][] data)

      Creates a dense Matrix Block and copies the given double matrix into it.
      Parameters:
        `data` - 2d double array
      Returns:
        matrix block

    - #### convertToMatrixBlock

          public static MatrixBlock convertToMatrixBlock(int[][] data)

      Converts an Integer matrix to an MatrixBlock
      Parameters:
        `data` - Int matrix input that is converted to double MatrixBlock
      Returns:
        The matrixBlock constructed.

    - #### convertToMatrixBlock

          public static MatrixBlock convertToMatrixBlock(double[] data,
                                                         boolean columnVector)

      Creates a dense Matrix Block and copies the given double vector into it.
      Parameters:
        `data` - double array
        `columnVector` - if true, create matrix with single column. if false, create matrix with single row
      Returns:
        matrix block

    - #### convertToMatrixBlock

          public static MatrixBlock convertToMatrixBlock(HashMap<MatrixIndexes,Double> map)

    - #### convertToMatrixBlock

          public static MatrixBlock convertToMatrixBlock(HashMap<MatrixIndexes,Double> map,
                                                         int rlen,
                                                         int clen)

      NOTE: this method also ensures the specified matrix dimensions
      Parameters:
        `map` - map of matrix index keys and double values
        `rlen` - number of rows
        `clen` - number of columns
      Returns:
        matrix block

    - #### convertToMatrixBlock

          public static MatrixBlock convertToMatrixBlock(CTableMap map)

    - #### convertToMatrixBlock

          public static MatrixBlock convertToMatrixBlock(CTableMap map,
                                                         int rlen,
                                                         int clen)

      NOTE: this method also ensures the specified matrix dimensions
      Parameters:
        `map` - ?
        `rlen` - number of rows
        `clen` - number of columns
      Returns:
        matrix block

    - #### convertToMatrixBlock

          public static MatrixBlock convertToMatrixBlock(FrameBlock frame)

      Converts a frame block with arbitrary schema into a matrix block. Since matrix block only supports value type double, we do a best effort conversion of non-double types which might result in errors for non-numerical data.
      Parameters:
        `frame` - frame block
      Returns:
        matrix block

    - #### convertToStringFrame

          public static String[][] convertToStringFrame(FrameBlock frame)

      Converts a frame block with arbitrary schema into a two dimensional string array.
      Parameters:
        `frame` - frame block
      Returns:
        2d string array

    - #### convertToFrameBlock

          public static FrameBlock convertToFrameBlock(String[][] data)

      Converts a two dimensions string array into a frame block of value type string. If the given array is null or of length 0, we return an empty frame block.
      Parameters:
        `data` - 2d string array
      Returns:
        frame block

    - #### convertToFrameBlock

          public static FrameBlock convertToFrameBlock(String[][] data,
                                                       Types.ValueType[] schema)

    - #### convertToFrameBlock

          public static FrameBlock convertToFrameBlock(String[][] data,
                                                       Types.ValueType[] schema,
                                                       String[] colnames)

    - #### convertToFrameBlock

          public static FrameBlock convertToFrameBlock(MatrixBlock mb)

      Converts a matrix block into a frame block of value type double.
      Parameters:
        `mb` - matrix block
      Returns:
        frame block of type double

    - #### convertToFrameBlock

          public static FrameBlock convertToFrameBlock(MatrixBlock mb,
                                                       Types.ValueType vt)

      Converts a matrix block into a frame block of a given value type.
      Parameters:
        `mb` - matrix block
        `vt` - value type
      Returns:
        frame block

    - #### convertToFrameBlock

          public static FrameBlock convertToFrameBlock(MatrixBlock mb,
                                                       Types.ValueType[] schema)

    - #### convertToTensorBlock

          public static TensorBlock convertToTensorBlock(MatrixBlock mb,
                                                         Types.ValueType vt,
                                                         boolean toBasicTensor)

    - #### convertToMatrixBlockPartitions

          public static MatrixBlock[] convertToMatrixBlockPartitions(MatrixBlock mb,
                                                                     boolean colwise)

    - #### convertToArray2DRowRealMatrix

          public static org.apache.commons.math3.linear.Array2DRowRealMatrix convertToArray2DRowRealMatrix(MatrixBlock mb)

      Helper method that converts SystemDS matrix variable (`varname`) into a Array2DRowRealMatrix format, which is useful in invoking Apache CommonsMath.
      Parameters:
        `mb` - matrix object
      Returns:
        matrix as a commons-math3 Array2DRowRealMatrix

    - #### convertToBlockRealMatrix

          public static org.apache.commons.math3.linear.BlockRealMatrix convertToBlockRealMatrix(MatrixBlock mb)

    - #### convertToMatrixBlock

          public static MatrixBlock convertToMatrixBlock(org.apache.commons.math3.linear.RealMatrix rm)

    - #### copyToDoubleVector

          public static void copyToDoubleVector(MatrixBlock mb,
                                                double[] dest,
                                                int destPos)

    - #### toString

          public static String toString(MatrixBlock mb)

    - #### toString

          public static String toString(MatrixBlock mb,
                                        boolean sparse,
                                        String separator,
                                        String lineseparator,
                                        int rowsToPrint,
                                        int colsToPrint,
                                        int decimal)

      Returns a string representation of a matrix
      Parameters:
        `mb` - matrix block
        `sparse` - if true, string will contain a table with row index, col index, value (where value != 0.0) otherwise it will be a rectangular string with all values of the matrix block
        `separator` - Separator string between each element in a row, or between the columns in sparse format
        `lineseparator` - Separator string between each row
        `rowsToPrint` - maximum number of rows to print, -1 for all
        `colsToPrint` - maximum number of columns to print, -1 for all
        `decimal` - number of decimal places to print, -1 for default
      Returns:
        matrix as a string

    - #### toString

          public static String toString(TensorBlock tb)

    - #### toString

          public static String toString(TensorBlock tb,
                                        boolean sparse,
                                        String separator,
                                        String lineseparator,
                                        String leftBorder,
                                        String rightBorder,
                                        int rowsToPrint,
                                        int colsToPrint,
                                        int decimal)

      Returns a string representation of a tensor
      Parameters:
        `tb` - tensor block
        `sparse` - if true, string will contain a table with row index, col index, value (where value != 0.0) otherwise it will be a rectangular string with all values of the tensor block
        `separator` - Separator string between each element in a row, or between the columns in sparse format
        `lineseparator` - Separator string between each row
        `leftBorder` - Characters placed at the start of a new dimension level
        `rightBorder` - Characters placed at the end of a new dimension level
        `rowsToPrint` - maximum number of rows to print, -1 for all
        `colsToPrint` - maximum number of columns to print, -1 for all
        `decimal` - number of decimal places to print, -1 for default
      Returns:
        tensor as a string

    - #### toString

          public static String toString(FrameBlock fb)

    - #### toString

          public static String toString(FrameBlock fb,
                                        boolean sparse,
                                        String separator,
                                        String lineseparator,
                                        int rowsToPrint,
                                        int colsToPrint,
                                        int decimal)

    - #### toString

          public static String toString(ListObject list,
                                        int rows,
                                        int cols,
                                        boolean sparse,
                                        String separator,
                                        String lineSeparator,
                                        int rowsToPrint,
                                        int colsToPrint,
                                        int decimal)

    - #### getTensorDimensions

          public static int[] getTensorDimensions(ExecutionContext ec,
                                                  CPOperand dims)

    - #### toDouble

          public static double[] toDouble(float[] data)

    - #### toDouble

          public static double[] toDouble(long[] data)

    - #### toDouble

          public static double[] toDouble(int[] data)

    - #### toDouble

          public static double[] toDouble(BitSet data,
                                          int len)

    - #### toDouble

          public static double[] toDouble(String[] data)

    - #### toFloat

          public static float[] toFloat(double[] data)

    - #### toInt

          public static int[] toInt(double[] data)

    - #### toLong

          public static long[] toLong(double[] data)

    - #### toBitSet

          public static BitSet toBitSet(double[] data)

    - #### toString

          public static String[] toString(double[] data)

Copyright © 2021 The Apache Software Foundation. All rights reserved.