org.apache.sysds.runtime.controlprogram.parfor

## Class RemoteParForUtils

- java.lang.Object
- - org.apache.sysds.runtime.controlprogram.parfor.RemoteParForUtils

-     public class RemoteParForUtils
      extends Object

  Common functionalities for parfor workers in MR jobs. Used by worker wrappers in mappers (base RemoteParFor) and reducers (fused data partitioning and parfor)

- - ### Constructor Summary

    Constructors
    Constructor and Description
    `RemoteParForUtils()`

  - ### Method Summary

    All Methods Static Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static void` | `cleanupWorkingDirectories()` Cleanup all temporary files created by this SystemDS process.
    `static void` | `cleanupWorkingDirectoriesOnShutdown()` Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.
    `static void` | `exportLineageItems(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, Lineage lineage)` Export lineage for remote Spark parfor workers.
    `static ArrayList<String>` | `exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars)` For remote Spark parfor workers.
    `static void` | `exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, HashMap<String,String> rvarFnames, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out)` For remote MR parfor workers.
    `static void` | `exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out)`
    `static Lineage[]` | `getLineages(LocalVariableMap[] results)`
    `static LocalVariableMap[]` | `getResults(List<scala.Tuple2<Long,String>> out, org.apache.commons.logging.Log LOG)`
    `static void` | `incrementParForMRCounters(org.apache.hadoop.mapred.Reporter reporter, long deltaTasks, long deltaIterations)`
    `static void` | `setupBufferPool(long workerID)` Init and register-cleanup of buffer pool

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### RemoteParForUtils

          public RemoteParForUtils()

  - ### Method Detail

    - #### incrementParForMRCounters

          public static void incrementParForMRCounters(org.apache.hadoop.mapred.Reporter reporter,
                                                       long deltaTasks,
                                                       long deltaIterations)

    - #### exportResultVariables

          public static void exportResultVariables(long workerID,
                                                   LocalVariableMap vars,
                                                   ArrayList<ParForStatementBlock.ResultVar> resultVars,
                                                   org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out)
                                            throws IOException

      Throws:
        `IOException`

    - #### exportResultVariables

          public static void exportResultVariables(long workerID,
                                                   LocalVariableMap vars,
                                                   ArrayList<ParForStatementBlock.ResultVar> resultVars,
                                                   HashMap<String,String> rvarFnames,
                                                   org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out)
                                            throws IOException

      For remote MR parfor workers.
      Parameters:
        `workerID` - worker id
        `vars` - local variable map
        `resultVars` - list of result variables
        `rvarFnames` - ?
        `out` - output collectors
      Throws:
        `IOException` - if IOException occurs

    - #### exportResultVariables

          public static ArrayList<String> exportResultVariables(long workerID,
                                                                LocalVariableMap vars,
                                                                ArrayList<ParForStatementBlock.ResultVar> resultVars)
                                                         throws IOException

      For remote Spark parfor workers. This is a simplified version compared to MR.
      Parameters:
        `workerID` - worker id
        `vars` - local variable map
        `resultVars` - list of result variables
      Returns:
        list of result variables
      Throws:
        `IOException` - if IOException occurs

    - #### exportLineageItems

          public static void exportLineageItems(long workerID,
                                                LocalVariableMap vars,
                                                ArrayList<ParForStatementBlock.ResultVar> resultVars,
                                                Lineage lineage)
                                         throws IOException

      Export lineage for remote Spark parfor workers.
      Parameters:
        `workerID` - worker id
        `vars` - local variable map
        `resultVars` - list of result variables
        `lineage` - lineage object
      Throws:
        `IOException` - if IOException occurs

    - #### cleanupWorkingDirectories

          public static void cleanupWorkingDirectories()

      Cleanup all temporary files created by this SystemDS process.

    - #### cleanupWorkingDirectoriesOnShutdown

          public static void cleanupWorkingDirectoriesOnShutdown()

      Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.

    - #### getLineages

          public static Lineage[] getLineages(LocalVariableMap[] results)

    - #### getResults

          public static LocalVariableMap[] getResults(List<scala.Tuple2<Long,String>> out,
                                                      org.apache.commons.logging.Log LOG)

    - #### setupBufferPool

          public static void setupBufferPool(long workerID)
                                      throws IOException

      Init and register-cleanup of buffer pool
      Parameters:
        `workerID` - worker id
      Throws:
        `IOException` - exception

Copyright © 2021 The Apache Software Foundation. All rights reserved.