Package org.apache.sysds.runtime.controlprogram.parfor

## Class RemoteParForUtils

- java.lang.Object
- - org.apache.sysds.runtime.controlprogram.parfor.RemoteParForUtils

-     public class RemoteParForUtils
      extends Object

  Common functionalities for parfor workers in MR jobs. Used by worker wrappers in mappers (base RemoteParFor) and reducers (fused data partitioning and parfor)

- - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `RemoteParForUtils()` |

  - ### Method Summary

    All Methods Static Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `static void` | `cleanupWorkingDirectories()` | Cleanup all temporary files created by this SystemDS process.
    `static void` | `cleanupWorkingDirectoriesOnShutdown()` | Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.
    `static void` | `exportLineageItems​(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, Lineage lineage)` | Export lineage for remote Spark parfor workers.
    `static ArrayList<String>` | `exportResultVariables​(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars)` | For remote Spark parfor workers.
    `static Lineage[]` | `getLineages​(LocalVariableMap[] results)` |
    `static LocalVariableMap[]` | `getResults​(List<scala.Tuple2<Long,​String>> out, org.apache.commons.logging.Log LOG)` |
    `static void` | `incrementParForMRCounters​(org.apache.hadoop.mapred.Reporter reporter, long deltaTasks, long deltaIterations)` |
    `static void` | `setupBufferPool​(long workerID, boolean isLocal)` | Init and register-cleanup of buffer pool

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### RemoteParForUtils

          public RemoteParForUtils()

  - ### Method Detail

    - #### incrementParForMRCounters

          public static void incrementParForMRCounters​(org.apache.hadoop.mapred.Reporter reporter,
                                                       long deltaTasks,
                                                       long deltaIterations)

    - #### exportResultVariables

          public static ArrayList<String> exportResultVariables​(long workerID,
                                                                LocalVariableMap vars,
                                                                ArrayList<ParForStatementBlock.ResultVar> resultVars)
                                                         throws IOException

      For remote Spark parfor workers. This is a simplified version compared to MR.
      Parameters:
        `workerID` - worker id
        `vars` - local variable map
        `resultVars` - list of result variables
      Returns:
        list of result variables
      Throws:
        `IOException` - if IOException occurs

    - #### exportLineageItems

          public static void exportLineageItems​(long workerID,
                                                LocalVariableMap vars,
                                                ArrayList<ParForStatementBlock.ResultVar> resultVars,
                                                Lineage lineage)
                                         throws IOException

      Export lineage for remote Spark parfor workers.
      Parameters:
        `workerID` - worker id
        `vars` - local variable map
        `resultVars` - list of result variables
        `lineage` - lineage object
      Throws:
        `IOException` - if IOException occurs

    - #### cleanupWorkingDirectories

          public static void cleanupWorkingDirectories()

      Cleanup all temporary files created by this SystemDS process.

    - #### cleanupWorkingDirectoriesOnShutdown

          public static void cleanupWorkingDirectoriesOnShutdown()

      Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.

    - #### getLineages

          public static Lineage[] getLineages​(LocalVariableMap[] results)

    - #### getResults

          public static LocalVariableMap[] getResults​(List<scala.Tuple2<Long,​String>> out,
                                                      org.apache.commons.logging.Log LOG)

    - #### setupBufferPool

          public static void setupBufferPool​(long workerID,
                                             boolean isLocal)
                                      throws IOException

      Init and register-cleanup of buffer pool
      Parameters:
        `workerID` - worker id
        `isLocal` - in local spark mode (single JVM)
      Throws:
        `IOException` - exception