Java Code Examples for org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils#localGlobber()

The following examples show how to use org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils#localGlobber() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: HistoryFileManager.java    From hadoop with Apache License 2.0 6 votes vote down vote up 
/**
 * Scans the intermediate directory to find user directories. Scans these for
 * history files if the modification time for the directory has changed. Once
 * it finds history files it starts the process of moving them to the done 
 * directory.
 * 
 * @throws IOException
 *           if there was a error while scanning
 */
void scanIntermediateDirectory() throws IOException {
  // TODO it would be great to limit how often this happens, except in the
  // case where we are looking for a particular job.
  List<FileStatus> userDirList = JobHistoryUtils.localGlobber(
      intermediateDoneDirFc, intermediateDoneDirPath, "");
  LOG.debug("Scanning intermediate dirs");
  for (FileStatus userDir : userDirList) {
    String name = userDir.getPath().getName();
    UserLogDir dir = userDirModificationTimeMap.get(name);
    if(dir == null) {
      dir = new UserLogDir();
      UserLogDir old = userDirModificationTimeMap.putIfAbsent(name, dir);
      if(old != null) {
        dir = old;
      }
    }
    dir.scanIfNeeded(userDir);
  }
}
Example 2
Source File: HistoryFileManager.java    From big-c with Apache License 2.0 6 votes vote down vote up 
/**
 * Scans the intermediate directory to find user directories. Scans these for
 * history files if the modification time for the directory has changed. Once
 * it finds history files it starts the process of moving them to the done 
 * directory.
 * 
 * @throws IOException
 *           if there was a error while scanning
 */
void scanIntermediateDirectory() throws IOException {
  // TODO it would be great to limit how often this happens, except in the
  // case where we are looking for a particular job.
  List<FileStatus> userDirList = JobHistoryUtils.localGlobber(
      intermediateDoneDirFc, intermediateDoneDirPath, "");
  LOG.debug("Scanning intermediate dirs");
  for (FileStatus userDir : userDirList) {
    String name = userDir.getPath().getName();
    UserLogDir dir = userDirModificationTimeMap.get(name);
    if(dir == null) {
      dir = new UserLogDir();
      UserLogDir old = userDirModificationTimeMap.putIfAbsent(name, dir);
      if(old != null) {
        dir = old;
      }
    }
    dir.scanIfNeeded(userDir);
  }
}
Example 3
Source File: HistoryFileManager.java    From hadoop with Apache License 2.0 2 votes vote down vote up 
/**
 * Finds all history directories with a timestamp component by scanning the
 * filesystem. Used when the JobHistory server is started.
 * 
 * @return list of history directories
 */
protected List<FileStatus> findTimestampedDirectories() throws IOException {
  List<FileStatus> fsList = JobHistoryUtils.localGlobber(doneDirFc,
      doneDirPrefixPath, DONE_BEFORE_SERIAL_TAIL);
  return fsList;
}
Example 4
Source File: HistoryFileManager.java    From big-c with Apache License 2.0 2 votes vote down vote up 
/**
 * Finds all history directories with a timestamp component by scanning the
 * filesystem. Used when the JobHistory server is started.
 * 
 * @return list of history directories
 */
protected List<FileStatus> findTimestampedDirectories() throws IOException {
  List<FileStatus> fsList = JobHistoryUtils.localGlobber(doneDirFc,
      doneDirPrefixPath, DONE_BEFORE_SERIAL_TAIL);
  return fsList;
}