Java Code Examples for org.apache.hadoop.fs.StorageType#isTransient()

The following examples show how to use org.apache.hadoop.fs.StorageType#isTransient() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: BlockStoragePolicy.java    From hadoop with Apache License 2.0 6 votes vote down vote up 
/**
 * @return a list of {@link StorageType}s for storing the replicas of a block.
 */
public List<StorageType> chooseStorageTypes(final short replication) {
  final List<StorageType> types = new LinkedList<StorageType>();
  int i = 0, j = 0;

  // Do not return transient storage types. We will not have accurate
  // usage information for transient types.
  for (;i < replication && j < storageTypes.length; ++j) {
    if (!storageTypes[j].isTransient()) {
      types.add(storageTypes[j]);
      ++i;
    }
  }

  final StorageType last = storageTypes[storageTypes.length - 1];
  if (!last.isTransient()) {
    for (; i < replication; i++) {
      types.add(last);
    }
  }
  return types;
}
Example 2
Source File: BlockStoragePolicy.java    From big-c with Apache License 2.0 6 votes vote down vote up 
/**
 * @return a list of {@link StorageType}s for storing the replicas of a block.
 */
public List<StorageType> chooseStorageTypes(final short replication) {
  final List<StorageType> types = new LinkedList<StorageType>();
  int i = 0, j = 0;

  // Do not return transient storage types. We will not have accurate
  // usage information for transient types.
  for (;i < replication && j < storageTypes.length; ++j) {
    if (!storageTypes[j].isTransient()) {
      types.add(storageTypes[j]);
      ++i;
    }
  }

  final StorageType last = storageTypes[storageTypes.length - 1];
  if (!last.isTransient()) {
    for (; i < replication; i++) {
      types.add(last);
    }
  }
  return types;
}
Example 3
Source File: BlockReaderLocalLegacy.java    From hadoop with Apache License 2.0 5 votes vote down vote up 
private static BlockLocalPathInfo getBlockPathInfo(UserGroupInformation ugi,
    ExtendedBlock blk, DatanodeInfo node, Configuration conf, int timeout,
    Token<BlockTokenIdentifier> token, boolean connectToDnViaHostname,
    StorageType storageType) throws IOException {
  LocalDatanodeInfo localDatanodeInfo = getLocalDatanodeInfo(node.getIpcPort());
  BlockLocalPathInfo pathinfo = null;
  ClientDatanodeProtocol proxy = localDatanodeInfo.getDatanodeProxy(ugi, node,
      conf, timeout, connectToDnViaHostname);
  try {
    // make RPC to local datanode to find local pathnames of blocks
    pathinfo = proxy.getBlockLocalPathInfo(blk, token);
    // We cannot cache the path information for a replica on transient storage.
    // If the replica gets evicted, then it moves to a different path.  Then,
    // our next attempt to read from the cached path would fail to find the
    // file.  Additionally, the failure would cause us to disable legacy
    // short-circuit read for all subsequent use in the ClientContext.  Unlike
    // the newer short-circuit read implementation, we have no communication
    // channel for the DataNode to notify the client that the path has been
    // invalidated.  Therefore, our only option is to skip caching.
    if (pathinfo != null && !storageType.isTransient()) {
      if (LOG.isDebugEnabled()) {
        LOG.debug("Cached location of block " + blk + " as " + pathinfo);
      }
      localDatanodeInfo.setBlockLocalPathInfo(blk, pathinfo);
    }
  } catch (IOException e) {
    localDatanodeInfo.resetDatanodeProxy(); // Reset proxy on error
    throw e;
  }
  return pathinfo;
}
Example 4
Source File: BlockReaderLocalLegacy.java    From big-c with Apache License 2.0 5 votes vote down vote up 
private static BlockLocalPathInfo getBlockPathInfo(UserGroupInformation ugi,
    ExtendedBlock blk, DatanodeInfo node, Configuration conf, int timeout,
    Token<BlockTokenIdentifier> token, boolean connectToDnViaHostname,
    StorageType storageType) throws IOException {
  LocalDatanodeInfo localDatanodeInfo = getLocalDatanodeInfo(node.getIpcPort());
  BlockLocalPathInfo pathinfo = null;
  ClientDatanodeProtocol proxy = localDatanodeInfo.getDatanodeProxy(ugi, node,
      conf, timeout, connectToDnViaHostname);
  try {
    // make RPC to local datanode to find local pathnames of blocks
    pathinfo = proxy.getBlockLocalPathInfo(blk, token);
    // We cannot cache the path information for a replica on transient storage.
    // If the replica gets evicted, then it moves to a different path.  Then,
    // our next attempt to read from the cached path would fail to find the
    // file.  Additionally, the failure would cause us to disable legacy
    // short-circuit read for all subsequent use in the ClientContext.  Unlike
    // the newer short-circuit read implementation, we have no communication
    // channel for the DataNode to notify the client that the path has been
    // invalidated.  Therefore, our only option is to skip caching.
    if (pathinfo != null && !storageType.isTransient()) {
      if (LOG.isDebugEnabled()) {
        LOG.debug("Cached location of block " + blk + " as " + pathinfo);
      }
      localDatanodeInfo.setBlockLocalPathInfo(blk, pathinfo);
    }
  } catch (IOException e) {
    localDatanodeInfo.resetDatanodeProxy(); // Reset proxy on error
    throw e;
  }
  return pathinfo;
}