Java Code Examples for org.apache.hadoop.hdfs.DFSUtil#convertToDFS()

The following examples show how to use org.apache.hadoop.hdfs.DFSUtil#convertToDFS() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: DistCp.java    From RDFS with Apache License 2.0 5 votes vote down vote up 
/**
 * @param srcs   source paths
 * @param dst    destination path
 * @param conf
 * @return True if, all source paths and destination path are DFS
 *         locations, and they are from the same DFS clusters. If
 *         it can't find the DFS cluster name since an older server
 *         build, it will assume cluster name matches.
 * @throws IOException
 */
static public boolean canUseFastCopy(List<Path> srcs, Path dst,
    Configuration conf) throws IOException {
  DistributedFileSystem dstdfs = DFSUtil.convertToDFS(dst
      .getFileSystem(conf));
  if (dstdfs == null) {
    return false;
  }

  String dstClusterName = dstdfs.getClusterName();
  for (Path src : srcs) {
    DistributedFileSystem srcdfs = DFSUtil.convertToDFS(src
        .getFileSystem(conf));
    if (srcdfs == null) {
      return false;
    } else if (dstClusterName != null) {
      // We assume those clusterName == null case was older
      // version of DFS. We always enable fastcopy for those
      // cases.
      String srcClusterName = srcdfs.getClusterName();
      if (srcClusterName != null && !srcClusterName.equals(dstClusterName)) {
        return false;
      }
    }
  }
  return true;
}
Example 2
Source File: DFSAdmin.java    From RDFS with Apache License 2.0 5 votes vote down vote up 
/** Constructor */
public DFSAdminCommand(FileSystem fs) {
  super(fs.getConf());
  DistributedFileSystem dfs = DFSUtil.convertToDFS(fs);      
  if (dfs == null) {
    throw new IllegalArgumentException("FileSystem " + fs.getUri() + 
        " is not a distributed file system");
  }
  this.dfs = dfs;
}
Example 3
Source File: FastCopy.java    From RDFS with Apache License 2.0 4 votes vote down vote up 
public static void runTool(String args[]) throws Exception {
  CommandLine cmd = parseCommandline(args);
  args = cmd.getArgs();
  int threadPoolSize = (cmd.hasOption('t')) ? Integer.parseInt(cmd
      .getOptionValue('t')) : THREAD_POOL_SIZE;
  
  List<CopyPath> srcs = new ArrayList<CopyPath>();
  String dst = parseFiles(srcs, args);
  
  Path dstPath = new Path(dst);
  DistributedFileSystem dstFileSys = DFSUtil.convertToDFS(dstPath
      .getFileSystem(defaultConf));

  DistributedFileSystem srcFileSys = DFSUtil.convertToDFS(srcs.get(0)
      .getSrcPath().getFileSystem(defaultConf));
  List<FastFileCopyRequest> requests = new ArrayList<FastFileCopyRequest>();
  FastCopy fcp = new FastCopy(new Configuration(), threadPoolSize);

  try {
    for (CopyPath copyPath : srcs) {
      Path srcPath = copyPath.getSrcPath();
      String src = srcPath.toString();
      try {
        // Perform some error checking and path manipulation.

        if (!srcFileSys.exists(srcPath)) {
          throw new IOException("File : " + src + " does not exists on "
              + srcFileSys);
        }

        String destination = copyPath.getDstPath().toString();
        LOG.debug("Copying : " + src + " to " + destination);
        requests.add(new FastFileCopyRequest(src, destination, srcFileSys, dstFileSys));
      } catch (Exception e) {
        LOG.warn("Fast Copy failed for file : " + src, e);
      }
    }
    fcp.copy(requests);
    LOG.debug("Finished copying");
  } finally {
    fcp.shutdown();
  }
}