Java Code Examples for org.apache.hadoop.mapreduce.filecache.DistributedCache#createSymlink()

The following examples show how to use org.apache.hadoop.mapreduce.filecache.DistributedCache#createSymlink() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: MRJobLauncher.java    From incubator-gobblin with Apache License 2.0 6 votes vote down vote up 
/**
 * Add local non-jar files the job depends on to DistributedCache.
 */
@SuppressWarnings("deprecation")
private void addLocalFiles(Path jobFileDir, String jobFileList, Configuration conf) throws IOException {
  DistributedCache.createSymlink(conf);
  for (String jobFile : SPLITTER.split(jobFileList)) {
    Path srcJobFile = new Path(jobFile);
    // DistributedCache requires absolute path, so we need to use makeQualified.
    Path destJobFile = new Path(this.fs.makeQualified(jobFileDir), srcJobFile.getName());
    // Copy the file from local file system to HDFS
    this.fs.copyFromLocalFile(srcJobFile, destJobFile);
    // Create a URI that is in the form path#symlink
    URI destFileUri = URI.create(destJobFile.toUri().getPath() + "#" + destJobFile.getName());
    LOG.info(String.format("Adding %s to DistributedCache", destFileUri));
    // Finally add the file to DistributedCache with a symlink named after the file name
    DistributedCache.addCacheFile(destFileUri, conf);
  }
}
Example 2
Source File: MRJobLauncher.java    From incubator-gobblin with Apache License 2.0 5 votes vote down vote up 
/**
 * Add non-jar files already on HDFS that the job depends on to DistributedCache.
 */
@SuppressWarnings("deprecation")
private void addHDFSFiles(String jobFileList, Configuration conf) {
  DistributedCache.createSymlink(conf);
  jobFileList = PasswordManager.getInstance(this.jobProps).readPassword(jobFileList);
  for (String jobFile : SPLITTER.split(jobFileList)) {
    Path srcJobFile = new Path(jobFile);
    // Create a URI that is in the form path#symlink
    URI srcFileUri = URI.create(srcJobFile.toUri().getPath() + "#" + srcJobFile.getName());
    LOG.info(String.format("Adding %s to DistributedCache", srcFileUri));
    // Finally add the file to DistributedCache with a symlink named after the file name
    DistributedCache.addCacheFile(srcFileUri, conf);
  }
}
Example 3
Source File: Job.java    From hadoop with Apache License 2.0 4 votes vote down vote up 
/**
 * Originally intended to enable symlinks, but currently symlinks cannot be
 * disabled.
 */
@Deprecated
public void createSymlink() {
  ensureState(JobState.DEFINE);
  DistributedCache.createSymlink(conf);
}
Example 4
Source File: Job.java    From big-c with Apache License 2.0 4 votes vote down vote up 
/**
 * Originally intended to enable symlinks, but currently symlinks cannot be
 * disabled.
 */
@Deprecated
public void createSymlink() {
  ensureState(JobState.DEFINE);
  DistributedCache.createSymlink(conf);
}