org.apache.flink.core.fs.FileSystemKind Java Examples

The following examples show how to use org.apache.flink.core.fs.FileSystemKind. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: HadoopFileSystem.java    From flink with Apache License 2.0 6 votes vote down vote up
/**
 * Gets the kind of the file system from its scheme.
 *
 * <p>Implementation note: Initially, especially within the Flink 1.3.x line
 * (in order to not break backwards compatibility), we must only label file systems
 * as 'inconsistent' or as 'not proper filesystems' if we are sure about it.
 * Otherwise, we cause regression for example in the performance and cleanup handling
 * of checkpoints.
 * For that reason, we initially mark some filesystems as 'eventually consistent' or
 * as 'object stores', and leave the others as 'consistent file systems'.
 */
static FileSystemKind getKindForScheme(String scheme) {
	scheme = scheme.toLowerCase(Locale.US);

	if (scheme.startsWith("s3") || scheme.startsWith("emr") || scheme.startsWith("oss") || scheme.startsWith("wasb")) {
		// the Amazon S3 storage or Aliyun OSS storage or Azure Blob Storage
		return FileSystemKind.OBJECT_STORE;
	}
	else if (scheme.startsWith("http") || scheme.startsWith("ftp")) {
		// file servers instead of file systems
		// they might actually be consistent, but we have no hard guarantees
		// currently to rely on that
		return FileSystemKind.OBJECT_STORE;
	}
	else {
		// the remainder should include hdfs, kosmos, ceph, ...
		// this also includes federated HDFS (viewfs).
		return FileSystemKind.FILE_SYSTEM;
	}
}
 
Example #2
Source File: HadoopFileSystem.java    From Flink-CEPplus with Apache License 2.0 6 votes vote down vote up
/**
 * Gets the kind of the file system from its scheme.
 *
 * <p>Implementation note: Initially, especially within the Flink 1.3.x line
 * (in order to not break backwards compatibility), we must only label file systems
 * as 'inconsistent' or as 'not proper filesystems' if we are sure about it.
 * Otherwise, we cause regression for example in the performance and cleanup handling
 * of checkpoints.
 * For that reason, we initially mark some filesystems as 'eventually consistent' or
 * as 'object stores', and leave the others as 'consistent file systems'.
 */
static FileSystemKind getKindForScheme(String scheme) {
	scheme = scheme.toLowerCase(Locale.US);

	if (scheme.startsWith("s3") || scheme.startsWith("emr") || scheme.startsWith("oss")) {
		// the Amazon S3 storage or Aliyun OSS storage
		return FileSystemKind.OBJECT_STORE;
	}
	else if (scheme.startsWith("http") || scheme.startsWith("ftp")) {
		// file servers instead of file systems
		// they might actually be consistent, but we have no hard guarantees
		// currently to rely on that
		return FileSystemKind.OBJECT_STORE;
	}
	else {
		// the remainder should include hdfs, kosmos, ceph, ...
		// this also includes federated HDFS (viewfs).
		return FileSystemKind.FILE_SYSTEM;
	}
}
 
Example #3
Source File: HadoopFileSystem.java    From flink with Apache License 2.0 6 votes vote down vote up
/**
 * Gets the kind of the file system from its scheme.
 *
 * <p>Implementation note: Initially, especially within the Flink 1.3.x line
 * (in order to not break backwards compatibility), we must only label file systems
 * as 'inconsistent' or as 'not proper filesystems' if we are sure about it.
 * Otherwise, we cause regression for example in the performance and cleanup handling
 * of checkpoints.
 * For that reason, we initially mark some filesystems as 'eventually consistent' or
 * as 'object stores', and leave the others as 'consistent file systems'.
 */
static FileSystemKind getKindForScheme(String scheme) {
	scheme = scheme.toLowerCase(Locale.US);

	if (scheme.startsWith("s3") || scheme.startsWith("emr") || scheme.startsWith("oss") || scheme.startsWith("wasb")) {
		// the Amazon S3 storage or Aliyun OSS storage or Azure Blob Storage
		return FileSystemKind.OBJECT_STORE;
	}
	else if (scheme.startsWith("http") || scheme.startsWith("ftp")) {
		// file servers instead of file systems
		// they might actually be consistent, but we have no hard guarantees
		// currently to rely on that
		return FileSystemKind.OBJECT_STORE;
	}
	else {
		// the remainder should include hdfs, kosmos, ceph, ...
		// this also includes federated HDFS (viewfs).
		return FileSystemKind.FILE_SYSTEM;
	}
}
 
Example #4
Source File: HdfsKindTest.java    From flink with Apache License 2.0 5 votes vote down vote up
@Test
public void testS3fileSystemSchemes() {
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("s3"));
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("s3n"));
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("s3a"));
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("EMRFS"));
}
 
Example #5
Source File: MapRFsFactoryTest.java    From flink with Apache License 2.0 5 votes vote down vote up
@Test
public void testMapRFsKind() throws Exception {
	final Path path = new Path("maprfs:///my/path");

	final FileSystem fs = path.getFileSystem();

	assertEquals(FileSystemKind.FILE_SYSTEM, fs.getKind());
}
 
Example #6
Source File: HadoopFileSystem.java    From flink with Apache License 2.0 5 votes vote down vote up
@Override
public FileSystemKind getKind() {
	if (fsKind == null) {
		fsKind = getKindForScheme(this.fs.getUri().getScheme());
	}
	return fsKind;
}
 
Example #7
Source File: MapRFsFactoryTest.java    From flink with Apache License 2.0 5 votes vote down vote up
@Test
public void testMapRFsKind() throws Exception {
	final Path path = new Path("maprfs:///my/path");

	final FileSystem fs = path.getFileSystem();

	assertEquals(FileSystemKind.FILE_SYSTEM, fs.getKind());
}
 
Example #8
Source File: HadoopFileSystem.java    From flink with Apache License 2.0 5 votes vote down vote up
@Override
public FileSystemKind getKind() {
	if (fsKind == null) {
		fsKind = getKindForScheme(this.fs.getUri().getScheme());
	}
	return fsKind;
}
 
Example #9
Source File: HdfsKindTest.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up
@Test
public void testS3fileSystemSchemes() {
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("s3"));
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("s3n"));
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("s3a"));
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("EMRFS"));
}
 
Example #10
Source File: HdfsKindTest.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up
@Test
public void testS3aKind() throws IOException {
	try {
		Class.forName("org.apache.hadoop.fs.s3a.S3AFileSystem");
	} catch (ClassNotFoundException ignored) {
		// not in the classpath, cannot run this test
		log.info("Skipping test 'testS3aKind()' because the S3AFileSystem is not in the class path");
		return;
	}

	final FileSystem s3a = new Path("s3a://myId:mySecret@bucket/some/bucket/some/object").getFileSystem();
	assertEquals(FileSystemKind.OBJECT_STORE, s3a.getKind());
}
 
Example #11
Source File: HdfsKindTest.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up
@Test
public void testS3nKind() throws IOException {
	try {
		Class.forName("org.apache.hadoop.fs.s3native.NativeS3FileSystem");
	} catch (ClassNotFoundException ignored) {
		// not in the classpath, cannot run this test
		log.info("Skipping test 'testS3nKind()' because the Native S3 file system is not in the class path");
		return;
	}

	final FileSystem s3n = new Path("s3n://myId:mySecret@bucket/some/bucket/some/object").getFileSystem();
	assertEquals(FileSystemKind.OBJECT_STORE, s3n.getKind());
}
 
Example #12
Source File: HdfsKindTest.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up
@Test
public void testS3Kind() throws IOException {
	try {
		Class.forName("org.apache.hadoop.fs.s3.S3FileSystem");
	} catch (ClassNotFoundException ignored) {
		// not in the classpath, cannot run this test
		log.info("Skipping test 'testS3Kind()' because the S3 file system is not in the class path");
		return;
	}

	final FileSystem s3 = new Path("s3://myId:mySecret@bucket/some/bucket/some/object").getFileSystem();
	assertEquals(FileSystemKind.OBJECT_STORE, s3.getKind());
}
 
Example #13
Source File: HadoopFileSystem.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up
@Override
public FileSystemKind getKind() {
	if (fsKind == null) {
		fsKind = getKindForScheme(this.fs.getUri().getScheme());
	}
	return fsKind;
}
 
Example #14
Source File: HdfsKindTest.java    From flink with Apache License 2.0 5 votes vote down vote up
@Test
public void testS3fileSystemSchemes() {
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("s3"));
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("s3n"));
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("s3a"));
	assertEquals(FileSystemKind.OBJECT_STORE, HadoopFileSystem.getKindForScheme("EMRFS"));
}
 
Example #15
Source File: HdfsKindTest.java    From flink with Apache License 2.0 5 votes vote down vote up
@Test
public void testS3Kind() throws IOException {
	try {
		Class.forName("org.apache.hadoop.fs.s3.S3FileSystem");
	} catch (ClassNotFoundException ignored) {
		// not in the classpath, cannot run this test
		log.info("Skipping test 'testS3Kind()' because the S3 file system is not in the class path");
		return;
	}

	final FileSystem s3 = new Path("s3://myId:mySecret@bucket/some/bucket/some/object").getFileSystem();
	assertEquals(FileSystemKind.OBJECT_STORE, s3.getKind());
}
 
Example #16
Source File: HdfsKindTest.java    From flink with Apache License 2.0 5 votes vote down vote up
@Test
public void testS3nKind() throws IOException {
	try {
		Class.forName("org.apache.hadoop.fs.s3native.NativeS3FileSystem");
	} catch (ClassNotFoundException ignored) {
		// not in the classpath, cannot run this test
		log.info("Skipping test 'testS3nKind()' because the Native S3 file system is not in the class path");
		return;
	}

	final FileSystem s3n = new Path("s3n://myId:mySecret@bucket/some/bucket/some/object").getFileSystem();
	assertEquals(FileSystemKind.OBJECT_STORE, s3n.getKind());
}
 
Example #17
Source File: HdfsKindTest.java    From flink with Apache License 2.0 5 votes vote down vote up
@Test
public void testS3aKind() throws IOException {
	try {
		Class.forName("org.apache.hadoop.fs.s3a.S3AFileSystem");
	} catch (ClassNotFoundException ignored) {
		// not in the classpath, cannot run this test
		log.info("Skipping test 'testS3aKind()' because the S3AFileSystem is not in the class path");
		return;
	}

	final FileSystem s3a = new Path("s3a://myId:mySecret@bucket/some/bucket/some/object").getFileSystem();
	assertEquals(FileSystemKind.OBJECT_STORE, s3a.getKind());
}
 
Example #18
Source File: DummyFSFileSystem.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public FileSystemKind getKind() {
	return FileSystemKind.OBJECT_STORE;
}
 
Example #19
Source File: HadoopOSSFileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
 
Example #20
Source File: HdfsKindTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Test
public void testViewFs() {
	assertEquals(FileSystemKind.FILE_SYSTEM, HadoopFileSystem.getKindForScheme("viewfs"));
}
 
Example #21
Source File: LocalFileSystem.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public FileSystemKind getKind() {
	return FileSystemKind.FILE_SYSTEM;
}
 
Example #22
Source File: PrestoS3FileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
 
Example #23
Source File: LocalFileSystemBehaviorTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.FILE_SYSTEM;
}
 
Example #24
Source File: LocalFileSystemTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Test
public void testKind() {
	final FileSystem fs = FileSystem.getLocalFileSystem();
	assertEquals(FileSystemKind.FILE_SYSTEM, fs.getKind());
}
 
Example #25
Source File: FlinkS3FileSystem.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public FileSystemKind getKind() {
	return FileSystemKind.OBJECT_STORE;
}
 
Example #26
Source File: AzureFileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
 
Example #27
Source File: HadoopS3FileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
 
Example #28
Source File: HadoopLocalFileSystemBehaviorTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.FILE_SYSTEM;
}
 
Example #29
Source File: HadoopLocalFileSystemBehaviorTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.FILE_SYSTEM;
}
 
Example #30
Source File: HdfsBehaviorTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public FileSystemKind getFileSystemKind() {
	// this tests tests only HDFS, so it should be a file system
	return FileSystemKind.FILE_SYSTEM;
}