Java Code Examples for org.apache.flink.core.fs.FileSystemKind#OBJECT_STORE

The following examples show how to use org.apache.flink.core.fs.FileSystemKind#OBJECT_STORE . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: HadoopFileSystem.java    From Flink-CEPplus with Apache License 2.0 6 votes vote down vote up 
/**
 * Gets the kind of the file system from its scheme.
 *
 * <p>Implementation note: Initially, especially within the Flink 1.3.x line
 * (in order to not break backwards compatibility), we must only label file systems
 * as 'inconsistent' or as 'not proper filesystems' if we are sure about it.
 * Otherwise, we cause regression for example in the performance and cleanup handling
 * of checkpoints.
 * For that reason, we initially mark some filesystems as 'eventually consistent' or
 * as 'object stores', and leave the others as 'consistent file systems'.
 */
static FileSystemKind getKindForScheme(String scheme) {
	scheme = scheme.toLowerCase(Locale.US);

	if (scheme.startsWith("s3") || scheme.startsWith("emr") || scheme.startsWith("oss")) {
		// the Amazon S3 storage or Aliyun OSS storage
		return FileSystemKind.OBJECT_STORE;
	}
	else if (scheme.startsWith("http") || scheme.startsWith("ftp")) {
		// file servers instead of file systems
		// they might actually be consistent, but we have no hard guarantees
		// currently to rely on that
		return FileSystemKind.OBJECT_STORE;
	}
	else {
		// the remainder should include hdfs, kosmos, ceph, ...
		// this also includes federated HDFS (viewfs).
		return FileSystemKind.FILE_SYSTEM;
	}
}
Example 2
Source File: HadoopFileSystem.java    From flink with Apache License 2.0 6 votes vote down vote up 
/**
 * Gets the kind of the file system from its scheme.
 *
 * <p>Implementation note: Initially, especially within the Flink 1.3.x line
 * (in order to not break backwards compatibility), we must only label file systems
 * as 'inconsistent' or as 'not proper filesystems' if we are sure about it.
 * Otherwise, we cause regression for example in the performance and cleanup handling
 * of checkpoints.
 * For that reason, we initially mark some filesystems as 'eventually consistent' or
 * as 'object stores', and leave the others as 'consistent file systems'.
 */
static FileSystemKind getKindForScheme(String scheme) {
	scheme = scheme.toLowerCase(Locale.US);

	if (scheme.startsWith("s3") || scheme.startsWith("emr") || scheme.startsWith("oss") || scheme.startsWith("wasb")) {
		// the Amazon S3 storage or Aliyun OSS storage or Azure Blob Storage
		return FileSystemKind.OBJECT_STORE;
	}
	else if (scheme.startsWith("http") || scheme.startsWith("ftp")) {
		// file servers instead of file systems
		// they might actually be consistent, but we have no hard guarantees
		// currently to rely on that
		return FileSystemKind.OBJECT_STORE;
	}
	else {
		// the remainder should include hdfs, kosmos, ceph, ...
		// this also includes federated HDFS (viewfs).
		return FileSystemKind.FILE_SYSTEM;
	}
}
Example 3
Source File: HadoopFileSystem.java    From flink with Apache License 2.0 6 votes vote down vote up 
/**
 * Gets the kind of the file system from its scheme.
 *
 * <p>Implementation note: Initially, especially within the Flink 1.3.x line
 * (in order to not break backwards compatibility), we must only label file systems
 * as 'inconsistent' or as 'not proper filesystems' if we are sure about it.
 * Otherwise, we cause regression for example in the performance and cleanup handling
 * of checkpoints.
 * For that reason, we initially mark some filesystems as 'eventually consistent' or
 * as 'object stores', and leave the others as 'consistent file systems'.
 */
static FileSystemKind getKindForScheme(String scheme) {
	scheme = scheme.toLowerCase(Locale.US);

	if (scheme.startsWith("s3") || scheme.startsWith("emr") || scheme.startsWith("oss") || scheme.startsWith("wasb")) {
		// the Amazon S3 storage or Aliyun OSS storage or Azure Blob Storage
		return FileSystemKind.OBJECT_STORE;
	}
	else if (scheme.startsWith("http") || scheme.startsWith("ftp")) {
		// file servers instead of file systems
		// they might actually be consistent, but we have no hard guarantees
		// currently to rely on that
		return FileSystemKind.OBJECT_STORE;
	}
	else {
		// the remainder should include hdfs, kosmos, ceph, ...
		// this also includes federated HDFS (viewfs).
		return FileSystemKind.FILE_SYSTEM;
	}
}
Example 4
Source File: HadoopOSSFileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 5
Source File: AnotherDummyFSFileSystem.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 6
Source File: DummyFSFileSystem.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 7
Source File: HadoopOSSFileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 8
Source File: PrestoS3FileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 9
Source File: HadoopS3FileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 10
Source File: AzureFileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 11
Source File: FlinkS3FileSystem.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 12
Source File: DummyFSFileSystem.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 13
Source File: FlinkS3FileSystem.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 14
Source File: PrestoS3FileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 15
Source File: HadoopS3FileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 16
Source File: AzureFileSystemBehaviorITCase.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 17
Source File: FlinkS3FileSystem.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 18
Source File: HadoopOSSFileSystemBehaviorITCase.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 19
Source File: PrestoS3FileSystemBehaviorITCase.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}
Example 20
Source File: HadoopS3FileSystemBehaviorITCase.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
@Override
public FileSystemKind getFileSystemKind() {
	return FileSystemKind.OBJECT_STORE;
}