FS Module

FS Module is a basis for building efficient, scalable remote file storage, supporting data redundancy, rebalancing and resharding. It provides an asynchronous abstraction on top of both local FS and remote FS with a simple FTP-like protocol with zero-overhead streaming.

All data in FS is organized in file systems. The key principle is effective data immutability, which enables simple design, low-overhead and aggressive caching. This approach allows to create high-throughput data storage and processing applications.

Similarly to the RPC module, it contains a set of strategies that can be simply combined. In this way FS module is a simple alternative for kernel-space distributed FS and allows to create combined, fault-tolerant cluster FS with repartitioning and rebalancing support.

You can add the FS module to your project by inserting a dependency in pom.xml:

<dependency>
    <groupId>io.datakernel</groupId>
    <artifactId>datakernel-fs</artifactId>
    <version>3.1.0</version>
</dependency>

This module includes FsClient interface which represents a client with upload, download, move, delete and list operations. There are several implementations of the interface:

  • LocalFsClient - a client that works with local file system and doesn’t involve working with networks.
  • RemoteFsClient - connects to a RemoteFsServer and communicates with it.
  • RemoteFsClusterClient - a client which operates on a map of other clients as a cluster.

Also, there is a RemoteFsServer which is an implementation of AbstractServer for Cloud-FS.

Examples

  1. Server Setup Example - configuring and launching RemoteFsServer.
  2. File Upload Example - uploading file to RemoteFsServer.
  3. File Download Example - downloading file from RemoteFsServer.

To run the examples in an IDE, you need to clone DataKernel:

$ git clone https://github.com/softindex/datakernel

And import it as a Maven project. Check out branch v3.1. Before running the example, build the project (Ctrl + F9 for IntelliJ IDEA).

First, open ServerSetupExample class which is located at datakernel -> examples -> cloud-> fs, and run its main() method.

Then, open FileUploadExample class which is located in the same directory, and run its main() method.

Finally, open FileDownloadExample class which is located in the same directory, and also run its main() method.

In the example we upload an “example.txt” file to the server and then download it back as a “download_example.txt”.

Let’s have a closer look at Server Setup Example. To make setup and launching as simple as possible, there is a special RemoteFsServerLauncher from Launchers module. It allows to simply setup an FS server:

public class ServerSetupExample extends RemoteFsServerLauncher {
	private Path storage;

	@Override
	protected void onInit(Injector injector) throws Exception {
		storage = Files.createTempDirectory("server_storage");
	}

	@Override
	protected Module getOverrideModule() {
		return new AbstractModule() {
			@Provides
			Config config() {
				return Config.create()
						.with("remotefs.path", storage.toString())
						.with("remotefs.listenAddresses", "6732");
			}
		};
	}

	@Override
	protected void run() throws Exception {
		awaitShutdown();
	}

	public static void main(String[] args) throws Exception {
		Launcher launcher = new ServerSetupExample();
		launcher.launch(args);
	}
}

File upload and download examples have alike implementations. Both of them extend Launcher and thus implement run() method which defines the main behaviour of the launcher. Also, both of the examples utilize CSP module - uploader uses ChannelFileReader while downloader uses ChannelFileWriter. They allow to asynchronously read/write data from/to files.