FS Module is a basis for building efficient, scalable remote file storage, supporting data redundancy, rebalancing and resharding. It provides an asynchronous abstraction on top of both local FS and remote FS with a simple FTP-like protocol with zero-overhead streaming.
All data in FS is organized in file systems. The key principle is effective data immutability, which enables simple design, low-overhead and aggressive caching. This approach allows to create high-throughput data storage and processing applications.
Similarly to the RPC module, it contains a set of strategies that can be simply combined. In this way FS module is a simple alternative for kernel-space distributed FS and allows to create combined, fault-tolerant cluster FS with repartitioning and rebalancing support.
You can add the FS module to your project by inserting a dependency in
This module includes FsClient interface which represents a client with upload, download, move, delete and list operations. There are several implementations of the interface:
Also, there is a RemoteFsServer which is an implementation of AbstractServer for Cloud-FS.
To run the examples in an IDE, you need to clone DataKernel:
And import it as a Maven project. Check out branch v3.1. Before running the example, build the project (Ctrl + F9 for IntelliJ IDEA).
ServerSetupExample class which is located at datakernel -> examples -> cloud-> fs, and run its main()
FileUploadExample class which is located in the same directory, and run its main() method.
FileDownloadExample class which is located in the same directory, and also run its main() method.
In the example we upload an “example.txt” file to the server and then download it back as a “download_example.txt”.
Let’s have a closer look at Server Setup Example. To make setup and launching as simple as possible, there is a
RemoteFsServerLauncher from Launchers module. It allows to simply setup an FS server:
File upload and download examples have alike implementations. Both of them extend
Launcher and thus
implement run() method which defines the main behaviour of the launcher.
Also, both of the examples utilize CSP module - uploader uses
ChannelFileReader while downloader uses
They allow to asynchronously read/write data from/to files.