Serializer Module

DataKernel features the fastest Serializer in the world! A bytecode generator of extremely fast and space-efficient serializers, which are used for transferring data over wire or adding it into a file system or database. Widely utilizes Codegen module.

Core Serializer features:

  • Implemented using runtime bytecode generation to be compatible with dynamically created classes (like intermediate POJOs created with the Codegen module).
  • Stable binary format with backward binary compatibility for efficient long-term code support.
  • Support of the scheme evolution: changeable versions, added and removed fields, etc.
  • Full support of Java subclasses, collections (including Maps) and also specialized collections like HPPC.
  • Works directly with Java classes utilizing annotations. Thus, it doesn’t create additional layers of intermediate DTO classes, which significantly speeds up the code and makes it easy-to-use.
  • Can be easily extended - you can write your own plugins for specific classes.
  • Includes special hints for even more efficient code (String formats, nullable, varlen, etc).

A common usage for Serializer is to pass some serialized class instances through the network to remote machines for further processing. This approach is used in RPC, Dataflow and LSM Tree OLAP Cube modules.

Benchmarks

We’ve measured Serializer performance using this benchmark tool and got the following results (compared to the second-fastest Java serializer Colfer):

You can add Serializer module to your project by inserting dependency in pom.xml:

<dependency>
    <groupId>io.datakernel</groupId>
    <artifactId>datakernel-serializer</artifactId>
    <version>3.1.0</version>
</dependency>

Examples

  1. Simple Object Serialization Example - serialization and deserialization of a simple object.
  2. Complex Object Serialization Example - serialization and deserialization of a more complex object which contains nullable fields, map, list and a two-dimensional array.
  3. Fixed Size Fields Serialization Example - example of serialization and deserialization of an object with fixed size fields.
  4. Generics And Interfaces Serialization Example - example of using generics and interfaces with serializers and deserializers.
Note: To run the examples, you need to clone DataKernel from GitHub:
$ git clone https://github.com/softindex/datakernel
And import it as a Maven project. Check out branch v3.1. Before running the examples, build the project.
These examples are located at datakernel -> examples -> core -> serializer

All of the examples consistently serialize and deserialize objects of different complexities.

Simple Object Serialization Example

If you run the Simple Object Serialization Example, you’ll get the following output:

10 10
abc abc
20 20
30 30
40 40
123 123

The first column represents values of testData1, the second - of testData2. testData2 was created with the help of serialization and deserialization of the testData1.

In order to create classes whose instances can be serialized/deserialized, special annotations should be used:

  • @Serialize annotation with order number on property getter. Ordering provides better compatibility in case classes are changed.
  • @Deserialize annotation with property name (which should be the same name as the one in getter) in constructor.
  • @SerializeNullable on properties that can have null values.

For example,

public static class TestDataSimple {
	public TestDataSimple(@Deserialize("finalInt") int finalInt,
			@Deserialize("finalString") String finalString) {
		this.finalInt = finalInt;
		this.finalString = finalString;
	}

	@Serialize(order = 0)
	public final int finalInt;

	@Serialize(order = 1)
	public final String finalString;

	private int i;
	private Integer iBoxed;

	private int getterInt;
	private String getterString;

	@Serialize(order = 2)
	public int getI() {
		return i;
	}


	...

Next, we need to create our BinarySerializer with the help of BinaryBuilder:

BinarySerializer<T> serializer = SerializerBuilder.create(getContextClassLoader())
		.build(typeToken);

Generated bytecode of the BinarySerializer will be saved in your working directory.

You can explore full example sources here.

Complex Object Serialization Example

An example of serialization and deserialization of a more complex object that contains nullable fields, map, list and a two-dimensional array. If you run the example, you’ll receive the following output:

null null
abc abc
[a, null, b] [a, null, b]
2 2
[a, null] [a, null]
null null
{1=abc, 2=null, null=xyz} {null=xyz, 1=abc, 2=null}

You can explore full example sources here.

Generics and Interfaces Serialization Example

Represents how to use generics and interfaces with serializers and deserializers. If you run the example, you’ll receive the following output:

2 2 
10 a, 10 a
20 b, 20 b

You can explore full example sources here.

Fixed Size Fields Serialization Example

If you run the example, you’ll get the following output:

[abc, null, 123, superfluous] [abc, null, 123]
[1, 2, 3, 4] [1, 2, 3, 4]

As you can see in the first line, dataObject2 differs from dataObject1. This is because @SerializeFixedSize annotation was set at value 3 for the String array. Thus, “superfluous” was removed from the array while serialization took place:

public static class TestDataFixedSize {
	@Serialize(order = 0)
	@SerializeFixedSize(3)
	@SerializeNullable(path = {0})
	public String[] strings;

	@Serialize(order = 1)
	@SerializeFixedSize(4)
	public byte[] bytes;
}

You can explore full example sources here.