<< Previous | Home | Next >> | Conferences & TechTalks Calendar | Legal Notice | Google+ Profile | Twitter Page | My Profile Page

CastMapR

The Hazelcast 3 MapReduce Framework

 A few days ago while porting our current system to Hazelcast 3 Snapshots I finally decided to start a MapReduce implementation for Hazelcast which I was missing for a long time.

Read more...

Categories : Java, Projects, Other

Property Accessors - 2

The first discussion results

 Here's a short update:

At the moment there's not much to tell but I'd found at lot of interest in properties support in Java and I'm glad to see people like the general idea.

Read more...

Property Accessors

A short introduction

As I stated yesterday I started the discussion about adding Property Accessors to Java. Today I want to show some more about my thoughts.

There are at least three questions?

  • What are Property Accessors?
  • Why Java needs them?
  • What could they look like?

 

Read more...

Categories : Projects, Other

Properties Proposal

Just started a discussion about the first ideas on how to implement Properties in Java. Many of you possibly know Properties from other languages like C#.

I would be happy if anyone of you interested in that feature would join the discussion and give opinions and improvements on how to make it great.

www.java.net/forum/topic/jcp/general-jsr-discussion/properties-proposal

 

Related Posts
The Hazelcast Incubator
snowcast - Hazelcast Client and the snowcast logo
snowcast - Migration and Failover


Categories : Projects, Other

Jenkins and the Git Branch Selection

How to make Jenkins aware of Git branches

 

This time I want to talk about something "totally" different. Something just from work.

At work we recently changed the revisionsystem to Git to meet our requirements in feature driven development using a lot of feature branches.

To fulfill our high standards in quality and stability we moved over to a silently different workflow where QA tests every feature branch before it'll merged in the final release. The day where the deployment meeting takes place it'll decided which branches are stable and will be merged into the actual release branch. To make it more convenient for our QA to build, deploy and test branches we setup a bunch of QA servers (one server for each QA tester) and gave them the chance to build any branch and deploy it to a choosen QA server.

Since Jenkins currently has no real Git branch support (beside giving the branchname by yourself) we had to figure out how to tell Jenkins which branches are available. To do so we made a small shellscript which collects all of the feature branches into some properties file that later will be read by Jenkins.

 

Read more...

Categories : Projects, Other

Lightning is moving onwards to the ASF

Lightning, the highspeed and low latency serializer for the Java environment will move it's way onwards to the Apache Software Foundation.

Born from the discussion to add a serializer connector for Lightning to Apache DirectMemory, I was asked to contribute Lightning as the later on default serializer for DirectMemory.

Being honored I agreed to move on from Github onto the care of the ASF and so becoming a sub-project of Apache DirectMemory.

So what does that means:
That means that the Maven coordinates and the package names are changing but since there was no final released version and not a stable public API that shouldn't be a real problem for most of the people.

What will not change is the separate availability.

Lightning will be usable without the need to use DirectMemory but will miss some additional features without it like highspeed serialization to off-heap memory.

At the moment Lightning is waiting in incubation for being taken over by DirectMemory Top Level Project. In that phase there will be no new features but changes to comply the Apache sourceocde needs like being compliant with the Apache License 2 (which was mostly given with a small exception) and to fulfill requirements in the Maven project descriptors.

I hope that this state will be finished by time and development can go on.

 

 

Related Posts
The Hazelcast Incubator
snowcast - Hazelcast Client and the snowcast logo
snowcast - Migration and Failover


Categories : Java, Projects, Lightning

Lightning

The Lightning Fast Java Serializer

It's been a pretty long time since the last blog entry. A lot of things happened holding me back from posting updates.

Hopefully this time has gone and I started a new project.

Currently I'm working on cleaning up the API for my serializer to define serializable classes and members, prior to release the first milestone.

But before asking my actual question I think there are some things needs to be clarified about Lightning.

Lightning is a highspeed, low latency serializer for value- or transportobjects (or whatever you want to call them). So it's not to be used for all cases as the standard serialization of Java but since most transferable objects can be implemented as plain valueobjects you can use Lightning for most cases where you want to serialize / transfer data.

A second important point is another "workaround" of a problem many people find when using standard serialization. Problems with (de-)serialization often happen hours, maybe days after startup of servers: problems when reading data using incompatible classversions. Lightning uses a ClassDefinitions to hold informations about serializable properties and distributes them for example inside a cluster. So the masternode transfers it's own ClassDefinitionContainer over to newly connecting nodes and the new node just tries to compare it with his container. If there's a non matching class definition the node can be disconnected before corrupting the clusterdata.

This means Lightning uses a Fast-Failing approach for being sure that class definitions will be consistent.

The most visible difference between Lightning and the standard serialization would be that you have to define which properties will be serialized. By default non of the class members will be serialized compared to standard serialization where all "non-transient" members are serialized by default.

I started Lightning as a Proof-Of-Concept implementation due to my old employer had the need for a really fast, simple to configure and low bytesize (for the datastream) serialization for distributing valueobject inside a JGroups featured cluster.

The projectname was choosen as an incentive to tell myself "if the name does not match the behavior, forget about it". I was impressed that the name seems to match it in any way as shown by small benchmarks (yeah I know, microbenchmarking ;-)):

Lightning Serializer build time: 199 ms
Lightning Serialization Avg: 1023,97 ns, runs: 800000, size: 40 bytes
Lightning Deserialization Avg: 1097,72 ns, runs: 800000, size: 45 bytes

Java Serialization Avg: 4069,38 ns, runs: 800000, size: 375 bytes
Java Deserialization Avg: 20290,70 ns, runs: 800000, size: 375 bytes

In this example you can see the difference in bytesize (the small differences belongs to implementation of the microbenchmark to prevent HotSpot optimizations) and in terms of speed.

What is "build time"? How does Lightning works internally? And last but not least why is Lightning that fast?

That's possibly the three questions that comes most people into mind by reading the benchmark results.

Let's start by a question that can even be answered by other serializers (like Kryo) "why is the bytestream that small?". This question can shortly be answered in two sentences. As mentioned above Lightning uses a ClassDefinitionContainer to know about the internals of the bytestream, so that nearly nothing beside real values needs to be in the stream. In addition to that, every class in the ClassDefinitionContainer has a unique ClassId so canonical classnames are only distributed when transfering the ClassDefinitionContainers.

The first of the three questions I want to move to the end of the description since it'll answer itself :-)

The other two questions "How does Lightning works internally?" and "Why is Lightning that fast?" can be answered together. Lightning uses a lot of tricks to speed up the serialization process which can be categorized by 4 categories.

1. Lightning knows different strategies to serialize objecttrees.
This means Lightning can be instructed to either know about multiple references to the same objects in an objecttree (and only serializes them once) or to ignore such multi-references objects resulting in possibly serialize them multiple times.

2. Lightning uses Bytecode-Generation.
After the definitions of serializable class members Lightning switches to a buildphase creating an bytecode implementation of every marsheller. This marshallers does not use any reflection and can be HotSpt-optimized later on.

3. Lightning uses direct memory acces using sun.misc.Unsafe.
Normally, on supported runtime environments, Lightning uses sun.misc.Unsafe to support direct memory access for properties. This means reading / writing directly from / to the memory areas of the objects in the heap. By using this technique a lot of memory- and boundchecks can be prevented resulting in a much higher speed (this access is even faster that accessing direct ByteBuffers - ByteBuffer.allocateDirect(...)). Even when using a class called Unsafe sounds pretty messy a lot of the internal implementation on SUN / Oracle JVMs using this class for high performance accesses and to prevent reflective access.

4. Lightning uses PropertyAccessor classes.
This is an extension to the systems described in point 2 and 3. Inside the buildphase all properties will be wrapped into such a PropertyAccessor to optimize access and prevent later request by the Java Security-System. In addition to this different kinds of access methods like reflection (on non Unsafe supporting environments) or bytecode access can be hidden and normalized.

I think this should be enough as a first introduction into Lightning. If there are any questions or remarks feel free to tell me.

For people who are read to that point and being interested in Lightning, the project is released using Apache License 2 on Github.
It would be nice to find more attendees being interested in implementation or extending Lightning.
https://github.com/noctarius/Lightning

Now my initial question:
As I mentioned above I'm working on cleaning up the API to define which class members should be serialized. To define them Lightning uses SerializerDefinitions, can be compared to Modules in Guice, and a Fluent-API to define properties. It would be fine if you could have a look over the new API to see if all fluent combinations are easy to understand and intuitionally. If there are any remarks or improvement proposals on the Fluent-API please let me know.

 

public class ExampleSerializerDefinition 
		extends AbstractSerializerDefinition {

 	@Override
	protected void configure() {
		// Define serializable class using custom
		// implementation of Marshaller
		serialize(Bar.class).using(new BarMarshaller());
		serialize(Bar.class).using(BarMarshaller.class);

// Define serializable class using annotated members
// or methods (by usage of Lightning's
// @com.github.lightning.metadata.Attribute annotation)
serialize(Foo.class).attributes();
serialize(Foo.class).attributes().exclude("value");
serialize(Foo.class).attributes().exclude("value1").exclude("value2");
serialize(Foo.class).attributes().excludes("value1", "value2");

// Define serializable class using annotated members / methods
// (by usage of custom annotation)
serialize(Foo.class).attributes(Attribute.class);
serialize(Foo.class).attributes(Attribute.class).exclude("value");
serialize(Foo.class).attributes(Attribute.class)
.exclude("value1").exclude("value2");
serialize(Foo.class).attributes(Attribute.class).excludes("value1", "value2");

// Define serializable class using custom definition of properties
serialize(Foo.class).properties(
property("value"), 
property("value").using(SomeSpecialIntegerMarshaller.class), 
property("value").using(new SomeSpecialIntegerMarshaller())
);

// Define serializable class using a different implementation
// of PropertyFinderStrategy
serialize(Foo.class).using(new FooPropertyFinderStrategy());
serialize(Foo.class).using(FooPropertyFinderStrategy.class);

// Install child definition
install(new SomeChildSerializerDefinition());
}
}

Related Posts
The Hazelcast Incubator
snowcast - Hazelcast Client and the snowcast logo
snowcast - Migration and Failover


Categories : Java, Projects, Lightning

When Thracia came ...

... bug reporting in Java gone easy

Today I will announce to move an old project which was a bit slow in development the last time. To make it matching the actual sourceprojects.org name schema the name changed from original jBugReporter to Thracia.

So what is Thracia?

Thracia is a library makes it easy to use clean bug reporting in production state Java systems. When an exception is thrown Thracia takes a snapshot of the actual state of the application by saving threads and their states, the complete stacktrace, the exceptions message and more information and writes them to a simple XML or other data structures. Every exception has it's own, recognizable hash so same exceptions does not get reported more than one time. On serverside there is a server that handles the reports automatically and a connector instance that makes it possible to the server to find and report issues to different bugtrackers.

So what can an example file look like:
2010070109401044722281283672518.xml (2.65KB / XML)

 

Related Posts
There are no related posts for this blog entry


Categories : Thracia, Java, Projects

Lycia Benchmark

on Windows and Linux

Hey readers,

today I want to show some interesting results from benchmarking Lycia. The most interesting isn't want it shows by the first look at the chart, I mean we all should know Java on Linux performs out Java on Windows. What was so impressive to me is: for some reason (maybe the internal design of Lycia) using reflection at generation of runtime parsers isn't much slower than to create them by using bytecode production.

Both systems run on latest Java 1.6.0_20 so both are using the latest HotSpot compiler. The next days I will retry the benchmark on some more JVMs and especially on some 1.5 versions to see if it is the new HotSpot or if it is the internal design of Lycia which makes that minimal difference between both generation strategies.

To explain what I was expecting:
For me it was totally clear that bytecode generation would be  more expensive in at creation but must perform out the reflection at runtime what is obviously not the fact.

For bytecode generation Lycia creates a complete new class at runtime which is compiled and loaded like any other precompiled class. This means, after some time, the HotSpot compiler make the bytecode into native cpu instructions.
For reflection, by contrast, a proxy class is instanced and the parser method is called by introspection of itself every time.

So there are 4 possibilities why the result is so "wrong" for me:
* Im wrong with my expectations
* Lycia's internal design thwarts bytecode generation from performing
* The HotSpot compiler in newer Java versions can optimize reflection calls
* The benchmark is just wrong (for example cause of using a constant expression for being parsed)

I attach both, the results as PDF and the benchmark class for being reviewed. It would be fine if someone could give informations and optinions about the benchmarking. Microbenchmarks are always hard to do.

BenchmarkReflectiveBytecodeTestCase.java (5.62KB / JAVA-Class)
benchmark.pdf (102.04KB / PDF)

Related Posts
There are no related posts for this blog entry


Categories : Lycia, Java, Projects

Lycia went threadsafe

Today Lycia went on to be fully threadsafe by implementing a threadsafe version of ParserContext.

Now you're able to use one instance of Lycia in as many threads as you like to, which can be very good for runtime behavior of webapplications to not initialize Lycia for every request.

Related Posts
There are no related posts for this blog entry


Categories : Lycia, Java, Projects