This blog shows timeout impelents under multi-thread Java application. Discuss its advantage and disadvantage, also tell difficulties we meet in practice.

Why shared timeout

For a complex micro service system, message is a basic way used for communication between all modules. So timeout is come into used, for every API or message, to avoid unfinished tasks.

Let’s think if you what to do something without timeout, it may never finished with response. So timeout does the matter to help user and system get rid of those situations.

But when we talk about shared timeout, what is it and what it does.

For example, one API may sending several message to execute on different services and finally finished. During the process, if a timeout happen, API should return as timeouted as we expected. So the design of timeout mechanism should share a same period of timeout.

Assume API timeout is T and three messages tend to be used for API’s execution, and message₁ use time T₁, message₂ use time T₂ and message₃ use time T₃ , in shared timeout situation, message₁‘s timeout is T , message₂‘s timeout is T - T₁, message₃‘s timeout is T - T₁ - T₂. The remaining timeout should be used for next message to confirm all sub messages could be timeouted as expected.

Application level implements

For message level timeout, the easiest way to implement timeout management is on message bus. Every timeout the message received, check its timeout header and calculate its remaining timeout seems to make sense.

But if we record its timeout by decrease message used time. The machanism became more complex, because every messages time usage need to be recorded but for most product the message profiling is disable to avoid any performance overhead.

In order to solve this problem, we use message deadline as metadata for message lifecyle. Every new coming message should be set a deadline due to its configuration and every sub message calculate its remaining timeout by using the deadline time substract current time. So with a general service to get current time this timeout mechanism get more efficient.

More challenges

In multi-thread application. Lifecyle maintanence of message’s timeout is really matter.

Espacially, ZStack use in-process micro services architecture, messages pass through memory or http, for API message, the timeout always works well, but for internal messages more problems came out.

For example, a async invoker util functios may send several messages but they shares the same thread before handling, so the thread’s context will be used as message’s initial context where the timeout stored.

When thread context changed we can clear the context by thread pool’s thread lifecyle hook.

But some user case the timeout do not work as expected.

• GC task (in memory task triggered by a fixed time rate or any system event)
• Thread level task (async and sync task queue)

GC task

GC task is used to handling some unexpected async operation or retry to delete some resource and so on.

If a thread submit the task executes the task itself, the context will be used directly, and actually the context mess up the GC task’s execution. So always use a new thread to start GC job is a good choice.

For multi-thread application, message dilevery and handling involves different threads, especially some task driver might be used to construct work flow and task execution. So the timeout context need be passed from one thread to another.

Assume Thread1 do task1 and finished with submit a new task2, maybe sometimes after Thread2 start to handle task2 but at this time task2 is required to contain timeout. Or the timeout cannot be passed.

Fixed thread task

Same thread handling all tasks, so task should store its context when submitted to task queue.

Execution need to recover task context before execution.

Benefits of the concepts

For a in-process arch, use a global level timeout during api or inner task lifecycle and the whole timeout can be managed.

Easy implements

In java program, use aop to maintain the timeout get/set seems a good choice.

A typical ZStack task workflow, usually use api at the first step. A new coming api message, ZStack will set timeout to it, but in order to know parent messages timeout, we need to manage the timeout information to the message.

So a design named TaskContext is created to contains the global variables during whole task lifecycle. Use aop, all async tasks will use its parent TaskContext and clear it before start. With TaskContext, the timeout can be managed.

But some user scenarios still need to be discussed, list it before details:

• Inner message is used as the start of a task, it should support timeout.
• API use a inner message configured with timeout which should be supported.
• How did new coming mechanism aware of the timeout from task context
• How to avoid task context be messed

Inner message

Inner message level timeout configuration need to be supported as some tasks is executed by GC task which we mentioned, may send inner message directly, so timeout maybe requested for those tasks

Duplicate configuration

For api messages, it may use a workflow contains several inner messsages when those messages all have timeout configuration, we need to use the origin timeout but do not use new configured one.

Aware of timeout

It not practical, becamse task context is a in-memory variable and marked by thread-id, so everytime the thread switched the task context need to be copied to the new thread. If any mechanism do not support task context copy, it will result in timeout loss, if any inner message used, a new timeout will be set from timeout manager. And it seems no good solution to make the new mechanism aware of this. That’s the shortage of using aop.

Do not touch task context

Only timeout manager should use task context for timeout handling and other task context usage including manually clear it or set value should be avoid.

But for some reason the access of TaskContext supposed to be available to core module for timeout or other context usage (for example, task id), so only keep it cleared after thread context switch and only assign value to framework known fields to avoid any mess up operations from other developer.

Conclusion

Actually task context is more likely a global variable for every thread to use. Keep it from abuse and oom is the first task and aop in involved to resolve this problem. So how to trace task context seems the next valuable target of this version of code.

Check the code

https://github.com/zstackio/zstack check the code if you are interested in this feature.

基于virtio1.2的推送，在virtio-balloon设备下有一条新特性：free page hints

因为不太了解这个东西具体做了啥，查了一番资料

KVM: Guest Free Page Hinting

在2019年2月有这样一封邮件记录 https://lwn.net/Articles/778432/

The following patch-set proposes an efficient mechanism for handing freed memory between the guest and the host. It enables the guests with no page cache to rapidly free and reclaims memory to and from the host respectively.

看起来主要目的是为了优化guest和host之间的空闲内存管理，避免出现需要快速释放或者回收page cache内存

同时里面提到了

Known code re-work:

• Plan to re-use Wei’s work, which communicates the poison value to the host.
• The nomenclatures used in virtio-balloon needs to be changed so that the code can easily be distinguished from Wei’s Free Page Hint code.
• Sorting based on zonenum, to avoid repetitive zone locks for the same zone.

需要对virtio-balloon做一些修改来来保证代码能够和这部分Free Page Hint的代码保持区别。

这么说感觉好像是Hint的代码和virtio-balloon是两套

Virtio-balloon: support free page reporting

基于上面查到的资料，又发现了另外一篇直接提到Virtio-balloon的改动 https://lwn.net/Articles/759413/

里面新增的 VIRTIO_BALLOON_F_FREE_PAGE_HINT 是可以和 https://docs.oasis-open.org/virtio/virtio/v1.2/virtio-v1.2.pdf virtio1.2的spec对应上的

摘抄一下里面的描述

1 2 3 4 5 6 7 8 9 10 11 12

Live migration needs to transfer the VM's memory from the source machine to the destination round by round. For the 1st round, all the VM's memory is transferred. From the 2nd round, only the pieces of memory that were written by the guest (after the 1st round) are transferred. One method that is popularly used by the hypervisor to track which part of memory is written is to write-protect all the guest memory. This feature enables the optimization by skipping the transfer of guest free pages during VM live migration. It is not concerned that the memory pages are used after they are given to the hypervisor as a hint of the free pages, because they will be tracked by the hypervisor and transferred in the subsequent round if they are used and written.

针对热迁移场景，会不停的copy memory。第一轮会复制所有内存，后续只需要复制guest写过的内存。

因此hypervisor需要记录guest写过哪些内存，然后全都复制一遍。

而这个功能是用来优化guest free page transfer的。即忽略这些已经被标记为free page的内容，如果后续这些page被使用了或者被写了，下一轮内存拷贝才考虑这些page。

通过这段描述，可以知道这个优化需要提供一个机制，来提供free page hint，并以此为基础来优化live migration。

小插曲

在准备看代码之前，发现了一段很有意思的内容

1 2 3 4

- mm/get_from_free_page_list: The new implementation to get free page hints based on the suggestions from Linus: https://lkml.org/lkml/2018/6/11/764 This avoids the complex call chain, and looks more prudent.

对获取 get_from_free_page_list 操作，linus回了很长的一段建议

里面很有意思的是，是不是要加一个新的 GFP_NONE，来标记分配失败？

1
2

Maybe it will help to have GFP_NONE which will make any allocation
fail if attempted. Linus, would this address your comment?

而linus的回复是，如果不用这么复杂的会引起内存分配的调用，用一个简单的机制来避免这个问题发生感觉更好

1
2
3
4

So instead of having virtio_balloon_send_free_pages() call a really
generic complex chain of functions that in _some_ cases can do memory
allocation, why isn't there a short-circuited "vitruque_add_datum()"
that is guaranteed to never do anything like that?

中间还有很长的一些简化代码的建议，里面有这么一句话，评价这部分代码太复杂并且太脆弱了

1
2

The whole sequence of events really looks "this is too much
complexity, and way too fragile" to me at so many levels.

让我联想到目前ZStack里面一些功能的实现逻辑存在情况

1. 实现了机制B解决机制A的问题
2. 复用了，不熟悉的机制A，忽略了A本身存在的问题

结合一个实际功能说一下这个问题，比如vm的kernel panic检测，有两个必要选项

1. 给vm增加一个pvpanic的xml配置
2. 虚拟机内部需要启用内核pvpanic模块

因此这个功能实现需要guest和host相互配合才能判定是否可用

基于这个前提，guest内部的逻辑需要提供传递guest内部是否支持pvpanic的信息，host上需要从配置中获取是否配置过pvpanic，因此实现这个逻辑的时候需要分别查询这两个信息。而查询host上的配置最终导致了一些控制面的bug。

后来反思这个问题的时候，只从host配置获取的逻辑出发，但是忽略了运行时配置不会变更的前提，其实并没有必要增加一个多余的查询逻辑，反而导致这个问题依赖了已有的配置查询机制，最终引起了更复杂的现象。

kernel的开源世界也会有人碰到这样的问题，所以整理好功能设计的方法论还是很重要的，至少能够指导怎么做能设计的更好，提升committer和coder的水平。

反过来想想：

1. guest tool在运行时返回的云主机所支持的特性，实际上总是和他的版本绑定的，只要获取过一次其实就不需要反复获取了。
2. 如果guest tool版本发生了变化，才需要重新获取这个信息
3. 提供主动更新guest tool特性的功能即可

其实这样拆解这个问题，云主机其实本身就应该保存这些特性信息而不需要总是去获取，这样机制的设计可以简化很多。

而之前的设计出发点并不是基于对整个功能的理解，而是类似新增 GFP_NONE 来解决问题的思路。

Work with Java reflection and invoke getSimpleName() met java.lang.NoClassDefFoundError

Use getName() instead, the logic seems work well.

Before compare these two methods’ difference go through the code quickly.

Class::getName()

for getName()

1
2
3
4
5
6

public String getName() {
    String name = this.name;
    if (name == null)
        this.name = name = getName0();
    return name;
}

and the getName0() is native method

1

private native String getName0();

Class::getSimpleName()

for getSimpleName()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

public String getSimpleName() {
    if (isArray())
        return getComponentType().getSimpleName()+"[]";

    String simpleName = getSimpleBinaryName();
    if (simpleName == null) { // top level class
        simpleName = getName();
        return simpleName.substring(simpleName.lastIndexOf(".")+1); // strip the package name
    }
    // According to JLS3 "Binary Compatibility" (13.1) the binary
    // name of non-package classes (not top level) is the binary
    // name of the immediately enclosing class followed by a '$' followed by:
    // (for nested and inner classes): the simple name.
    // (for local classes): 1 or more digits followed by the simple name.
    // (for anonymous classes): 1 or more digits.

    // Since getSimpleBinaryName() will strip the binary name of
    // the immediatly enclosing class, we are now looking at a
    // string that matches the regular expression "\$[0-9]*"
    // followed by a simple name (considering the simple of an
    // anonymous class to be the empty string).

    // Remove leading "\$[0-9]*" from the name
    int length = simpleName.length();
    if (length < 1 || simpleName.charAt(0) != '$')
        throw new InternalError("Malformed class name");
    int index = 1;
    while (index < length && isAsciiDigit(simpleName.charAt(index)))
        index++;
    // Eventually, this is the empty string iff this is an anonymous class
    return simpleName.substring(index);
}

The main part is

1

String simpleName = getSimpleBinaryName();

And then get enclosingClass:

1
2
3
4
5
6
7
8
9
10
11

private String getSimpleBinaryName() {
    Class<?> enclosingClass = getEnclosingClass();
    if (enclosingClass == null) // top level class
        return null;
    // Otherwise, strip the enclosing class' name
    try {
        return getName().substring(enclosingClass.getName().length());
    } catch (IndexOutOfBoundsException ex) {
        throw new InternalError("Malformed class name", ex);
    }
}

what is enclosingClass:

1
2
3
4
5
6

// There are five kinds of classes (or interfaces):
// a) Top level classes
// b) Nested classes (static member classes)
// c) Inner classes (non-static member classes)
// d) Local classes (named classes declared within a method)
// e) Anonymous classes

in my case it tend to be a) Top level classes, so next part of code goes to getDeclaringClass()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

// JVM Spec 4.8.6: A class must have an EnclosingMethod
// attribute if and only if it is a local class or an
// anonymous class.
EnclosingMethodInfo enclosingInfo = getEnclosingMethodInfo();
Class<?> enclosingCandidate;

if (enclosingInfo == null) {
    // This is a top level or a nested class or an inner class (a, b, or c)
    enclosingCandidate = getDeclaringClass();
} else {
    Class<?> enclosingClass = enclosingInfo.getEnclosingClass();
    // This is a local class or an anonymous class (d or e)
    if (enclosingClass == this || enclosingClass == null)
        throw new InternalError("Malformed enclosing method information");
    else
        enclosingCandidate = enclosingClass;
}

and then getDeclaringClass0() will be used.

1
2
3
4
5
6
7
8
9
10

@CallerSensitive
public Class<?> getDeclaringClass() throws SecurityException {
    final Class<?> candidate = getDeclaringClass0();

    if (candidate != null)
        candidate.checkPackageAccess(
                ClassLoader.getClassLoader(Reflection.getCallerClass()), true);
    return candidate;
}

which is a native method:

1

private native Class<?> getDeclaringClass0();

Comparision

From the code before this section, aboveously the getSimpleName() always call native method but getName() may use Class’s local variable directly.

So see the local variable before we come to conclusion.

1
2

// cache the name to reduce the number of calls into the VM
private transient String name;

the name used by getName() use a transient String as cache to reduce the number of calls into VM.

And combine to getName()’s implement, the first time getName0() invoked, this name is set.

And go for Java doc:

1

public String getSimpleName()

Returns the simple name of the underlying class as given in the source code. Returns an empty string if the underlying class is anonymous.

The simple name of an array is the simple name of the component type with “[]” appended. In particular the simple name of an array whose component type is anonymous is “[]”.

• Returns:

  the simple name of the underlying class

• Since:

  1.5

1

public String getName()

Returns the name of the entity (class, interface, array class, primitive type, or void) represented by this Class object, as a String.

If this class object represents a reference type that is not an array type then the binary name of the class is returned, as specified by The Java™ Language Specification.

If this class object represents a primitive type or void, then the name returned is a String equal to the Java language keyword corresponding to the primitive type or void.

If this class object represents a class of arrays, then the internal form of the name consists of the name of the element type preceded by one or more ‘[‘ characters representing the depth of the array nesting. The encoding of element type names is as follows:

Element Type		Encoding
boolean		Z
byte		B
char		C
class or interface		Lclassname;
double		D
float		F
int		I
long		J
short		S

The class or interface name classname is the binary name of the class specified above.

Examples:

1 2 3 4 5 6 7 8 9

String.class.getName() returns "java.lang.String" byte.class.getName() returns "byte" (new Object[3]).getClass().getName() returns "[Ljava.lang.Object;" (new int[3][4][5][6][7][8][9]).getClass().getName() returns "[[[[[[[I"

• Returns:

  the name of the class or interface represented by this object.

and more details for the error:

1
2

public class NoClassDefFoundError
extends LinkageError

Thrown if the Java Virtual Machine or a ClassLoader instance tries to load in the definition of a class (as part of a normal method call or as part of creating a new instance using the new expression) and no definition of the class could be found.

The searched-for class definition existed when the currently executing class was compiled, but the definition can no longer be found.

So that means class found at compile time but not available at runtime Refer this link

Check for our configuration:

1
2
3
4
5
6
7

<dependency>
    <groupId>xxx</groupId>
    <artifactId>xxx</artifactId>
    <version>1.1.1</version>
    <scope>system</scope>
    <systemPath>${project.basedir}/ext-libs/xxx</systemPath>
</dependency>

use a system scope.

According to maven doc:

There are 6 scopes:

• compile
  This is the default scope, used if none is specified. Compile dependencies are available in all classpaths of a project. Furthermore, those dependencies are propagated to dependent projects.
• provided
  This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. A dependency with this scope is added to the classpath used for compilation and test, but not the runtime classpath. It is not transitive.
• runtime
  This scope indicates that the dependency is not required for compilation, but is for execution. Maven includes a dependency with this scope in the runtime and test classpaths, but not the compile classpath.
• test
  This scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases. This scope is not transitive. Typically this scope is used for test libraries such as JUnit and Mockito. It is also used for non-test libraries such as Apache Commons IO if those libraries are used in unit tests (src/test/java) but not in the model code (src/main/java).
• system
  This scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository.
• import
  This scope is only supported on a dependency of type pom in the <dependencyManagement> section. It indicates the dependency is to be replaced with the effective list of dependencies in the specified POM’s <dependencyManagement> section. Since they are replaced, dependencies with a scope of import do not actually participate in limiting the transitivity of a dependency.

system most like provided but a dependency with this scope is added to the classpath used for compilation and test, but not the runtime classpath

so runtime getSimpleName() will met exception.

Class loader

get back to the error call trace again:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

Caused by: java.lang.NoClassDefFoundError: xxxxx
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_161]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_161]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_161]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) ~[?:1.8.0_161]
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73) ~[?:1.8.0_161]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368) ~[?:1.8.0_161]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362) ~[?:1.8.0_161]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_161]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_161]
        at java.lang.Class.getDeclaringClass0(Native Method) ~[?:1.8.0_161]

and another class not found exception:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

Caused by: java.lang.ClassNotFoundException: xxxxx
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_161]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_161]
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_161]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_161]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_161]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) ~[?:1.8.0_161]
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73) ~[?:1.8.0_161]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368) ~[?:1.8.0_161]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362) ~[?:1.8.0_161]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_161]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_161]
        at java.lang.Class.getDeclaringClass0(Native Method) ~[?:1.8.0_161]

when try to get simple name of class A extends B

find A and try to define A but need to define B first, but B is not available at runtime so a exception raised.