When use virt-v2v to convert vm from vmware to kvm, windows guest sometimes meet issue after start on kvm. For quite old windows os version, typically windows 2003, may meet following blue screen issue:

with code 0x0000007B.

So I write a hands-on blog to help fix this issue.

When search for this error on vmware kb or MSDN, we can find out that, wrong driver is the main reason cause this problem.

Usually, it’s recommanded to do following check for the v2v vm:

1. Confirm the vm running well on original host, if its already error check configuration first.
2. Check if virt-v2v failed to install virtio driver to guest during virt-v2v convert.
3. Check kvm configuration, that root disk use ide to keep compitable.
4. Check if windows disabled driver installatin by its group policy.

Besides those issues, if you migrate a windows 2003 with cd-rom you will also met this blue screen issue. Due to windows will change device order after v2v and root disk not work as expected.

So I write this practical blog to resolve device changed issue on windows 2003 v2v.

1. Uninstall vmware guest tools and drivers (avoid failed to install driver issue)

2. Change the disk controller to ide and reboot guest

   

3. Attach MergeIde.iso to guest

   

   the iso contains a .reg and a .bat file. Run the .reg first and run the .bat. The device will be recorded and prepare for hypervisor change.

4. 

   export ovf template or use virt-v2v to export the vm the kvm.

5. 

   check the vm started without blue screen.

For windows, it seems that with cd-rom or disk order changed guest, after hypervisor migration, windows can not identify the original order of thoes devices due to hardware change. So record the device seems a easy solution for v2v case.

Note: this solution is also useful for Windows XP v2v

Download MergeIDE.iso from: https://www.virtualbox.org/wiki/Migrate_Windows

Refer to Microsoft commands: https://learn.microsoft.com/en-US/troubleshoot/windows-client/performance/stop-error-7b-or-inaccessible-boot-device-troubleshooting

Introduction

Software’s complexity grows as time goes by which is suitable to describe ZStack now. Quantities of features has been integrated in the past years and developers suffered on customizing features for some commercial reasons which pushed us to think about how to get rid of customizing feature efforts and focus on core function to earn more technical advantages.

Generallly, customizing features is harmful when compared to normal features (but maybe it can get commercial benifits so do not take this as absolute rule), because more customers use normal features.

But because some integration requests already designed which not only require maintainance but also need development for new integration. For example, a feature for security is desing to use third-part machine to do encryption，but when customer use the feature, they all use different machine which caused lots of development cost for us to do the integrations.

In these cases, a plugable architecture is required to cut down the cost and this is the reason to write this blog to record related design and make it a repeatable practice.

Requirements

For existing systems which need to support multi types of resources, typically use a register way to connect the management layer and application layers. And the application layers use a couple of standard interfaces to keep consistency and seperate implementation from mechanism.

So for the aim of delveloping a plugable module architecture if we still follows the pattern, easily we will be distrubed by the change of core codes from the mechanism. For example, if the core changes the mechanism as a result, application should change all of the interfaces as expected, so development is required which is not satisfied with out target.

Just after that small think, list all requirements before starting design. Following points are the aim of this architecture:

• Low development efforts or configuration as a service.
• No core code dependency. Any changes to mechanism should be avoid.
• Security. Plugin module should not cause any security issue.

Base on those points, the architecture should be changed to expose its mechanism by interfaces.

For example

1. read plugin configuration and verify it
2. generate code proxy according to the configuration
3. limited db access controled by the code proxy
4. Invokations can be verified by unit test to check the plugin can work as expected
5. change the configuration the refresh it could update the code proxy
6. plugin configurations use a specified syntax
7. if code proxy support runtime refresh can be configured

From writing configurations, functional features can be easily supported.

If any requests need data structure support, a configuration map shoud be support for more customize and in this way we can easily create a http client to support some modules.

But if customize feature use any third-part libs, only configuration is not enough to support.

For code level, maybe open source interfaces is still needed.

We should offer read only data structure and functional interfaces but should not involve core codes of Java which in most cases, are aspectj support, spring containers and so on. Only pure java should be used for all plugin modules.

In next section, we will raise examples of two module, one is not suitable for plugable module and other one is suitable for plugable.

Login module

Multiple login methods should be supported because many standard third-part authentication is already exists for example, CAS, oauth2, ldap and so on.

If you already have a customized system for authentication, I seems diffcult to integrate another system with existing one. Because the authentication actually need to transfer sensitive information to do authenticate but which is not safe enough to expose those data for plugin exactly.

From the usage of CAS, oauth2 we can see a generally authentication service just redirect authentication to itself and returns a authentication result and finally redirect to the right page. So sensitive information do not transferred but use a token to verify client’s response instead.

For ldap, just access its database directly to verify if the user is exists and finish authentication. Because ldap just integrated with ZStack so also no data transferred but ldap module itself need to care about the security issue which already resolve by third-part libs.

So when we tries to support plugin for login, password or any key liked information need to be exposed which is quite unsafe and have potential security risks. Compared to ldap verification, password is used directly and also faces the same problem.

For login module integration, two ways are recommended. One is using open-source authentication like CAS or oauth2. Another is integrating ZStack API to use account/user directly.

Following figure shows the flows of login module

• Support plugable system, main auth or additinal auth should be exposed
• Sensitive information go through the whole flow
• CAS and oauth2 use their own authentication which is not related the current architecture

Alarm/Event system

For alarm event system, only expose monitor messages which seems more suitable to be developed as a plugable system.

When have lots of endpoint support, some use stantard libs like dingtalk and microsoft teams, offers api for message sending. But when sms system require integration, this became diffcult due to the lack of stantard apis.

So a plugable system can be powerful to reduce development efforts.

For example, sms systems all have their own apis the usages, so the interation of api is the first step. But actually the business scenarios are not only limited on send message. Maybe message distribution strategy is required, and message format is required.

So seperate those parts as followings:

• Send sms message
• Distribution strategy
• Message format
• ….

When design plugable system, the most important thing is that we should seperate mechanism requirements and strategy requirements.

For message format, actually, all kinds of message required this feature and can be noted as general feature and mechanism and we can also call it customize message format. The format change do not influence how the message send or comsumed.

Distribution strategy, from the name, we can known that no specific rules or format could be announced because the strategy always changes from one the another.

Send sms message. also the integration part but the message with a format is what we can offered and only how to send the message should be defined and which is I think can be seperated from the system.

And more informations maybe, endpoint type, status and those ZStack defined fields also need to be involved in the plugable system so we get a architecture like following:

A plugable endpoint in designed to manage endpoint plugins and plugin should obey following rules:

• Offer endpoint type
• Implement interface to send message
• Unit test for endpoint plugin

And the design of plugable endpoint may like following:

So plugable endpoint should define a interface require type and send() method’s implementation for the usage. Take the interface to a seperate open source package and publish to maven repository, its easy for develop and easy to use. By reflections, endpoint could be initialized from jar so just add jar to your dependency directory is ok.

To develop a plugin for new sns endpoint no need to known the logic or usage of orignal application system but developer could focus on the send method itself.

On the other hand, application level should extend orignal CURD api to suitable for those plugin definition and keep compitable with other endpoints.

TODO

• Database access is not limited which should add limited on database service itself.
• Rules for all kind of plugins so that plugins can be managed together and distribution can keep consistency
• More examples
• For exsiting modules, try to refactor the plugin types to get rid of module dependency

Background

Out system became unavailable due to java.lang.OutOfMemoryError: Metaspace and according to a stackoverflow answer: https://stackoverflow.com/questions/36051813/java8-java-lang-outofmemoryerror-metaspace, I started to figure out what wrong with the system.

Before finding start analysis jvm memory dump, we need to known what means memory leak. In this case, we found thread failed to execution because create task failed due to OOM. But there are two typical reason.

• Too many classes need to be loaded
• Memory leak causes some classes not unloaded

Everytime you new a object, metaspace will be allocated to store metadata of the object.

Following code could reproduce this issue:

1
2
3
4
5
6
7
8
9
10
11

import javassist.ClassPool;

public class MetaspaceOOM {
    static ClassPool cp = ClassPool.getDefault();

    public static void main(String[] args) throws Exception{
        for (int i = 0; ; i++) {
            Class c = cp.makeClass("eu.plumbr.demo.Generated" + i).toClass();
        }
    }
}

pom.xml

1
2
3
4
5

<dependency>
    <groupId>org.javassist</groupId>
    <artifactId>javassist</artifactId>
    <version>3.27.0-GA</version>
</dependency>

so dynamic created class will add new class information into metaspace but if there is no class loader reference to the class, the class will be cleared after GC.

So it’s clear that if memory leak happened to your metaspace, you can find your application no problem when just started and oom happens after it runs for a period of time.

Tools

A couple of tools are available for memory leak analysis and only refer to what I have used when I figure out this issue.

JConsole

JConsole is a graphical monitoring tool to monitor Java Virtual Machine (JVM) and Java applications both on a local or remote machine.

JConsole uses underlying features of Java Virtual Machine to provide information on performance and resource consumption of applications running on the Java platform using Java Management Extensions (JMX) technology. JConsole comes as part of Java Development Kit (JDK) and the graphical console can be started using “jconsole” command.

As JConsole is a part of JDK, you can easily find it. On macOS, you can use commandline directly

1

jconsole

or find your java_home by

1

/usr/libexec/java_home -V

the jconsole is under

1

/Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home/bin/jconsole

and then connect to local or remote application.

Note:

In order to use JConsole, jmx need to be enabled for your application’s jvm options

1

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=your_ip_address

make sure your ip adress and port is available for remote access.

From JConsole we can monitor loaded class number, unloaded class number, perform gc and check if metaspace is continuously increasing.

 VisualVM

A replacement for JConsole except jvm monitor, heap dump, application snapshot is available which is quite efficient for memory leak analysis.

Memory Analyzer (MAT)

The Eclipse Memory Analyzer is a fast and feature-rich Java heap analyzer that helps you find memory leaks and reduce memory consumption.Use the Memory Analyzer to analyze productive heap dumps with hundreds of millions of objects, quickly calculate the retained sizes of objects, see who is preventing the Garbage Collector from collecting objects, run a report to automatically extract leak suspects.

My application use openjdk-1.8, so only MemoryAnalyzer-1.11.0.20201202-macosx.cocoa.x86_64.dmg

could be used. After that MAT only support java11.

Start debug

Combining background and tools, finding continuously increasing memory is easy but how to get to the code seems more diffcult.

But we could seperate debug procedure to several steps

• Do heap dump during a period to known always in memory classes and dynamic classes
• Compare heap dump to figure out what increase metaspace
• Find a operation that causes the leak

Use visualVM connect to java application and monitor for a time

Metaspace leak can be monitored because after manually perforce GC, it still not decrease.

GC performed we have 72246 loaded classes.

after aboubt 1minutes, GC performed but loaded class increased to 72262


Aboviously there are some leak problems, so still use 1 minute as period and collect two heap dump for next step analysis.

Open the heap dumps with MAT to do compare, from heap dump’s dominator tree

Luckily, we found the increased class at first glance. The SessionFactoryImpl seems to blame.

Except this we found another issue with Groovy’s GStringTemplateEngine which causes groovy.reflection.ClassInfo increases and we will tell the details in next section.

Expand the suspects to get more details about it. By list all objects, query plan cache increased, so just google for this value

from https://docs.jboss.org/hibernate/orm/5.0/javadocs/org/hibernate/engine/query/spi/QueryPlanCache.html: Acts as a cache for compiled query plans, as well as query-parameter metadata.

And check its source code:

1
2
3
4

/**
 * the cache of the actual plans...
 */
private final BoundedConcurrentHashMap queryPlanCache;

it seems a bounded hashmap and created like below:

1

queryPlanCache = new BoundedConcurrentHashMap( maxQueryPlanCount, 20, BoundedConcurrentHashMap.Eviction.LIRS );

and hibernate use default 2048 as maxQueryPlanCount and LIRS as eviction strategy. So no need to solve a cache increasing.

GStringTemplateEngine

By debug steps, we find GStringTemplateScript occupied metaspace and thousands of classes are created.

Check its implements,

1

groovyClass = loader.parseClass(new GroovyCodeSource(templateExpressions.toString(), "GStringTemplateScript" + GStringTemplateEngine.counter.incrementAndGet() + ".groovy", "x"));

GStringTemplateScript will be created every GStringTemplate created. So for template creation, will cause dynamic class creation and finally let jvm run out of memory.

According to some related fix, https://issues.apache.org/jira/browse/GROOVY-7017 this fix use a seperate class loader to make sure gstring template could be gced from heap but metaspace is still occupied.

And another article talk about: https://tigase.net/how-aws-helped-us-optimize-memory-usage-tigase-http-api/ AWS improve Tigase HTTP API Memory usage, following fix is suggested:

• We load all templates at once using single GStringTemplateEngine and cache generate templates. No more automatic reloading of templates.
• When manual reload of templates is initiated we release old instance of GStringTemplateEngine and parse templates using the new one.

So do some test before change our code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

import groovy.text.GStringTemplateEngine;
import javassist.ClassPool;

import java.io.IOException;
import java.io.StringReader;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryPoolMXBean;
import java.util.HashMap;

public class GroovyMetaspaceOOM {
    static ClassPool cp = ClassPool.getDefault();

    public static void main(String[] args) throws Exception{
        final String EVENT_CHINESE_TEMPLATE = "事件 发生了 事件详情: 名称:";

        GStringTemplateEngine engine = new GStringTemplateEngine();
        HashMap<Integer, groovy.text.Template> templateHashMap = new HashMap<>();

        while (true) {
            groovy.text.Template template = templateHashMap.get(EVENT_CHINESE_TEMPLATE.hashCode());
            if (template == null) {
                try {
                    template = (groovy.text.Template) engine.createTemplate(new StringReader(EVENT_CHINESE_TEMPLATE));
                } catch (ClassNotFoundException | IOException e) {
                    throw new RuntimeException(e);
                }

                templateHashMap.put(EVENT_CHINESE_TEMPLATE.hashCode(), template);
            }

            System.out.println(template.make().toString());

            for (MemoryPoolMXBean memoryMXBean : (ManagementFactory.getMemoryPoolMXBeans())) {
                if ("Metaspace".equals(memoryMXBean.getName())) {
                    System.out.println(memoryMXBean.getUsage().getUsed()/1024/1024 + " mb");
                }
            }

            System.gc();
        }
    }
}

by use templateHashMap as cache to avoid duplicate template creation, we can see Metaspace memory usage not changed

but if change the code without templateHashMap:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

import groovy.text.GStringTemplateEngine;
import javassist.ClassPool;

import java.io.IOException;
import java.io.StringReader;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryPoolMXBean;

public class GroovyMetaspaceOOM {
    static ClassPool cp = ClassPool.getDefault();

    public static void main(String[] args) throws Exception{
        final String EVENT_CHINESE_TEMPLATE = "事件 发生了 事件详情: 名称:";

        GStringTemplateEngine engine = new GStringTemplateEngine();

        while (true) {
            groovy.text.Template template;
            //= templateHashMap.get(EVENT_CHINESE_TEMPLATE.hashCode());
            try {
                template = (groovy.text.Template) engine.createTemplate(new StringReader(EVENT_CHINESE_TEMPLATE));
            } catch (ClassNotFoundException | IOException e) {
                throw new RuntimeException(e);
            }

            System.out.println(template.make().toString());

            for (MemoryPoolMXBean memoryMXBean : (ManagementFactory.getMemoryPoolMXBeans())) {
                if ("Metaspace".equals(memoryMXBean.getName())) {
                    System.out.println(memoryMXBean.getUsage().getUsed()/1024/1024 + " mb");
                }
            }

            System.gc();
        }
    }
}

the metaspace will increase until oom.