languages►java

JVM loaded classes rapidly increased issue

Rapidly increased JVM loaded classes

When debug a oom issue on new version of application. I noticed that

Old version:

New version:

The classes increased almost twice compare to the old version.

So I just export loaded classes to figure out what classes increased at first.

After a compare, we got a obvious increased class:

for the new version

about 25000 Doc classes is loaded and 45439 + 25000 = 70439 seems very near to 74.3k.

So check the heap dump by Dominator tree, by comparing heap usage, one class come into view is that:

Code reading

ZStack use reflections to get information and offer framework level capabilities.

1
2
3
public static Reflections reflections = new Reflections(ClasspathHelper.forPackage("org.zstack"),
        new SubTypesScanner(), new MethodAnnotationsScanner(), new FieldAnnotationsScanner(),
        new TypeAnnotationsScanner(), new MethodParameterScanner());

for reflections-0.9.10

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
protected void scan(URL url) {
    Vfs.Dir dir = Vfs.fromURL(url);

    try {
        for (final Vfs.File file : dir.getFiles()) {
            // scan if inputs filter accepts file relative path or fqn
            Predicate<String> inputsFilter = configuration.getInputsFilter();
            String path = file.getRelativePath();
            String fqn = path.replace('/', '.');
            if (inputsFilter == null || inputsFilter.apply(path) || inputsFilter.apply(fqn)) {
                Object classObject = null;
                for (Scanner scanner : configuration.getScanners()) {
                    try {
                        if (scanner.acceptsInput(path) || scanner.acceptResult(fqn)) {
                            classObject = scanner.scan(file, classObject);
                        }
                    } catch (Exception e) {
                        if (log != null && log.isDebugEnabled())
                            log.debug("could not scan file " + file.getRelativePath() + " in url " + url.toExternalForm() + " with scanner " + scanner.getClass().getSimpleName(), e.getMessage());
                    }
                }
            }
        }
    } finally {
        dir.close();
    }
}

with a groovy closure org.zstack.sso.header.APICreateCasClientEventDoc_zh_cn$_run_closure1

this part of code executed will skip filter and goes throught all scanners with their acceptsInput and acceptResult directly

refer to ZStack used Scanner, following code will be used:

FieldAnnotationsScanner

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
public class FieldAnnotationsScanner extends AbstractScanner {
    public void scan(final Object cls) {
        final String className = getMetadataAdapter().getClassName(cls);
        List<Object> fields = getMetadataAdapter().getFields(cls);
        for (final Object field : fields) {
            List<String> fieldAnnotations = getMetadataAdapter().getFieldAnnotationNames(field);
            for (String fieldAnnotation : fieldAnnotations) {

                if (acceptResult(fieldAnnotation)) {
                    String fieldName = getMetadataAdapter().getFieldName(field);
                    getStore().put(fieldAnnotation, String.format("%s.%s", className, fieldName));
                }
            }
        }
    }
}

SubTypeScanner

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
public class SubTypesScanner extends AbstractScanner {

    /** created new SubTypesScanner. will exclude direct Object subtypes */
    public SubTypesScanner() {
        this(true); //exclude direct Object subtypes by default
    }

    /** created new SubTypesScanner.
     * @param excludeObjectClass if false, include direct {@link Object} subtypes in results.  */
    public SubTypesScanner(boolean excludeObjectClass) {
        if (excludeObjectClass) {
            filterResultsBy(new FilterBuilder().exclude(Object.class.getName())); //exclude direct Object subtypes
        }
    }

    @SuppressWarnings({"unchecked"})
    public void scan(final Object cls) {
		String className = getMetadataAdapter().getClassName(cls);
		String superclass = getMetadataAdapter().getSuperclassName(cls);

        if (acceptResult(superclass)) {
            getStore().put(superclass, className);
        }

		for (String anInterface : (List<String>) getMetadataAdapter().getInterfacesNames(cls)) {
			if (acceptResult(anInterface)) {
                getStore().put(anInterface, className);
            }
        }
    }
}

MethodAnnotationsScanner

1
2
3
4
5
6
7
8
9
10
11
public class MethodAnnotationsScanner extends AbstractScanner {
    public void scan(final Object cls) {
        for (Object method : getMetadataAdapter().getMethods(cls)) {
            for (String methodAnnotation : (List<String>) getMetadataAdapter().getMethodAnnotationNames(method)) {
                if (acceptResult(methodAnnotation)) {
                    getStore().put(methodAnnotation, getMetadataAdapter().getMethodFullKey(cls, method));
                }
            }
        }
    }
}

TypeAnnotationsScanner

1
2
3
4
5
6
7
8
9
10
11
12
13
14
public class TypeAnnotationsScanner extends AbstractScanner {
    public void scan(final Object cls) {
      final String className = getMetadataAdapter().getClassName(cls);

        for (String annotationType : (List<String>) getMetadataAdapter().getClassAnnotationNames(cls)) {

            if (acceptResult(annotationType) ||
                annotationType.equals(Inherited.class.getName())) { //as an exception, accept Inherited as well
                getStore().put(annotationType, className);
            }
        }
    }

}

MethodParameterScanner

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
public class MethodParameterScanner extends AbstractScanner {

    @Override
    public void scan(Object cls) {
        final MetadataAdapter md = getMetadataAdapter();

        for (Object method : md.getMethods(cls)) {

            String signature = md.getParameterNames(method).toString();
            if (acceptResult(signature)) {
                getStore().put(signature, md.getMethodFullKey(cls, method));
            }

            String returnTypeName = md.getReturnTypeName(method);
            if (acceptResult(returnTypeName)) {
                getStore().put(returnTypeName, md.getMethodFullKey(cls, method));
            }

            List<String> parameterNames = md.getParameterNames(method);
            for (int i = 0; i < parameterNames.size(); i++) {
                for (Object paramAnnotation : md.getParameterAnnotationNames(method, i)) {
                    if (acceptResult((String) paramAnnotation)) {
                        getStore().put((String) paramAnnotation, md.getMethodFullKey(cls, method));
                    }
                }
            }
        }
    }
}

All of them only offer scan and not extra processes were defined.

Check abount their acceptsInput() and acceptResult() methods from AbstractScanner:

1
2
3
4
5
6
7
 public boolean acceptsInput(String file) {
     return getMetadataAdapter().acceptsInput(file);
 }
 
 public boolean acceptResult(final String fqn) {
   return fqn != null && resultFilter.apply(fqn);
}

acceptResult will be used by SubTypesScanner

1
2
3
4
5
public SubTypesScanner(boolean excludeObjectClass) {
    if (excludeObjectClass) {
        filterResultsBy(new FilterBuilder().exclude(Object.class.getName())); //exclude direct Object subtypes
    }
}

to reject direct Object subtypes

and acceptsInput use metadataAdapter(JavassistAdapter.java)，check if file end with .class

so for reflections-0.9.10 if groovy closure file exists will .class suffix and not extend object directly, will be loaded by reflection.

After upgrade to reflections-0.10.2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
(configuration.isParallel() ? urls.stream().parallel() : urls.stream()).forEach(url -> {
    Vfs.Dir dir = null;
    try {
        dir = Vfs.fromURL(url);
        for (Vfs.File file : dir.getFiles()) {
            if (doFilter(file, configuration.getInputsFilter())) {
                ClassFile classFile = null;
                for (Scanner scanner : configuration.getScanners()) {
                    try {
                        if (doFilter(file, scanner::acceptsInput)) {
                            List<Map.Entry<String, String>> entries = scanner.scan(file);
                            if (entries == null) {
                                if (classFile == null) classFile = getClassFile(file);
                                entries = scanner.scan(classFile);
                            }
                            if (entries != null) collect.get(scanner.index()).addAll(entries);
                        }
                    } catch (Exception e) {
                        if (log != null) log.trace("could not scan file {} with scanner {}", file.getRelativePath(), scanner.getClass().getSimpleName(), e);
                    }
                }
            }
        }
    } catch (Exception e) {
        if (log != null) log.warn("could not create Vfs.Dir from url. ignoring the exception and continuing", e);
    } finally {
        if (dir != null) dir.close();
    }
});

code of scan changed to new version and scanners moved to enum

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
public enum Scanners implements Scanner, QueryBuilder, NameHelper {

    /** scan type superclasses and interfaces
     * <p></p>
     * <i>Note that {@code Object} class is excluded by default, in order to reduce store size.
     * <br>Use {@link #filterResultsBy(Predicate)} to change, for example {@code SubTypes.filterResultsBy(c -> true)}</i>
     * */
    SubTypes {
        /* Object class is excluded by default from subtypes indexing */
        { filterResultsBy(new FilterBuilder().excludePattern("java\\.lang\\.Object")); }

        @Override
        public void scan(ClassFile classFile, List<Map.Entry<String, String>> entries) {
            entries.add(entry(classFile.getSuperclass(), classFile.getName()));
            entries.addAll(entries(Arrays.asList(classFile.getInterfaces()), classFile.getName()));
        }
    },

    /** scan method annotations */
    MethodsAnnotated {
        @Override
        public void scan(ClassFile classFile, List<Map.Entry<String, String>> entries) {
            getMethods(classFile).forEach(method ->
                entries.addAll(entries(getAnnotations(method::getAttribute), methodName(classFile, method))));
        }
    },
    
    /** scan field annotations */
    FieldsAnnotated {
        @Override
        public void scan(ClassFile classFile, List<Map.Entry<String, String>> entries) {
            classFile.getFields().forEach(field ->
                entries.addAll(entries(getAnnotations(field::getAttribute), fieldName(classFile, field))));
        }
    },

    /** scan type annotations */
    TypesAnnotated {
        @Override
        public boolean acceptResult(String annotation) {
            return super.acceptResult(annotation) || annotation.equals(Inherited.class.getName());
        }

        @Override
        public void scan(ClassFile classFile, List<Map.Entry<String, String>> entries) {
            entries.addAll(entries(getAnnotations(classFile::getAttribute), classFile.getName()));
        }
    },

    /** scan method parameters types and annotations */
    MethodsParameter {
        @Override
        public void scan(ClassFile classFile, List<Map.Entry<String, String>> entries) {
            getMethods(classFile).forEach(method -> {
                String value = methodName(classFile, method);
                entries.addAll(entries(getParameters(method), value));
                getParametersAnnotations(method).forEach(annotations -> entries.addAll(entries(annotations, value)));
            });
        }
    },

almost same logic is supported but check details about the scan() method

1
2
3
4
5
6
7
8
9
10
11
12
13
14
for (Scanner scanner : configuration.getScanners()) {
    try {
        if (doFilter(file, scanner::acceptsInput)) {
            List<Map.Entry<String, String>> entries = scanner.scan(file);
            if (entries == null) {
                if (classFile == null) classFile = getClassFile(file);
                entries = scanner.scan(classFile);
            }
            if (entries != null) collect.get(scanner.index()).addAll(entries);
        }
    } catch (Exception e) {
        if (log != null) log.trace("could not scan file {} with scanner {}", file.getRelativePath(), scanner.getClass().getSimpleName(), e);
    }
}

acceptsInput will be used to do filter which check file end with .class suffix but filterResultsBy is not executed only if entries = scanner.scan(classFile) is used.

So when List<Map.Entry<String, String>> entries = scanner.scan(file); return entries the result won’t be exclude.

Hands-on test

I set up a maven project to test reflections issue with a project:

and main code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
package org.zstack;

import org.reflections.Reflections;
import org.reflections.scanners.*;
import org.reflections.util.ClasspathHelper;

import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.util.Set;

public class TestReflections {
    public static void main(String[] args) throws IOException {
        System.out.println("==========================");
        printLoadedClasses(null);
        System.out.println("==========================");


        // reflections 0.9.10
        Reflections reflections = new Reflections(ClasspathHelper.forPackage("org.zstack"),
            new SubTypesScanner(false), new MethodAnnotationsScanner(), new FieldAnnotationsScanner(),
            new TypeAnnotationsScanner(), new MethodParameterScanner());

        // reflections 0.10.2
//        Reflections reflections = new Reflections(ClasspathHelper.forPackage("org.zstack"),
//            new SubTypesScanner(false), new MethodAnnotationsScanner(), new FieldAnnotationsScanner(),
//            new TypeAnnotationsScanner(), new MethodParameterScanner());

        System.out.println("==========================");
        printLoadedClasses(reflections);
        System.out.println("==========================");
    }

    private static void printLoadedClasses(Reflections reflections) throws IOException {
        if (reflections != null) {
            Set<String> types = reflections.getAllTypes();

            types.forEach(System.out::println);

            System.out.println("loaded class number from reflection: " + types.size());
        }

        System.out.println("loaded class number from jmx: " + ManagementFactory.getClassLoadingMXBean().getLoadedClassCount());
    }
}

finally the output of reflections 0.10.2:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
groovy.lang.GroovyObject
java.lang.Cloneable
org.codehaus.groovy.runtime.GeneratedClosure
groovy.lang.GroovyObjectSupport
groovy.lang.Closure
java.util.concurrent.Callable
java.lang.Object
groovy.lang.GroovyCallable
groovy.lang.Script
java.lang.Runnable
java.io.Serializable
org.test.TestGroovy2$_run_closure1
org.zstack.TestGroovy2$_run_closure1
org.test.TestGroovy$_run_closure1
org.zstack.TestGroovy$_run_closure1$_closure2
org.zstack.TestGroovy$_run_closure1
org.zstack.TestGroovy3$_run_closure1$_closure2
org.test.TestGroovy$_run_closure1$_closure2
org.test.TestGroovy3$_run_closure1$_closure2
org.test.TestGroovy3$_run_closure1
org.test.TestGroovy2$_run_closure1$_closure2
org.zstack.TestGroovy2$_run_closure1$_closure2
org.zstack.TestGroovy3$_run_closure1
org.zstack.TestReflections
org.test.TestGroovy2
org.test.TestGroovy3
org.zstack.TestGroovy
org.test.TestGroovy
org.zstack.TestGroovy2
org.zstack.TestGroovy3
loaded class number from reflection: 30

and the reflection 0.10.9:

1
2
org.zstack.TestReflections
loaded class number from reflection: 1

we can found out that even a specified reflection from class path org.zstack unrelated class still appears.

So I just goolge the reflection issue, it come out directly:

https://github.com/ronmamo/reflections/issues/373

we can solve this issue by adding some workaround

1
2
3
4
5
6
7
8
9
10
11
// reflections 0.10.2
ConfigurationBuilder builder = ConfigurationBuilder.build()
        .setUrls(ClasspathHelper.forPackage("org.zstack"))
        .setScanners(new SubTypesScanner(false),
                new MethodAnnotationsScanner(),
                new FieldAnnotationsScanner(),
                new TypeAnnotationsScanner(),
                new MethodParameterScanner())
        .setExpandSuperTypes(false)
        .filterInputsBy(new FilterBuilder().includePackage("org.zstack"));
Reflections reflections = new Reflections(builder);

Where is root cause

But after a solve the problem by add some hack for reflections, I still want to known what is the root cause.

So I check the code again try to find out what happened.

Compare scanners between 0.10.2 and 0.9.10

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// SubTypes
// reflections 0.10.2
entries.add(this.entry(classFile.getSuperclass(), classFile.getName()));
entries.addAll(this.entries(Arrays.asList(classFile.getInterfaces()), classFile.getName()));

// reflections 0.9.10
filterResultsBy(new FilterBuilder().exclude(Object.class.getName())); //exclude direct Object subtypes

String className = getMetadataAdapter().getClassName(cls);
String superclass = getMetadataAdapter().getSuperclassName(cls);

if (acceptResult(superclass)) {
    getStore().put(superclass, className);
}

for (String anInterface : (List<String>) getMetadataAdapter().getInterfacesNames(cls)) {
		if (acceptResult(anInterface)) {
        getStore().put(anInterface, className);
    }
}

0.9.10 will filter the superclass before put it into reflections store but 0.10.2 use it directly.

in 0.9.10 scan works like following:

1
2
3
4
5
6
7
8
9
10
11
12
13
if (inputsFilter == null || inputsFilter.apply(path) || inputsFilter.apply(fqn)) {
    Object classObject = null;
    for (Scanner scanner : configuration.getScanners()) {
        try {
            if (scanner.acceptsInput(path) || scanner.acceptResult(fqn)) {
                classObject = scanner.scan(file, classObject);
            }
        } catch (Exception e) {
            if (log != null && log.isDebugEnabled())
                log.debug("could not scan file " + file.getRelativePath() + " in url " + url.toExternalForm() + " with scanner " + scanner.getClass().getSimpleName(), e.getMessage());
        }
    }
}

scanner will check fqn of class first and then check the interface but in 0.10.2 version:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
for (Scanner scanner : configuration.getScanners()) {
    try {
        if (doFilter(file, scanner::acceptsInput)) {
            List<Map.Entry<String, String>> entries = scanner.scan(file);
            if (entries == null) {
                if (classFile == null) classFile = getClassFile(file);
                entries = scanner.scan(classFile);
            }
            if (entries != null) collect.get(scanner.index()).addAll(entries);
        }
    } catch (Exception e) {
        if (log != null) log.trace("could not scan file {} with scanner {}", file.getRelativePath(), scanner.getClass().getSimpleName(), e);
    }
}

File will be checked before scan by Filter but superclass’s belonging is not checked which seems to blame.

But actually, groovy closure does not extend Object but GroovyObject, so reflections will still load groovy closures. So check reflections getAllTypes()

for 0.9.10

1
2
3
4
5
6
7
8
public Set<String> getAllTypes() {
    Set<String> allTypes = Sets.newHashSet(store.getAll(index(SubTypesScanner.class), Object.class.getName()));
    if (allTypes.isEmpty()) {
        throw new ReflectionsException("Couldn't find subtypes of Object. " +
                "Make sure SubTypesScanner initialized to include Object class - new SubTypesScanner(false)");
    }
    return allTypes;
}

for 0.10.2

1
2
3
4
public Set<String> getAll(Scanner scanner) {
    Map<String, Set<String>> map = store.getOrDefault(scanner.index(), Collections.emptyMap());
    return Stream.concat(map.keySet().stream(), map.values().stream().flatMap(Collection::stream)).collect(Collectors.toCollection(LinkedHashSet::new));
}

All most same but in 0.9.10 Object related types will be returned. So as a result, only java objects is returned.

Work around on 0.10.2 can use following codes:

1
2
3
4
5
6
7
8
9
10
ConfigurationBuilder builder = ConfigurationBuilder.build()
        .setUrls(ClasspathHelper.forPackage("org.zstack"))
        .setScanners(new SubTypesScanner(false),
                new MethodAnnotationsScanner(),
                new FieldAnnotationsScanner(),
                new TypeAnnotationsScanner(),
                new MethodParameterScanner())
        .setExpandSuperTypes(false)
        .filterInputsBy(new FilterBuilder().includePackage("org.zstack"));
Reflections reflections = new Reflections(builder);

filterInputsBy will filter class not in package org.zstack and setExpandSuperTypes will ignore super types of scanned result.

Note: but the result still contains groovy closure even its count cut down to acceptable numbers.