Many workloads do not work in CDH6.0.0( Hadoop3 )

[godtree]

I find that the HiBench is supported the CDH5.
I test the HiBench in CDH6.0.0 .
But CDH 6 is based on Apache Hadoop 3.
And many workloads don't work.
Such as enhanced DFSIO, Bayesian Classification, K-means clustering, Scan, Join, Aggregate, Nutch indexing, and so on.

The Errors show like this:
Exception in thread "main" java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator

When will support CDH6？When will support Hadoop 3？

[gczsjdy]

We may only support Hadoop 3.x after #586 is merged. Also, there may be a few changes(or none) you may need to do on your own to run on Spark 3.0/Spark 3.1(as that one is for Hadoop 3.2).

As for CDH 6.0, similar answer, we currently don't have plans to officially support it. But I suppose the code changes needed should not be too much.

[zhizhunbao]

i have the same issue

[fuzhenhe]

Hi, Sir

I have a question that about Hadoop version-hadoop-3.3, because I noticed that the Hadoop ARM version is already supported on the Apache website. Do you have plan working on hadoop-3.3.0-aarch64, many thanks! zfu from ampere

[jasonwzs]

Hi @gczsjdy , does the latest hibench release or trunk support any Hadoop 3.x version? I see the README says it support following:
Hadoop: Apache Hadoop 3.0.x, Apache Hadoop 3.1.x, Apache Hadoop 3.2.x, Apache Hadoop 2.x, CDH5, HDP
However #586 is not merged yet. Does it mean it still has some problem support Hadoop 3.x?
Thanks!

[gczsjdy]

@jasonwzs , Hadoop 3.x(until 3.3.0, exclusive) is supported, please try out. #586 was the old story. : )

[gczsjdy]

@fuzhenhe , I am not sure if the ARM part is the problem, I suppose JVM will take care of the ARM part?
Hadoop 3.0 is the version we haven't officially support.
If you have tried that out yourself, it's welcome if you left some feedback.