JDK8 和 JDK17 下基于JDBC连接Kerberos认证的Hive(代码已测试通过)

0.背景

之前自研平台是基于jdk8开发的,连接带Kerberos的hive也是jdk8,现在想升级jdk到17,发现过Kerberos的hive有点不一样,特地记录

1. 连接Kerberos,krb5.conf 和对应服务的keytab文件以及对应的principal肯定是需要提前准备的, - 一般从服务器或者运维那里获取krb5.conf 与 Hive对应的keytab文件。
2. 注意获取前先在服务器上用kinit -kt keytab_file_path principal命令检验一下 keytab文件和principal,确认: - Hive开启了Kerberos认证- keytab文件/principal/krb5.conf 是正确无误的/匹配的!
3. 后文如果出现xxx类(UserGroupInformation/metrics2)找不到优先考虑hive的jdbc jar包问题,我推荐https://github.com/timveil/hive-jdbc-uber-jar 这个

想直接看源码?👉 https://github.com/mizuhokaga/kerberos-hive-jdb

1. JDK8 使用JDBC 连接 Kerberos认证的Hive

根据自身实际情况添加依赖

pom.xml

<properties><hive.version>2.1.1</hive.version></properties><dependencies><dependency><groupId>org.apache.hive</groupId><artifactId>hive-jdbc</artifactId><version>${hive.version}</version></dependency></dependencies>

HiveJdbcByJdk8.java

我在jdk8上连接并没有花太多时间,网上大多数JDK8过Kerberos代码也是这样的:

第一步设置系统属性,配置

krb5.conf

,
第二步通过

UserGroupInformation

设置principal 和 对应的keytab文件

由于我们配置了

System.setProperty(“sun.security.krb5.debug”, “true”);

打开了debug后,如果是正常情况,我们能观察到控制台会打印到krb5.conf相关内容信息
出现问题优先根据debug日志查看,如果没有打印出krb5文件内容,去看看

相关文件的路径是否存在

或者是

文件权限程序无法访问(建议测试时调777)

importorg.apache.hadoop.security.UserGroupInformation;importorg.apache.hadoop.conf.Configuration;importjava.io.IOException;importjava.sql.*;publicclassHiveJdbcByJdk8{publicstaticvoidmain(String[] args)throwsClassNotFoundException,IOException{// 1.加载Kerberos配置文件,必须写在Configuration对象之前System.setProperty("sun.security.krb5.debug","true");System.setProperty("java.security.krb5.conf","/home/guzhenzhen/yf-kerberos/test/krb5.conf");// 2.设置Kerberos认证Configuration configuration =newConfiguration();
        configuration.set("hadoop.security.authentication","Kerberos");UserGroupInformation.setConfiguration(configuration);UserGroupInformation.loginUserFromKeytab("hive/admin","/home/guzhenzhen/yf-kerberos/test/hive.keytab");// 3.JDBC连接字符串String jdbcURL ="jdbc:hive2://yfashmd02.yfco.yanfengco.com:10000/test_jt;principal=hive/[email protected]";Class.forName("org.apache.hive.jdbc.HiveDriver");try{// 4.创建Hive连接Connection connection =DriverManager.getConnection(jdbcURL,"","");// 5.执行Hive查询Statement statement = connection.createStatement();ResultSet rs = statement.executeQuery("SELECT id,name,age FROM student");// 6.处理查询结果while(rs.next()){System.out.println(rs.getInt(1)+","+ rs.getString(2)+","+ rs.getInt(3));}// 7.关闭连接
            rs.close();
            statement.close();
            connection.close();}catch(SQLException e){
            e.printStackTrace();}}}

如果出现了

jdk8报错:Message stream modified (41)报错

需要注释krb5.conf 的renew_lifetime属性,解决问题来源https://bbs.huaweicloud.com/forum/thread-72437-1-1.html
下面是krb5.conf参考

[libdefaults]#    default_realm = EXAMPLE.COM 
    default_realm = AUTOEXPR.COM
    dns_lookup_realm =false
    dns_lookup_kdc =false
    ticket_lifetime = 24h
#   jdk8报错:Message stream modified (41)报错,需要注释renew_lifetime#   https://bbs.huaweicloud.com/forum/thread-72437-1-1.html#    renew_lifetime = 7d
    allow_weak_crypto =true
    forwardable =true
    default_ccache_name = FILE:/tmp/krb5cc_%{uid}[realms]...
[domain_realm]...
[logging]...

下面出现的问题

Peer indicated failure: Unsupported mechanism type PLAIN

java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.UserGroupInformation

均可以通过搜索+更换jar包依赖替换解决问题,

2. JDK17 使用JDBC 连接 Kerberos认证的Hive

pom.xml

pom.xml不变

<properties><hive.version>2.1.1</hive.version></properties><dependencies><dependency><groupId>org.apache.hive</groupId><artifactId>hive-jdbc</artifactId><version>${hive.version}</version></dependency></dependencies>

第一种 HiveJdbcByJdk17

importorg.apache.hadoop.conf.Configuration;importorg.apache.hadoop.security.UserGroupInformation;importjava.io.IOException;importjava.sql.*;publicclassHiveJdbcByJdk17{publicstaticvoidmain(String[] args)throwsClassNotFoundException,IOException{// 1.加载Kerberos配置文件//        System.setProperty("java.security.auth.login.config", "/home/guzhenzhen/yf-kerberos/test/gss-jaas.conf");//        System.setProperty("sun.security.jgss.debug", "true");//        System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");System.setProperty("java.security.krb5.conf","/home/guzhenzhen/yf-kerberos/test/krb5.conf");System.setProperty("sun.security.krb5.debug","true");// 2.设置Kerberos认证Configuration configuration =newConfiguration();
        configuration.set("hadoop.security.authentication","Kerberos");UserGroupInformation.setConfiguration(configuration);UserGroupInformation.loginUserFromKeytab("hive/admin","/home/guzhenzhen/yf-kerberos/test/hive.keytab");// 3.JDBC连接字符串String jdbcURL ="jdbc:hive2://yfashmd02.yfco.yanfengco.com:10000/test_jt;principal=hive/[email protected]";Class.forName("org.apache.hive.jdbc.HiveDriver");try{// 4.创建Hive连接Connection connection =DriverManager.getConnection(jdbcURL,"","");//             5.执行Hive查询Statement statement = connection.createStatement();ResultSet rs = statement.executeQuery("SELECT id,name,age FROM student");// 6.处理查询结果while(rs.next()){System.out.println(rs.getInt(1)+","+ rs.getString(2)+","+ rs.getInt(3));}// 7.关闭连接
            rs.close();
            statement.close();

            connection.close();}catch(SQLException e){
            e.printStackTrace();}}}

代码本质和jdk8一模一样,但是会报错

Caused by: java.lang.IllegalAccessException: class org.apache.hadoop.security.authentication.util.KerberosUtil
cannot access class sun.security.krb5.Config (in module java.security.jgss)
because module java.security.jgss does not export sun.security.krb5 to unnamed module @770c2e6b

报错信息;
KerberosUtil 无法访问类sun.security.krb5.Config (in module java.security.jgss) 因为

java.security.jgss

没有向未命名模块到出

java.security.jgss

官网文档写了sun.security.krb5.Config 包位于java8中的rt.jar,从java9之后rt.jar and tools.jar 了
在Java 9及之后引入的模块系统中，模块必须显式声明它们要导出的包，以便其他模块可以访问。

一个简单的解决办法,添加JVM启动参数来临时解决这个问题(来源https://www.baeldung.com/spring-security-kerberos-integration):

–add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED

在idea中调试中,我们
Run/Debug Configurations ->Modify options -> add VM options,在VM options的input框填入上述参数即可,具体参考图片

若是jar包启动,则是

java --add-exports=java.security.jgss/sun.security.krb5=ALL UNNAMED -jar yourJarName.jar

第二种 HiveJdbcByJdk17

上面的代码集成到公司自研平台的项目代码我发现会报错:

Missing artifact jdk.tools:jdk.tools:jar:1.6 

和

Missing artifact jdk.tools:jdk.tools:jar:1.7

查询发现是引入hive-jdbc 依赖时,hive-jdbc底层一些依赖需要用到,按网上的说法手动安装jar 到maven 本地仓库之后引入仍然不行,遂放弃引入 gss-jass.conf

通过配置

gss-jaas.conf 

,这样就不用导入

UserGroupInformation

和

Configuration

了,

当然VM options

--add-exports=java.security.jgss/sun.security.krb5=ALL UNNAMED

还是要的

importjava.io.IOException;importjava.sql.*;publicclassHiveJdbcByJdk17{publicstaticvoidmain(String[] args)throwsClassNotFoundException,IOException{// 1.加载Kerberos配置文件System.setProperty("java.security.auth.login.config","/home/guzhenzhen/yf-kerberos/test/gss-jaas.conf");System.setProperty("sun.security.jgss.debug","true");System.setProperty("javax.security.auth.useSubjectCredsOnly","false");System.setProperty("java.security.krb5.conf","/home/guzhenzhen/yf-kerberos/test/krb5.conf");System.setProperty("sun.security.krb5.debug","true");// 2.设置Kerberos认证//        Configuration configuration = new Configuration();//        configuration.set("hadoop.security.authentication", "Kerberos");//        UserGroupInformation.setConfiguration(configuration);//        UserGroupInformation.loginUserFromKeytab("hive/admin", "/home/guzhenzhen/yf-kerberos/test/hive.keytab");// 3.JDBC连接字符串String jdbcURL ="jdbc:hive2://yfashmd02.yfco.yanfengco.com:10000/test_jt;principal=hive/[email protected]";Class.forName("org.apache.hive.jdbc.HiveDriver");try{// 4.创建Hive连接Connection connection =DriverManager.getConnection(jdbcURL,"","");//             5.执行Hive查询Statement statement = connection.createStatement();ResultSet rs = statement.executeQuery("SELECT id,name,age FROM student");// 6.处理查询结果while(rs.next()){System.out.println(rs.getInt(1)+","+ rs.getString(2)+","+ rs.getInt(3));}// 7.关闭连接
            rs.close();
            statement.close();

            connection.close();}catch(SQLException e){
            e.printStackTrace();}}}

gss-jaas.conf

参考

com.sun.security.jgss.initiate{
   com.sun.security.auth.module.Krb5LoginModule required
   doNotPrompt=true
   useTicketCache=true
   useKeyTab=true
   renewTGT=true
   debug=true
   ticketCache="/tmp/krb5cc_1000"keyTab="/home/guzhenzhen/yf-kerberos/test/hive.keytab"principal="hive/[email protected]";};