Hadoop依赖打包jar-with-dependencies的包冲突问题

前言

最近在写一个Java项目,需要访问HDFS并执行一些操作。为了部署方便,使用Maven的assembly插件打包成一个可执行包。
在本地用IDEA调试时,各个功能一切正常,但是当打包成jar包,丢到服务器上执行,却出问题了。

问题出现

将打包好的jar包放到服务器上执行,出现如下报错信息:

org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3372)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3392)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3443)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3411)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
    at com.sinovatio.bigdata.datacleaner.db.hive.HiveCleaner.init(HiveCleaner.java:121)
    at com.sinovatio.bigdata.datacleaner.db.hive.HiveCleaner.<init>(HiveCleaner.java:75)
    at com.sinovatio.bigdata.datacleaner.core.ScheduleTask.execute(ScheduleTask.java:76)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:207)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:560)

问题定位

查阅网上相关信息,定位到问题所在。

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>3.1.1</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs-client</artifactId>
    <version>3.1.1</version>
</dependency>

项目中同时引用了这两个包,这两个包存在一个同名的service文件。在使用assembly插件打包时,同名文件默认是不覆盖原则,即先加载哪个包,service里面就会保留哪个文件。但这并不是我们想要的结果,"hdfs"的schema文件系统说明在hadoop-hdfs-client中,这就导致无法找到"hdfs"的schema(org.apache.hadoop.hdfs.DistributedFileSystem)。
- hadoop-common中 org.apache.hadoop.fs.FileSystem文件内容如下

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
org.apache.hadoop.fs.LocalFileSystem
org.apache.hadoop.fs.viewfs.ViewFileSystem
org.apache.hadoop.fs.HarFileSystem
org.apache.hadoop.fs.http.HttpFileSystem
org.apache.hadoop.fs.http.HttpsFileSystem
  • hadoop-hdfs-client中org.apache.hadoop.fs.FileSystem文件内容如下
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
org.apache.hadoop.hdfs.DistributedFileSystem
org.apache.hadoop.hdfs.web.WebHdfsFileSystem
org.apache.hadoop.hdfs.web.SWebHdfsFileSystem

问题解决

找到问题原因了,接下来就可以准备解决这个问题了,如下方法都可以解决这个问题。

业务代码中指定对应schema

原始业务代码如下

    val conf = new Configuration();
    conf.addResource(new Path("./core-site.xml"));
    conf.addResource(new Path("./hdfs-site.xml"));
    val hdfs = FileSystem.get(conf);

将其改为如下

    val conf = new Configuration();
    conf.addResource(new Path("./core-site.xml"));
    conf.addResource(new Path("./hdfs-site.xml"));
    // 指定"hdfs"对应的文件系统
    conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfsDistributedFileSystem");
    val hdfs = FileSystem.get(conf);

手动合并文件并替换jar包内容

jar包打包好后,用解压软件打开jar包,找到对应的org.apache.hadoop.fs.FileSystem文件,修改为如下

org.apache.hadoop.fs.LocalFileSystem
org.apache.hadoop.fs.viewfs.ViewFileSystem
org.apache.hadoop.fs.HarFileSystem
org.apache.hadoop.fs.http.HttpFileSystem
org.apache.hadoop.fs.http.HttpsFileSystem
org.apache.hadoop.hdfs.DistributedFileSystem
org.apache.hadoop.hdfs.web.WebHdfsFileSystem
org.apache.hadoop.hdfs.web.SWebHdfsFileSystem

使用assembly插件的文件合并功能(推荐)

根据官方jar-with-dependencies配置,自定义一个配置文件,增加文件合并的配置。文件内容如下

<assembly>
    <id>jar-with-dependencies</id>
    <formats>
        <format>jar</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <!-- 新增加的文件合并配置 -->
    <containerDescriptorHandlers>
        <containerDescriptorHandler>
            <handlerName>metaInf-services</handlerName>
        </containerDescriptorHandler>
    </containerDescriptorHandlers>
    <dependencySets>
        <dependencySet>
            <outputDirectory>/</outputDirectory>
            <useProjectArtifact>true</useProjectArtifact>
            <unpack>true</unpack>
            <scope>runtime</scope>
        </dependencySet>
    </dependencySets>
</assembly>

对应的assembly配置如下

   <plugin>
       <artifactId>maven-assembly-plugin</artifactId>
       <executions>
           <execution>
               <id>assembly</id>
               <phase>package</phase>
               <configuration>
                   <descriptors>
                       <descriptor>jar-with-dependencies.xml</descriptor>
                   </descriptors>
               </configuration>
            </execution>
       </executions>
   </plugin>
暂无评论

发送评论 编辑评论


				
|´・ω・)ノ
ヾ(≧∇≦*)ゝ
(☆ω☆)
(╯‵□′)╯︵┴─┴
 ̄﹃ ̄
(/ω\)
∠( ᐛ 」∠)_
(๑•̀ㅁ•́ฅ)
→_→
୧(๑•̀⌄•́๑)૭
٩(ˊᗜˋ*)و
(ノ°ο°)ノ
(´இ皿இ`)
⌇●﹏●⌇
(ฅ´ω`ฅ)
(╯°A°)╯︵○○○
φ( ̄∇ ̄o)
ヾ(´・ ・`。)ノ"
( ง ᵒ̌皿ᵒ̌)ง⁼³₌₃
(ó﹏ò。)
Σ(っ °Д °;)っ
( ,,´・ω・)ノ"(´っω・`。)
╮(╯▽╰)╭
o(*////▽////*)q
>﹏<
( ๑´•ω•) "(ㆆᴗㆆ)
😂
😀
😅
😊
🙂
🙃
😌
😍
😘
😜
😝
😏
😒
🙄
😳
😡
😔
😫
😱
😭
💩
👻
🙌
🖕
👍
👫
👬
👭
🌚
🌝
🙈
💊
😶
🙏
🍦
🍉
😣
Source: github.com/k4yt3x/flowerhd
颜文字
Emoji
小恐龙
花!
上一篇
下一篇