设置
我正试图用bde2020提供的图像构建一个轻量级的最小hadoop堆栈(学习目的)。现在,堆栈包括
名称节点
数据注解
色调
基本上,我从大数据欧洲官方docker compose开始,并根据他们的文档添加了一个色调图像
问题
hue的文件浏览器无法访问hdfs:
Cannot access: /user/dav. The HDFS REST service is not available. Note: you are a Hue admin but not a HDFS superuser, "hdfs" or part of HDFS supergroup, "supergroup".
HTTPConnectionPool(host='namenode', port=50070): Max retries exceeded with url: /webhdfs/v1/user/dav?op=GETFILESTATUS&user.name=hue&doas=dav (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f8119a3cf10>: Failed to establish a new connection: [Errno 111] Connection refused',))
我迄今为止试图界定的问题
将所有服务显式放置在同一网络上
指向dfs\u webhdfs\u url localhost:9870/webhdfs/v1
在namenode env文件(源)中 hue.ini
在色调的容器中相应地(通过添加 webhdfs_url=http://namenode:9870/webhdfs/v1
)
当我登录到hue的容器时,我可以看到namenode的端口9870是打开的( nmap -p 9870 namenode
). 50070不是。我不认为我的问题与网络有关。尽管编辑 hue.ini
,还是去50070港。那么,在我当前的设置中,如何强制hue去端口9870呢(如果这是原因)
Docker
version: '3.7'
services:
namenode:
image: bde2020/hadoop-namenode:2.0.0-hadoop3.1.1-java8
container_name: namenode
hostname: namenode
domainname: hadoop
ports:
- 9870:9870
volumes:
- hadoop_namenode:/hadoop/dfs/name
- ./entrypoints/namenode/entrypoint.sh:/entrypoint.sh
env_file:
- ./hadoop.env
- .env
networks:
- hadoop_net
# TODO adduser --ingroup hadoop dav
datanode1:
image: bde2020/hadoop-datanode:2.0.0-hadoop3.1.1-java8
container_name: datanode
hostname: datanode1
domainname: hadoop
volumes:
- hadoop_datanode:/hadoop/dfs/data
environment:
SERVICE_PRECONDITION: "namenode:9870"
env_file:
- ./hadoop.env
networks:
- hadoop_net
resourcemanager:
image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.1.1-java8
container_name: resourcemanager
environment:
SERVICE_PRECONDITION: "namenode:9870 datanode:9864"
env_file:
- ./hadoop.env
networks:
- hadoop_net
nodemanager1:
image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.1.1-java8
container_name: nodemanager
environment:
SERVICE_PRECONDITION: "namenode:9870 datanode:9864 resourcemanager:8088"
env_file:
- ./hadoop.env
networks:
- hadoop_net
historyserver:
image: bde2020/hadoop-historyserver:2.0.0-hadoop3.1.1-java8
container_name: historyserver
environment:
SERVICE_PRECONDITION: "namenode:9870 datanode:9864 resourcemanager:8088"
volumes:
- hadoop_historyserver:/hadoop/yarn/timeline
env_file:
- ./hadoop.env
networks:
- hadoop_net
filebrowser:
container_name: hue
image: bde2020/hdfs-filebrowser:3.11
ports:
- "8088:8088"
env_file:
- ./hadoop.env
volumes: # BYPASS DEFAULT webhdfs url
- ./overrides/hue/hue.ini:/opt/hue/desktop/conf.dist/hue.ini
environment:
- NAMENODE_HOST=namenode
networks:
- hadoop_net
networks:
hadoop_net:
volumes:
hadoop_namenode:
hadoop_datanode:
hadoop_historyserver:
2条答案
按热度按时间hvvq6cgz1#
是的,找到了。几个关键要素:
在hadoop3.*中,webhdfs不再监听50070,而9870是标准端口
覆盖
hue.ini
需要挂载一个名为hue-overrides.ini
来自gethue的色调图像比来自bde2020的色调图像更新得多(不过,他们的hadoop堆栈非常棒)Docker
hadoop.env文件
色调覆盖.ini
谢谢@cricket\u 007
pkmbmrz72#
我可以让文件浏览器和这个ini一起工作
这首曲子
CORE_CONF_fs_defaultFS=hdfs://namenode:8020
CORE_CONF_hadoop_http_staticuser_user=root
CORE_CONF_hadoop_proxyuser_hue_hosts=*
CORE_CONF_hadoop_proxyuser_hue_groups=*
HDFS_CONF_dfs_replication=1
HDFS_CONF_dfs_webhdfs_enabled=true
HDFS_CONF_dfs_permissions_enabled=false