By default, access to the HTTP-based services and UI’s for the cluster are not configured to require authentication.
If we want to secure the Hadoop components webUIs (e.g. Namenode UI, JobHistory UI, Yarn ResourceManager UI etc…) then we need to configure kerberos authentication for the all required hadoop Web UIs.
For example now without spnego you can access NN UI without any TGT or authentication.
I have created this article to secure your web UIs by configuring SPNEGO Authentication. Please follow the below steps and secure your UIs.
Step 1: First create a secret key file anywhere on your server used for signing authentication tokens. This file should contain random data and be placed on every host in the cluster. It should also be owned by the hdfs user and group owned by the hadoop group. Permissions should be set to 440. For example:
[root@m1 ~]# vi http_secret
[root@m1 ~]# cat http_secret
dd if=/dev/urandom of=/etc/security/http_secret bs=1024 count=1
[root@m1 ~]# chown hdfs:hadoop /etc/security/http_secret
[root@m1 ~]# chmod 440 /etc/security/http_secret
Step 2: Place the file in Ambari resources dir and restart ambari-server so it gets pushed to all nodes
[root@m1 ~]# cp http_secret /var/lib/ambari-server/resources/host_scripts/
[root@m1 ~]# ambari-server restart
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources…
Ambari database consistency check started…
No errors were found.
Ambari database consistency check finished
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start………………..
Ambari Server ‘start’ completed successfully.
Step 3: Please wait 30 seconds for the http_secret file to get pushed to all nodes under /var/lib/ambari-agent/cache/host_scripts and then on non-Ambari nodes, once the above file is available, run below to put it in right dir and correct its permissions
[root@m1 ~]# cp /var/lib/ambari-agent/cache/host_scripts/http_secret /etc/security/
[root@m1 ~]# chown hdfs:hadoop /etc/security/http_secret
[root@m1 ~]# chmod 440 /etc/security/http_secret
Step 4: Add or modify the following configuration properties to Custom core-site
hadoop.http.authentication.simple.anonymous.allowed=false
hadoop.http.authentication.signature.secret.file=/etc/security/http_secret
hadoop.http.authentication.type=kerberos
hadoop.http.authentication.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab
hadoop.http.authentication.kerberos.principal=HTTP/_HOST@HADOOPADMIN.COM
hadoop.http.filter.initializers=org.apache.hadoop.security.AuthenticationFilterInitializer
hadoop.http.authentication.cookie.domain=HADOOPADMIN.COM
Step 5: Save the configuration, then restart the affected services
Now when you will try to open any of the web UIs like below you will get 401: Authentication required
HDFS: Namenode UI
Mapreduce: Job history UI
YARN: Resource Manager UI
[root@m1 ~]# curl -i “http://m1.hdp22:50070/webhdfs/v1/tmp?user.name=saurkuma&op=GETFILESTATUS”
HTTP/1.1 401 Authentication required
Cache-Control: must-revalidate,no-cache,no-store
Date: Wed, 01 Mar 2017 13:47:29 GMT
Pragma: no-cache
Date: Wed, 01 Mar 2017 13:47:29 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
X-FRAME-OPTIONS: SAMEORIGIN
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=; Path=/; HttpOnly
Content-Length: 1407
Server: Jetty(6.1.26.hwx)
Solution:
Enable Spnego in firefox : To enable SPNEGO, the URL in question must be whitelisted using the Firefox configuration setting network.negotiate-auth.trusted-uris. This takes as its value a comma-separated list of URL prefixes and/or domain names. Suitable values in this instance would be www.example.com or http://www.example.com/.
Step 1: Navigate to the URL and type about:config in browser then click past the warning of harmful consequences.
Step 2: Type negotiate-auth into the filter at the top of the page, in order to remove most of the irrelevant settings from the list. Then double-click on network.negotiate-auth.trusted-uris. A dialogue box for editing the value should appear.
Enter the required hostname(s) and/or URL prefix(es) then click OK.
[root@m1 ~]# curl -i –negotiate -u: “http://m1.hdp22:50070/webhdfs/v1/?op=LISTSTATUS”
HTTP/1.1 401 Authentication required
Cache-Control: must-revalidate,no-cache,no-store
Date: Thu, 02 Mar 2017 09:20:11 GMT
Pragma: no-cache
Date: Thu, 02 Mar 2017 09:20:11 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
X-FRAME-OPTIONS: SAMEORIGIN
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=; Path=/; HttpOnly
Content-Length: 1404
Server: Jetty(6.1.26.hwx)
HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Thu, 02 Mar 2017 09:20:11 GMT
Date: Thu, 02 Mar 2017 09:20:11 GMT
Pragma: no-cache
Expires: Thu, 02 Mar 2017 09:20:11 GMT
Date: Thu, 02 Mar 2017 09:20:11 GMT
Pragma: no-cache
Content-Type: application/json
X-FRAME-OPTIONS: SAMEORIGIN
Set-Cookie: hadoop.auth=”u=root&p=root@HADOOPADMIN.COM&t=kerberos&e=1488482411979&s=R3lcDka3CXvrJ0+wcJ97ofd4pso=”; Path=/; HttpOnly
Transfer-Encoding: chunked
Server: Jetty(6.1.26.hwx)
{“FileStatuses”:{“FileStatus”:[
{“accessTime”:0,”blockSize”:0,”childrenNum”:8,”fileId”:16389,”group”:”hadoop”,”length”:0,”modificationTime”:1486727731946,”owner”:”yarn”,”pathSuffix”:”app-logs”,”permission”:”777″,”replication”:0,”storagePolicy”:0,”type”:”DIRECTORY”},
I hope this article helped you to secure your cluster’s UIs , please feel free to give your valuable suggestion or feedback.