if you are seeing issues with kerberos based hdfs client access against an Isilon cluster, increasing the kerberos logging level on the client can show you a lot more information(a lot)
Change the logging level with the following:
export HADOOP_ROOT_LOGGER=”TRACE,console”
export HADOOP_OPTS=”-Dsun.security.krb5.debug=true”
quick example of a successful kerberized hadoop call from a kerberized hortoworks cluster against Isilon:
[root@hdp4 jvm]# su – hdpuser3
[hdpuser3@hdp4 ~]$ kinit
Password for hdpuser3@FOO.COM:
[hdpuser3@hdp4 ~]$ klist -e
Ticket cache: FILE:/tmp/krb5cc_49003
Default principal: hdpuser3@FOO.COM
Valid starting Expires Service principal
04/22/16 12:18:40 04/22/16 22:17:00 krbtgt/FOO.COM@FOO.COM
renew until 04/29/16 12:18:40, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
[hdpuser3@hdp4 ~]$ export HADOOP_ROOT_LOGGER=”TRACE,console”
[hdpuser3@hdp4 ~]$ export HADOOP_OPTS=”-Dsun.security.krb5.debug=true”
[hdpuser3@hdp4 ~]$ hadoop fs -ls /
16/04/22 12:13:50 DEBUG util.Shell: setsid exited with exit code 0
16/04/22 12:13:51 DEBUG conf.Configuration: parsing URL jar:file:/usr/hdp/2.3.4.0-3485/hadoop/hadoop-common-2.7.1.2.3.4.0-3485.jar!/core-default.xml
16/04/22 12:13:51 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@7cbd213e
16/04/22 12:13:51 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/2.3.4.0-3485/0/core-site.xml
16/04/22 12:13:51 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@5cdd8682
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
16/04/22 12:13:52 DEBUG security.Groups: Creating new Groups object
16/04/22 12:13:52 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library…
16/04/22 12:13:52 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
16/04/22 12:13:52 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
16/04/22 12:13:52 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
16/04/22 12:13:52 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
>>>KinitOptions cache name is /tmp/krb5cc_49003
>>>DEBUG <CCacheInputStream> client principal is hdpuser3@FOO.COM
>>>DEBUG <CCacheInputStream> server principal is krbtgt/FOO.COM@FOO.COM
>>>DEBUG <CCacheInputStream> key type: 18
>>>DEBUG <CCacheInputStream> auth time: Fri Apr 22 11:56:32 EDT 2016
>>>DEBUG <CCacheInputStream> start time: Fri Apr 22 11:58:12 EDT 2016
>>>DEBUG <CCacheInputStream> end time: Fri Apr 22 21:56:32 EDT 2016
>>>DEBUG <CCacheInputStream> renew_till time: Fri Apr 29 11:58:12 EDT 2016
>>> CCacheInputStream: readFlags() FORWARDABLE; RENEWABLE; INITIAL; PRE_AUTH;
16/04/22 12:13:52 DEBUG security.UserGroupInformation: hadoop login
16/04/22 12:13:52 DEBUG security.UserGroupInformation: hadoop login commit
16/04/22 12:13:52 DEBUG security.UserGroupInformation: using kerberos user:hdpuser3@FOO.COM
16/04/22 12:13:52 DEBUG security.UserGroupInformation: Using user: “hdpuser3@FOO.COM” with name hdpuser3@FOO.COM
16/04/22 12:13:52 DEBUG security.UserGroupInformation: User entry: “hdpuser3@FOO.COM”
16/04/22 12:13:52 DEBUG security.UserGroupInformation: UGI loginUser:hdpuser3@FOO.COM (auth:KERBEROS)
16/04/22 12:13:52 DEBUG security.UserGroupInformation: Found tgt Ticket (hex) =
0000: 61 82 03 F8 30 82 03 F4 A0 03 02 01 05 A1 09 1B a…0………..
0010: 07 46 4F 4F 2E 43 4F 4D A2 1C 30 1A A0 03 02 01 .FOO.COM..0…..
0020: 02 A1 13 30 11 1B 06 6B 72 62 74 67 74 1B 07 46 …0…krbtgt..F
0030: 4F 4F 2E 43 4F 4D A3 82 03 C2 30 82 03 BE A0 03 OO.COM….0…..
0040: 02 01 12 A1 03 02 01 02 A2 82 03 B0 04 82 03 AC …………….
0050: 80 45 B8 57 D9 53 73 66 A9 20 92 30 46 2E 64 A6 .E.W.Ssf. .0F.d.
0060: 2B 42 0E 1E 71 6D 56 35 88 78 A5 CF 2D 3C 01 24 +B..qmV5.x..-<.$
0070: 49 5D FA 7C 88 31 03 0F 26 9A C1 29 63 30 67 B3 I]…1..&..)c0g.
0080: 20 AC 73 F8 35 D3 08 E9 FD 81 ED 66 C3 41 AD 4B .s.5……f.A.K
0090: 09 B6 DF B1 3B C6 FC 5F 1B A5 B5 FC A9 8A D3 A4 ….;.._……..
00A0: 5F 2B 31 73 0F 03 87 93 43 35 2E 6D 9D 4A 97 C8 _+1s….C5.m.J..
00B0: 3C 5C 45 B2 B0 21 D5 C8 B5 0A 73 10 76 92 AB 99 <\E..!….s.v…
00C0: B7 EE 63 C8 37 94 1B 57 E7 A0 57 7E 84 17 1E 04 ..c.7..W..W…..
00D0: 13 F9 5D 62 D6 0C 02 3D 46 EF 3C AE 27 D6 74 A1 ..]b…=F.<.’.t.
00E0: A9 A2 42 17 3A DA E3 DF F5 0B B0 F8 16 0F 77 59 ..B.:………wY
00F0: 84 61 DF 94 75 39 C8 6C AD 67 20 E2 EC 49 54 FE .a..u9.l.g ..IT.
0100: 14 4C 2E 9C 30 AC 1C 42 37 01 27 A0 C0 7F 7E 32 .L..0..B7.’….2
0110: DC D0 8F 14 36 A4 B7 D4 2B 0A ED DC BD AC 9E BD ….6…+…….
0120: 4A 43 C7 FB 60 B2 30 9F F7 E1 50 88 44 AC 07 06 JC..`.0…P.D…
0130: 4E 1C FE 3A 40 88 EF D3 20 82 7C 39 89 BC CA 4F N..:@… ..9…O
0140: 58 3B FD 3D A5 24 62 C9 75 C3 47 F3 E0 EA 83 B6 X;.=.$b.u.G…..
0150: 0C FA F3 25 8A BC 42 22 59 54 81 5D 25 64 E9 7E …%..B”YT.]%d..
0160: 9E 73 70 DD 24 35 E5 94 3D 71 B6 C2 14 C1 3F 5C .sp.$5..=q….?\
0170: 75 2B 0B 99 4E FC 29 40 8B B7 1D 09 A1 AA 52 41 u+..N.)@……RA
0180: BF 16 91 67 5F 21 53 68 3E 98 65 4E E2 AF FC 7C …g_!Sh>.eN….
0190: 96 70 13 74 FB B5 E5 9F 87 C5 39 49 B4 51 BA 8F .p.t……9I.Q..
01A0: F0 42 4B B4 4C F8 CF D3 5C 69 3C 53 8B 42 E8 F3 .BK.L…\i<S.B..
01B0: CC 71 F6 57 AF 5E 0F 1E 0C 63 49 0A 86 8B A6 00 .q.W.^…cI…..
01C0: 1A AF 8F B8 6E 2C 81 1E 8B B4 49 DF 61 95 77 3B ….n,….I.a.w;
01D0: A6 BB 48 4C A5 ED 03 F2 3D A2 BF 00 27 90 34 5A ..HL….=…’.4Z
01E0: 9E A9 57 D7 45 AB DC 85 71 37 D7 65 27 82 04 41 ..W.E…q7.e’..A
01F0: 5E 5D 14 40 94 AC 95 1E 4C F0 23 B1 94 96 A7 89 ^].@….L.#…..
0200: 03 6C 67 2D C0 08 18 22 2A DE FD DC 03 88 FD 3D .lg-…”*……=
0210: A4 95 C7 31 6C 28 F2 AD 2F C2 21 65 51 03 A9 8F …1l(../.!eQ…
0220: 85 C5 52 07 A3 81 05 E5 7F A4 EE 17 F5 71 C6 AC ..R……….q..
0230: F0 38 F3 EA FF 96 75 95 ED 03 2C 72 4C 3B C6 E1 .8….u…,rL;..
0240: 63 65 13 E6 45 69 F6 91 CD 39 B9 E8 CE F0 A6 96 ce..Ei…9……
0250: C0 AF 4A 43 FB 18 F8 E7 09 26 7C A5 0D 24 FB 7D ..JC…..&…$..
0260: CF C3 64 C4 1E 40 28 D7 D5 48 15 15 AA 60 8C 73 ..d..@(..H…`.s
0270: 6C 0F 36 97 46 B0 BE B6 EE C4 1F 6B 88 F5 1E F2 l.6.F……k….
0280: F8 93 04 B7 9C 1C 21 5D D6 86 4D 45 DE 57 68 49 ……!]..ME.WhI
0290: 85 D1 A2 5F 0A 13 2E 33 DB A2 2E 90 9D 4F A2 E2 …_…3…..O..
02A0: 99 CB 3A BE 4E A1 C2 B3 D5 0F 7A 56 2E BA DB 15 ..:.N…..zV….
02B0: 2D 6F DA 51 B6 E8 B8 E7 19 94 A3 7D 0E B2 31 4F -o.Q……….1O
02C0: E5 21 C4 D3 DC 11 64 DA 7E 37 89 E3 81 1F 24 C0 .!….d..7….$.
02D0: 4B 2C 11 46 A1 06 09 0B 0E 61 3A EA 59 09 83 8B K,.F…..a:.Y…
02E0: A8 6F FC 78 14 41 EB 2F 32 81 AC CA 3B 3F 5E CF .o.x.A./2…;?^.
02F0: 59 1F 48 C1 36 7A A2 82 84 6D D4 36 1F 54 9E 7C Y.H.6z…m.6.T..
0300: 20 DC DA D2 68 19 B8 DF 14 25 25 C0 E9 DC 71 56 …h….%%…qV
0310: 51 73 63 44 86 A5 33 B6 25 0A 57 49 1F 0A 02 4F QscD..3.%.WI…O
0320: 70 93 AC 52 8A BC 7A 7F 10 3C F1 D8 DB CE B7 CF p..R..z..<……
0330: FA 84 FC C0 79 10 5F 25 3D 38 B7 4F D8 EA 4D 51 ….y._%=8.O..MQ
0340: 74 D3 C0 E1 04 D1 6E 34 C1 62 4F 4F 16 86 DE E8 t…..n4.bOO….
0350: B8 5B B1 50 13 05 30 09 59 58 A6 A9 E3 7E 9F 79 .[.P..0.YX…..y
0360: 8D E9 8B 25 D9 0E DB 17 26 FF A1 AD 8B 8A 55 52 …%….&…..UR
0370: 35 9F 3C 9E 56 7B 61 A1 AC 08 E0 37 9D EA 77 D9 5.<.V.a….7..w.
0380: 28 75 3D 39 C2 08 BF E9 8B 4A 8A 60 3A 69 3E 1C (u=9…..J.`:i>.
0390: 6F E7 B9 92 96 F4 79 A5 87 0B 25 B2 11 5B FC ED o…..y…%..[..
03A0: 19 36 D5 35 94 F5 74 90 82 70 CF F8 6A BB F2 80 .6.5..t..p..j…
03B0: FC 37 01 0A 41 AE 52 0C A1 23 2F 62 4B 68 35 FF .7..A.R..#/bKh5.
03C0: 3E DD 12 7E 27 5F 1D AF 66 52 D0 8C 06 60 46 5C >…’_..fR…`F\
03D0: 08 BF EA 5A AD F1 BD 14 50 7A AF 48 1C 48 91 0D …Z….Pz.H.H..
03E0: D8 0E 2E 44 9D 92 6E 0C AA 0F C1 E5 6E 18 69 BB …D..n…..n.i.
03F0: 4F 90 EE F7 22 9F F0 DE F1 E9 69 4D O…”…..iM
Client Principal = hdpuser3@FOO.COM
Server Principal = krbtgt/FOO.COM@FOO.COM
Session Key = EncryptionKey: keyType=18 keyBytes (hex dump)=
0000: 40 19 60 17 41 7F 9B A6 FA 6E D2 93 A6 F0 B3 2B @.`.A….n…..+
0010: 20 BC 93 4D 72 E2 23 75 AF DA CF 5B C7 48 09 C6 ..Mr.#u…[.H..
Forwardable Ticket true
Forwarded Ticket false
Proxiable Ticket false
Proxy Ticket false
Postdated Ticket false
Renewable Ticket true
Initial Ticket true
Auth Time = Fri Apr 22 11:56:32 EDT 2016
Start Time = Fri Apr 22 11:58:12 EDT 2016
End Time = Fri Apr 22 21:56:32 EDT 2016
Renew Till = Fri Apr 29 11:58:12 EDT 2016
Client Addresses Null
16/04/22 12:13:52 DEBUG security.UserGroupInformation: Current time is 1461341632663
16/04/22 12:13:52 DEBUG security.UserGroupInformation: Next refresh is 1461369412000
16/04/22 12:13:52 TRACE tracing.SpanReceiverHost: No span receiver names found in dfs.client.htrace.spanreceiver.classes.
16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = true
16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = /var/lib/hadoop-hdfs/dn_socket
16/04/22 12:13:53 DEBUG retry.RetryUtils: multipleLinearRandomRetry = null
16/04/22 12:13:53 DEBUG ipc.Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@18df8434
16/04/22 12:13:53 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@6c1a5b54
16/04/22 12:13:54 DEBUG unix.DomainSocketWatcher: org.apache.hadoop.net.unix.DomainSocketWatcher$2@2a2a8820: starting with interruptCheckPeriodMs = 60000
16/04/22 12:13:54 TRACE unix.DomainSocketWatcher: DomainSocketWatcher(1426041738): adding notificationSocket 351, connected to 350
16/04/22 12:13:54 DEBUG shortcircuit.DomainSocketFactory: The short-circuit local reads feature is enabled.
16/04/22 12:13:54 DEBUG sasl.DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Call -> rip1-horton2.foo.com/172.16.201.73:8020: getFileInfo {src: “/”}
16/04/22 12:13:54 DEBUG ipc.Client: The ping interval is 60000 ms.
16/04/22 12:13:54 DEBUG ipc.Client: Connecting to rip1-horton2.foo.com/172.16.201.73:8020
16/04/22 12:13:54 DEBUG security.UserGroupInformation: PrivilegedAction as:hdpuser3@FOO.COM (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:722)
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE
16/04/22 12:13:54 DEBUG azure.NativeAzureFileSystem: finalize() called.
16/04/22 12:13:54 DEBUG azure.NativeAzureFileSystem: finalize() called.
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: NEGOTIATE
auths {
method: “TOKEN”
mechanism: “DIGEST-MD5”
protocol: “default”
challenge: “nonce=\”p0L557HD2nzgyagR7F5o3hTweljDX9n6kd7M0YAT368=\”,realm=\”rip1-1\”,qop=\”auth,auth-int,auth-conf\”,cipher=\”rc4-40,rc4-56,rc4,des,3des\”,maxbuf=65536,charset=utf-8,algorithm=md5-sess”
}
auths {
method: “KERBEROS”
mechanism: “GSSAPI”
protocol: “hdfs”
serverId: “rip1.foo.com”
}
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:@org.apache.hadoop.security.token.TokenInfo(value=class org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSelector)
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=, serverPrincipal=dfs.namenode.kerberos.principal)
16/04/22 12:13:54 DEBUG security.SaslRpcClient: RPC Server’s Kerberos principal name for protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is hdfs/rip1.foo.com@FOO.COM
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Creating SASL GSSAPI(KERBEROS) client to authenticate to service at rip1.foo.com
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Use KERBEROS authentication for protocol ClientNamenodeProtocolPB
Found ticket for hdpuser3@FOO.COM to go to krbtgt/FOO.COM@FOO.COM expiring on Fri Apr 22 21:56:32 EDT 2016
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for hdpuser3@FOO.COM to go to krbtgt/FOO.COM@FOO.COM expiring on Fri Apr 22 21:56:32 EDT 2016
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 18 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KdcAccessibility: reset
>>> KrbKdcReq send: kdc=rs-2008r2-dc.foo.com UDP:88, timeout=30000, number of retries =3, #bytes=1336
>>> KDCCommunication: kdc=rs-2008r2-dc.foo.com UDP:88, timeout=30000,Attempt =1, #bytes=1336
>>> KrbKdcReq send: #bytes read=1304
>>> KdcAccessibility: remove rs-2008r2-dc.foo.com
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 634089161
Created InitSecContextToken:
0000: 01 00 6E 82 04 D7 30 82 04 D3 A0 03 02 01 05 A1 ..n…0………
0010: 03 02 01 0E A2 07 03 05 00 20 00 00 00 A3 82 03 ……… ……
0020: EC 61 82 03 E8 30 82 03 E4 A0 03 02 01 05 A1 09 .a…0……….
0030: 1B 07 46 4F 4F 2E 43 4F 4D A2 1F 30 1D A0 03 02 ..FOO.COM..0….
0040: 01 00 A1 16 30 14 1B 04 68 64 66 73 1B 0C 72 69 ….0…hdfs..ri
0050: 70 31 2E 66 6F 6F 2E 63 6F 6D A3 82 03 AF 30 82 p1.foo.com….0.
0060: 03 AB A0 03 02 01 12 A1 03 02 01 04 A2 82 03 9D …………….
0070: 04 82 03 99 DC D0 4F F8 8C 20 BB BC 88 B5 42 24 ……O.. ….B$
0080: E7 FD B3 D5 E3 D7 CA A6 62 8C B7 E6 88 61 A3 B5 ……..b….a..
0090: B6 ED E9 61 FC 19 00 B9 A6 8C 63 26 E6 F5 87 25 …a……c&…%
00A0: 85 B6 0F 31 9B A6 49 8D B2 95 5B 74 39 9C 6C 0A …1..I…[t9.l.
00B0: 2C 6D 1A E7 BC 4B 0F FA 4B DA C0 F7 81 83 C7 95 ,m…K..K…….
00C0: AD A8 66 33 18 30 3E 19 CC AB 7B F5 6D 18 EF CC ..f3.0>…..m…
00D0: AB 05 8B F2 00 1A 3A 8E B3 E0 40 34 AC 6C 63 E8 ……:…@4.lc.
00E0: 9A 9A 62 13 87 81 48 15 97 2A 7F 8D C3 04 2B D9 ..b…H..*….+.
00F0: C5 10 7C 40 13 18 5A 93 1F D4 1B 23 7E 9F 6A 6C …@..Z….#..jl
0100: AE 05 10 FC 4C 52 5F BD E9 4B 19 03 DE 08 AD 22 ….LR_..K…..”
0110: 7F 8E 95 D6 07 D5 65 FA 4A F2 9E 1B E8 08 20 D3 ……e.J….. .
0120: 54 A1 49 98 6C 92 4B B4 31 B5 32 6D 43 A1 0D 56 T.I.l.K.1.2mC..V
0130: 81 92 48 AA F4 4F F4 7D 9E 9C 7B 56 56 AD 61 89 ..H..O…..VV.a.
0140: 70 24 05 6C 75 00 F2 28 45 6A 80 A0 E6 0D 67 45 p$.lu..(Ej….gE
0150: A2 DD F9 AA 90 36 AB 5D 50 9A 58 F7 4A 15 0C 88 …..6.]P.X.J…
0160: 8C FF AB B9 36 1E 40 82 28 7B 88 BF 4C 7C A6 FE ….6.@.(…L…
0170: F1 1C 61 E4 31 8A 5D 51 41 A8 78 B1 E6 36 74 65 ..a.1.]QA.x..6te
0180: D3 24 75 32 7F 42 58 37 B7 93 48 56 93 74 15 3F .$u2.BX7..HV.t.?
0190: FF F5 94 7B 6D 19 15 DF 24 5A D4 75 12 BA BC 13 ….m…$Z.u….
01A0: 9E E7 DB 1C CD 36 90 A2 3B 48 D1 85 01 5B 5C CC …..6..;H…[\.
01B0: 72 B0 37 9E 69 D6 C5 80 2E 12 FD FF 9D 9E 6D 8F r.7.i………m.
01C0: 25 57 E5 E2 75 4E 89 54 8A A7 44 1D 11 8F D8 F5 %W..uN.T..D…..
01D0: C7 19 33 1E FD 15 8B 57 74 EE 53 77 C3 7D A5 3F ..3….Wt.Sw…?
01E0: F7 39 76 2A A7 4F DF E5 88 8B E5 02 D1 E7 AF 7E .9v*.O……….
01F0: 3A 00 7D 83 A9 34 68 5F 1F F7 87 86 D5 D3 9E C0 :….4h_……..
0200: F3 DA 3A C0 7F 46 5B CF 47 79 F0 7B 20 73 68 4D ..:..F[.Gy.. shM
0210: 56 88 3E B4 76 25 FD 47 2C 4B 37 38 15 1A E5 BE V.>.v%.G,K78….
0220: FC AB 48 C3 68 53 F7 C7 A1 D0 1D C2 D5 1A 38 2A ..H.hS……..8*
0230: 4F 4B 71 34 FD 2B C6 69 57 4B 2A D8 32 1C 54 72 OKq4.+.iWK*.2.Tr
0240: CB C7 1B A0 EF D9 7B DE 75 7C AC C5 38 3A 4D 45 ……..u…8:ME
0250: 19 78 2E B5 C1 80 9B F0 A7 F1 AC 54 9B 43 7E 10 .x………T.C..
0260: 06 40 28 F1 76 AF 3E D8 4F 49 91 AF B0 FD A3 1F .@(.v.>.OI……
0270: 4B CF 6D 64 91 78 F9 3F CC 2B 68 B5 E2 8B 8C A5 K.md.x.?.+h…..
0280: 69 AA 29 67 3A 48 0E 4E 93 91 E9 B0 8A FE 00 F5 i.)g:H.N……..
0290: 12 B3 8C 07 FE 36 57 6C 68 D8 78 F5 F8 6B 73 0A …..6Wlh.x..ks.
02A0: 7D 57 E6 ED E2 A4 6A 46 91 75 2C B3 D5 43 D7 81 .W….jF.u,..C..
02B0: 5B 77 D6 92 08 A9 F6 BB B6 57 80 2E D6 37 F8 73 [w…….W…7.s
02C0: BC A3 5C 82 34 2A CD 03 97 74 FB 14 9A 18 B9 2F ..\.4*…t…../
02D0: D4 01 37 83 1F 6A 99 31 7E E5 EE 7A 97 B8 D7 81 ..7..j.1…z….
02E0: 77 04 D5 81 01 09 96 53 97 22 CB 65 48 29 F7 18 w……S.”.eH)..
02F0: DE DA CB 55 3E BC D7 FC BB 0B 49 91 BC 4E 10 21 …U>…..I..N.!
0300: F8 D4 4E D8 17 77 7B B6 BD 41 23 DB 38 5A 59 D9 ..N..w…A#.8ZY.
0310: E9 E8 E4 8E 23 C5 D6 56 3E 88 79 5B F4 24 63 96 ….#..V>.y[.$c.
0320: 19 76 F2 B4 23 97 E3 28 8C 9D F5 CB 37 6A 99 4C .v..#..(….7j.L
0330: DB 3B 23 0D A8 F2 42 33 0C 35 5D D9 EE 5B 75 04 .;#…B3.5]..[u.
0340: A7 D4 94 5F 49 9A A3 06 15 53 E2 2C 21 1F F9 CC …_I….S.,!…
0350: 51 83 58 18 8E CB BD 36 09 50 5B 90 22 8C 53 93 Q.X….6.P[.”.S.
0360: 87 B2 2E A5 55 0E 09 02 B7 5A 4D B7 53 58 19 BF ….U….ZM.SX..
0370: 6C 6F B3 10 F3 E6 59 69 FF 66 C8 56 1A AA 53 22 lo….Yi.f.V..S”
0380: F0 5D B8 22 16 5E AD C0 B0 34 F1 04 9B 64 96 26 .].”.^…4…d.&
0390: 5A 04 7F 0A 9B 30 9E 0E 9B 83 27 83 18 0E E9 2E Z….0….’…..
03A0: 42 67 B3 53 87 78 12 12 A9 14 5A 05 ED 6A 43 41 Bg.S.x….Z..jCA
03B0: BE 4E A9 91 68 6C 1B 89 74 14 C7 86 C2 47 FE 30 .N..hl..t….G.0
03C0: 6C 42 D9 59 8F A6 1A 50 40 07 70 11 DE B2 0E D1 lB.Y…P@.p…..
03D0: CE 3C 8F D1 DD F7 C4 12 CD B8 4B 0A 99 B4 EF BB .<……..K…..
03E0: FB 4A 7B EE 9C E2 37 E0 06 EC 14 13 F1 F9 44 71 .J….7…….Dq
03F0: A3 F6 FE 6D 40 46 B4 4E D2 25 5A E0 5E 9B 1B 28 …m@F.N.%Z.^..(
0400: 9D 91 C1 2E 07 E2 7B CE 45 D2 5B 84 56 A4 81 CD ……..E.[.V…
0410: 30 81 CA A0 03 02 01 12 A2 81 C2 04 81 BF 1A 70 0…………..p
0420: A8 2D 8B 91 A7 12 8F 2C 1A B6 F1 E1 E2 F1 E4 C4 .-…..,……..
0430: F1 D1 14 77 8E 5A 73 DC 42 A9 F4 D5 5A 90 35 9A …w.Zs.B…Z.5.
0440: 95 BD 99 B6 D7 7A 2D F8 3D 36 71 5C E3 11 76 A6 …..z-.=6q\..v.
0450: 0F 62 D8 30 8F E9 B1 58 EE 90 31 ED FF 3A BF 53 .b.0…X..1..:.S
0460: FA BE 45 AE A3 E6 9D 30 58 39 8E C3 11 B5 63 F0 ..E….0X9….c.
0470: 12 A7 B6 BC FA D9 75 D9 35 0A 81 AB 53 89 75 4C ……u.5…S.uL
0480: EF 15 90 3D 4C F0 C5 D3 D2 A2 C8 0E 08 F7 0C 42 …=L……….B
0490: C3 91 64 F2 2B 35 59 02 34 80 64 44 38 3F 52 C7 ..d.+5Y.4.dD8?R.
04A0: 3C ED 01 07 6C FC 60 42 A4 8D D7 FD D0 83 31 FA <…l.`B……1.
04B0: 54 57 09 BA 5F 05 EB E7 3E DA 0A 2A 0F 49 B1 4D TW.._…>..*.I.M
04C0: C8 57 5E 5E 6F 57 32 B5 54 FC 55 B0 F1 06 16 32 .W^^oW2.T.U….2
04D0: B7 19 01 EE 66 EA 3F 52 07 9B 2B D9 AF ….f.?R..+..
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: INITIATE
token: “`\202\004\350\006\t*\206H\206\367\022\001\002\002\001\000n\202\004\3270\202\004\323\240\003\002\001\005\241\003\002\001\016\242\a\003\005\000 \000\000\000\243\202\003\354a\202\003\3500\202\003\344\240\003\002\001\005\241\t\033\aFOO.COM\242\0370\035\240\003\002\001\000\241\0260\024\033\004hdfs\033\frip1.foo.com\243\202\003\2570\202\003\253\240\003\002\001\022\241\003\002\001\004\242\202\003\235\004\202\003\231\334\320O\370\214 \273\274\210\265B$\347\375\263\325\343\327\312\246b\214\267\346\210a\243\265\266\355\351a\374\031\000\271\246\214c&\346\365\207%\205\266\0171\233\246I\215\262\225[t9\234l\n,m\032\347\274K\017\372K\332\300\367\201\203\307\225\255\250f3\0300>\031\314\253{\365m\030\357\314\253\005\213\362\000\032:\216\263\340@4\254lc\350\232\232b\023\207\201H\025\227\215\303\004+\331\305\020|@\023\030Z\223\037\324\033#~\237jl\256\005\020\374LR_\275\351K\031\003\336\b\255\\216\225\326\a\325e\372J\362\236\033\350\b \323T\241I\230l\222K\2641\2652mC\241\rV\201\222H\252\364O\364}\236\234{VV\255a\211p$\005lu\000\362(Ej\200\240\346\rgE\242\335\371\252\2206\253]P\232X\367J\025\f\210\214\377\253\2716\036@\202({\210\277L|\246\376\361\034a\3441\212]QA\250x\261\3466te\323$uBX7\267\223HV\223t\025?\377\365\224{m\031\025\337$Z\324u\022\272\274\023\236\347\333\034\3156\220\242;H\321\205\001[\\\314r\2607\236i\326\305\200.\022\375\377\235\236m\217%W\345\342uN\211T\212\247D\035\021\217\330\365\307\0313\036\375\025\213Wt\356Sw\303}\245?\3679v*\247O\337\345\210\213\345\002\321\347\257~:\000}\203\2514h_\037\367\207\206\325\323\236\300\363\332:\30F[\317Gy\360{ shMV\210>\264v%\375G,K78\025\032\345\276\374\253H\303hS\367\307\241\320\035\302\325\0328*OKq4\375+\306iWK*\3302\034Tr\313\307\033\240\357\331{\336u|\254\3058:ME\031x.\265\301\200\233\360\247\361\254T\233C~\020\006@(\361v\257>\330OI\221\257\260\375\243\037K\317md\221x\371?\314+h\265\342\213\214\245i\252)g:H\016N\223\221\351\260\212\376\000\365\022\263\214\a\3766Wlh\330x\365\370ks\n}W\346\355\342\244jF\221u,\263\325C\327\201[w\326\222\b\251\366\273\266W\200.\3267\370s\274\243\\\2024*\315\003\227t\373\024\232\030\271/\324\0017\203\037j\2311~\345\356z\227\270\327\201w\004\325\201\001\t\226S\227\”\313eH)\367\030\336\332\313U>\274\327\374\273\vI\221\274N\020!\370\324N\330\027w{\266\275A#\3338ZY\331\351\350\344\216#\305\326V>\210y[\364$c\226\031v\362\264#\227\343(\214\235\365\3137j\231L\333;#\r\250\362B3\f5]\331\356[u\004\247\324\224_I\232\243\006\025S\342,!\037\371\314Q\203X\030\216\313\2756\tP[\220\”\214S\223\207\262.\245U\016\t\002\267ZM\267SX\031\277lo\263\020\363\346Yi\377f\310V\032\252S\”\360]\270\”\026^\255\300\2604\361\004\233d\226&Z\00\n\2330\236\016\233\203\’\203\030\016\351.Bg\263S\207x\022\022\251\024Z\005\355jCA\276N\251\221hl\033\211t\024\307\206\302G\3760lB\331Y\217\246\032P@\ap\021\336\262\016\321\316<\217\321\335\367\304\022\315\270K\n\231\264\357\273\373J{\356\234\3427\340\006\354\024\023\361\371Dq\243\366\376m@F\264N\322%Z\340^\233\033(\235\221\301.\a\342{\316E\322[\204V\244\201\3150\201\312\240\003\002\001\022\242\201\302\004\201\277\032p\250-\213\221\247\022\217,\032\266\361\341\342\361\344\304\361\321\024w\216Zs\334B\251\364\325Z\2205\232\225\275\231\266\327z-\370=6q\\\343\021v\246\017b\3300\217\351\261X\356\2201\355\377:\277S\372\276E\256\243\346\2350X9\216\303\021\265c\360\022\247\266\274\372\331u\3315\n\201\253S\211uL\357\025\220=L\360\305\323\322\242\310\016\b\367\fB\303\221d\362+5Y\0024\200dD8?R\307<\355\001\al\374`B\244\215\327\375\320\2031\372TW\t\272_\005\353\347>\332\n*\017I\261M\310W^^oW2\265T\374U\260\361\006\0262\267\031\001\356f\352?R\a\233+\331\257”
auths {
method: “KERBEROS”
mechanism: “GSSAPI”
protocol: “hdfs”
serverId: “rip1.foo.com”
}
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: CHALLENGE
token: “`\201\231\006\t*\206H\206\367\022\001\002\002\002\000o\201\2110\201\206\240\003\002\001\005\241\003\002\001\017\242z0x\240\003\002\001\022\242q\004o\t`\313Hu\3150\231z\345\333\3508\302x\n\337[H\357\226\b\r\377O\350\246A\240\221\223\355\243@\2651V\032\330\251A\314\361!\270UO\303L\030|z\342\341O\3050\000\310!\231V\316\204)>9c\345\326\346\300\253X\303b6\266-^\217%J\224\246W\270\330\313g/\205,h,c\037\306Z\273{B\003\260CZ\004\247S\252\331”
Entered Krb5Context.initSecContext with state=STATE_IN_PROCESS
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting peerSeqNumber to: 254781051
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE
token: “”
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: CHALLENGE
token: “\005\004\005\377\000\f\000\000\000\000\000\000\017/\246{\a\001\000\000\201`y\240\327\375\271K\033\027\227C”
Krb5Context.unwrap: token=[05 04 05 ff 00 0c 00 00 00 00 00 00 0f 2f a6 7b 07 01 00 00 81 60 79 a0 d7 fd b9 4b 1b 17 97 43 ]
Krb5Context.unwrap: data=[07 01 00 00 ]
Krb5Context.wrap: data=[01 01 00 00 ]
Krb5Context.wrap: token=[05 04 04 ff 00 0c 00 00 00 00 00 00 25 cb 6e c9 01 01 00 00 5b d4 02 3e c7 c8 71 42 27 10 b8 3d ]
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE
token: “\005\004\004\377\000\f\000\000\000\000\000\000%\313n\311\001\001\000\000[\324\002>\307\310qB\’\020\270=”
16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: SUCCESS
16/04/22 12:13:54 DEBUG ipc.Client: Negotiated QOP is :auth
16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM: starting, having connections 1
16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM sending #0
16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM got value #0
16/04/22 12:13:54 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 691ms
16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Response <- rip1-horton2.foo.com/172.16.201.73:8020: getFileInfo {fs { fileType: IS_DIR path: “” length: 0 permission { perm: 509 } owner: “1002” group: “500” modification_time: 1461254680300 access_time: 1461254680300 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410103 childrenNum: -1 }}
16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Call -> rip1-horton2.foo.com/172.16.201.73:8020: getListing {src: “/” startAfter: “” needLocation: false}
16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM sending #1
16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM got value #1
16/04/22 12:13:54 DEBUG ipc.ProtobufRpcEngine: Call: getListing took 13ms
16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Response <- rip1-horton2.foo.com/172.16.201.73:8020: getListing {dirList { partialListing { fileType: IS_DIR path: “mr-history” length: 0 permission { perm: 511 } owner: “horton-mapred” group: “500” modification_time: 1418425889000 access_time: 1418425889000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410104 childrenNum: -1 } partialListing { fileType: IS_DIR path: “user” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1461340729681 access_time: 1461340729681 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410158 childrenNum: -1 } partialListing { fileType: IS_DIR path: “wiki” length: 0 permission { perm: 493 } owner: “508” group: “500” modification_time: 1418943800000 access_time: 1418943800000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410547 childrenNum: -1 } partialListing { fileType: IS_DIR path: “tmp” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1461251997618 access_time: 1461251997618 symlink: “” block_replication: 0 blocksize: 0 fileId: 4301390448 childrenNum: -1 } partialListing { fileType: IS_DIR path: “webhdfs” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1436279388000 access_time: 1436279388000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410575 childrenNum: -1 } partialListing { fileType: IS_FILE path: “ID.txt” length: 3082 permission { perm: 420 } owner: “root” group: “wheel” modification_time: 1461254430635 access_time: 1461254430635 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4303225170 childrenNum: 0 } partialListing { fileType: IS_DIR path: “mapred” length: 0 permission { perm: 509 } owner: “horton-mapred” group: “500” modification_time: 1446485241000 access_time: 1446485241000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410577 childrenNum: -1 } partialListing { fileType: IS_DIR path: “temp” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1421787820000 access_time: 1421787820000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296934115 childrenNum: -1 } partialListing { fileType: IS_FILE path: “USERS.txt” length: 17432 permission { perm: 420 } owner: “root” group: “wheel” modification_time: 1461092279350 access_time: 1461092279350 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4300669033 childrenNum: 0 } partialListing { fileType: IS_FILE path: “ID2.txt” length: 344 permission { perm: 420 } owner: “root” group: “wheel” modification_time: 1461255046159 access_time: 1461255046159 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4302307618 childrenNum: 0 } partialListing { fileType: IS_DIR path: “hdfs” length: 0 permission { perm: 493 } owner: “508” group: “500” modification_time: 1444892583000 access_time: 1444892583000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296934116 childrenNum: -1 } partialListing { fileType: IS_DIR path: “apps” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1436198205000 access_time: 1436198205000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296934022 childrenNum: -1 } partialListing { fileType: IS_DIR path: “app-logs” length: 0 permission { perm: 511 } owner: “507” group: “500” modification_time: 1461340151745 access_time: 1461340151745 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296279240 childrenNum: -1 } partialListing { fileType: IS_DIR path: “_tmp” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1461171839874 access_time: 1461171839874 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410553 childrenNum: -1 } partialListing { fileType: IS_FILE path: “ThisIsIsilon.txt” length: 0 permission { perm: 420 } owner: “root” group: “500” modification_time: 1418689161466 access_time: 1418689161466 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4296279310 childrenNum: 0 } partialListing { fileType: IS_DIR path: “hdp” length: 0 permission { perm: 493 } owner: “508” group: “500” modification_time: 1458852305525 access_time: 1458852305525 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410635 childrenNum: -1 } remainingEntries: 0 }}
Found 16 items
-rw-r–r– 3 root wheel 3082 2016-04-21 12:00 /ID.txt
-rw-r–r– 3 root wheel 344 2016-04-21 12:10 /ID2.txt
-rw-r–r– 3 root 500 0 2014-12-15 19:19 /ThisIsIsilon.txt
-rw-r–r– 3 root wheel 17432 2016-04-19 14:57 /USERS.txt
drwxrwxrwx – 508 500 0 2016-04-20 13:03 /_tmp
drwxrwxrwx – 507 500 0 2016-04-22 11:49 /app-logs
drwxrwxrwx – 508 500 0 2015-07-06 11:56 /apps
drwxr-xr-x – 508 500 0 2015-10-15 03:03 /hdfs
drwxr-xr-x – 508 500 0 2016-03-24 16:45 /hdp
drwxrwxr-x – horton-mapred 500 0 2015-11-02 12:27 /mapred
drwxrwxrwx – horton-mapred 500 0 2014-12-12 18:11 /mr-history
drwxrwxrwx – 508 500 0 2015-01-20 16:03 /temp
drwxrwxrwx – 508 500 0 2016-04-21 11:19 /tmp
drwxrwxrwx – 508 500 0 2016-04-22 11:58 /user
drwxrwxrwx – 508 500 0 2015-07-07 10:29 /webhdfs
drwxr-xr-x – 508 500 0 2014-12-18 18:03 /wiki
16/04/22 12:13:54 DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@6c1a5b54
16/04/22 12:13:54 DEBUG ipc.Client: removing client from cache: org.apache.hadoop.ipc.Client@6c1a5b54
16/04/22 12:13:55 DEBUG ipc.Client: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@6c1a5b54
16/04/22 12:13:55 DEBUG ipc.Client: Stopping client
16/04/22 12:13:55 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM: closed
16/04/22 12:13:55 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM: stopped, remaining connections 0