if you are seeing issues with kerberos based hdfs client access against an Isilon cluster, increasing the kerberos logging level on the client can show you a lot more information(a lot)

Change the logging level with the following:

export HADOOP_ROOT_LOGGER=”TRACE,console”

export HADOOP_OPTS=”-Dsun.security.krb5.debug=true”

 

 

quick example of a successful kerberized hadoop call from a kerberized hortoworks cluster against Isilon:

[root@hdp4 jvm]# su – hdpuser3

[hdpuser3@hdp4 ~]$ kinit

Password for hdpuser3@FOO.COM:

[hdpuser3@hdp4 ~]$ klist -e

Ticket cache: FILE:/tmp/krb5cc_49003

Default principal: hdpuser3@FOO.COM

Valid starting     Expires            Service principal

04/22/16 12:18:40  04/22/16 22:17:00  krbtgt/FOO.COM@FOO.COM

renew until 04/29/16 12:18:40, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

[hdpuser3@hdp4 ~]$ export HADOOP_ROOT_LOGGER=”TRACE,console”

[hdpuser3@hdp4 ~]$ export HADOOP_OPTS=”-Dsun.security.krb5.debug=true”

[hdpuser3@hdp4 ~]$ hadoop fs -ls /

16/04/22 12:13:50 DEBUG util.Shell: setsid exited with exit code 0

16/04/22 12:13:51 DEBUG conf.Configuration: parsing URL jar:file:/usr/hdp/2.3.4.0-3485/hadoop/hadoop-common-2.7.1.2.3.4.0-3485.jar!/core-default.xml

16/04/22 12:13:51 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@7cbd213e

16/04/22 12:13:51 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/2.3.4.0-3485/0/core-site.xml

16/04/22 12:13:51 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@5cdd8682

Java config name: null

Native config name: /etc/krb5.conf

Loaded from native config

16/04/22 12:13:52 DEBUG security.Groups:  Creating new Groups object

16/04/22 12:13:52 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library…

16/04/22 12:13:52 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library

16/04/22 12:13:52 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution

16/04/22 12:13:52 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping

16/04/22 12:13:52 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000

>>>KinitOptions cache name is /tmp/krb5cc_49003

>>>DEBUG <CCacheInputStream>  client principal is hdpuser3@FOO.COM

>>>DEBUG <CCacheInputStream> server principal is krbtgt/FOO.COM@FOO.COM

>>>DEBUG <CCacheInputStream> key type: 18

>>>DEBUG <CCacheInputStream> auth time: Fri Apr 22 11:56:32 EDT 2016

>>>DEBUG <CCacheInputStream> start time: Fri Apr 22 11:58:12 EDT 2016

>>>DEBUG <CCacheInputStream> end time: Fri Apr 22 21:56:32 EDT 2016

>>>DEBUG <CCacheInputStream> renew_till time: Fri Apr 29 11:58:12 EDT 2016

>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE; INITIAL; PRE_AUTH;

16/04/22 12:13:52 DEBUG security.UserGroupInformation: hadoop login

16/04/22 12:13:52 DEBUG security.UserGroupInformation: hadoop login commit

16/04/22 12:13:52 DEBUG security.UserGroupInformation: using kerberos user:hdpuser3@FOO.COM

16/04/22 12:13:52 DEBUG security.UserGroupInformation: Using user: “hdpuser3@FOO.COM” with name hdpuser3@FOO.COM

16/04/22 12:13:52 DEBUG security.UserGroupInformation: User entry: “hdpuser3@FOO.COM”

16/04/22 12:13:52 DEBUG security.UserGroupInformation: UGI loginUser:hdpuser3@FOO.COM (auth:KERBEROS)

16/04/22 12:13:52 DEBUG security.UserGroupInformation: Found tgt Ticket (hex) =

0000: 61 82 03 F8 30 82 03 F4   A0 03 02 01 05 A1 09 1B  a…0………..

0010: 07 46 4F 4F 2E 43 4F 4D   A2 1C 30 1A A0 03 02 01  .FOO.COM..0…..

0020: 02 A1 13 30 11 1B 06 6B   72 62 74 67 74 1B 07 46  …0…krbtgt..F

0030: 4F 4F 2E 43 4F 4D A3 82   03 C2 30 82 03 BE A0 03  OO.COM….0…..

0040: 02 01 12 A1 03 02 01 02   A2 82 03 B0 04 82 03 AC  …………….

0050: 80 45 B8 57 D9 53 73 66   A9 20 92 30 46 2E 64 A6  .E.W.Ssf. .0F.d.

0060: 2B 42 0E 1E 71 6D 56 35   88 78 A5 CF 2D 3C 01 24  +B..qmV5.x..-<.$

0070: 49 5D FA 7C 88 31 03 0F   26 9A C1 29 63 30 67 B3  I]…1..&..)c0g.

0080: 20 AC 73 F8 35 D3 08 E9   FD 81 ED 66 C3 41 AD 4B   .s.5……f.A.K

0090: 09 B6 DF B1 3B C6 FC 5F   1B A5 B5 FC A9 8A D3 A4  ….;.._……..

00A0: 5F 2B 31 73 0F 03 87 93   43 35 2E 6D 9D 4A 97 C8  _+1s….C5.m.J..

00B0: 3C 5C 45 B2 B0 21 D5 C8   B5 0A 73 10 76 92 AB 99  <\E..!….s.v…

00C0: B7 EE 63 C8 37 94 1B 57   E7 A0 57 7E 84 17 1E 04  ..c.7..W..W…..

00D0: 13 F9 5D 62 D6 0C 02 3D   46 EF 3C AE 27 D6 74 A1  ..]b…=F.<.’.t.

00E0: A9 A2 42 17 3A DA E3 DF   F5 0B B0 F8 16 0F 77 59  ..B.:………wY

00F0: 84 61 DF 94 75 39 C8 6C   AD 67 20 E2 EC 49 54 FE  .a..u9.l.g ..IT.

0100: 14 4C 2E 9C 30 AC 1C 42   37 01 27 A0 C0 7F 7E 32  .L..0..B7.’….2

0110: DC D0 8F 14 36 A4 B7 D4   2B 0A ED DC BD AC 9E BD  ….6…+…….

0120: 4A 43 C7 FB 60 B2 30 9F   F7 E1 50 88 44 AC 07 06  JC..`.0…P.D…

0130: 4E 1C FE 3A 40 88 EF D3   20 82 7C 39 89 BC CA 4F  N..:@… ..9…O

0140: 58 3B FD 3D A5 24 62 C9   75 C3 47 F3 E0 EA 83 B6  X;.=.$b.u.G…..

0150: 0C FA F3 25 8A BC 42 22   59 54 81 5D 25 64 E9 7E  …%..B”YT.]%d..

0160: 9E 73 70 DD 24 35 E5 94   3D 71 B6 C2 14 C1 3F 5C  .sp.$5..=q….?\

0170: 75 2B 0B 99 4E FC 29 40   8B B7 1D 09 A1 AA 52 41  u+..N.)@……RA

0180: BF 16 91 67 5F 21 53 68   3E 98 65 4E E2 AF FC 7C  …g_!Sh>.eN….

0190: 96 70 13 74 FB B5 E5 9F   87 C5 39 49 B4 51 BA 8F  .p.t……9I.Q..

01A0: F0 42 4B B4 4C F8 CF D3   5C 69 3C 53 8B 42 E8 F3  .BK.L…\i<S.B..

01B0: CC 71 F6 57 AF 5E 0F 1E   0C 63 49 0A 86 8B A6 00  .q.W.^…cI…..

01C0: 1A AF 8F B8 6E 2C 81 1E   8B B4 49 DF 61 95 77 3B  ….n,….I.a.w;

01D0: A6 BB 48 4C A5 ED 03 F2   3D A2 BF 00 27 90 34 5A  ..HL….=…’.4Z

01E0: 9E A9 57 D7 45 AB DC 85   71 37 D7 65 27 82 04 41  ..W.E…q7.e’..A

01F0: 5E 5D 14 40 94 AC 95 1E   4C F0 23 B1 94 96 A7 89  ^].@….L.#…..

0200: 03 6C 67 2D C0 08 18 22   2A DE FD DC 03 88 FD 3D  .lg-…”*……=

0210: A4 95 C7 31 6C 28 F2 AD   2F C2 21 65 51 03 A9 8F  …1l(../.!eQ…

0220: 85 C5 52 07 A3 81 05 E5   7F A4 EE 17 F5 71 C6 AC  ..R……….q..

0230: F0 38 F3 EA FF 96 75 95   ED 03 2C 72 4C 3B C6 E1  .8….u…,rL;..

0240: 63 65 13 E6 45 69 F6 91   CD 39 B9 E8 CE F0 A6 96  ce..Ei…9……

0250: C0 AF 4A 43 FB 18 F8 E7   09 26 7C A5 0D 24 FB 7D  ..JC…..&…$..

0260: CF C3 64 C4 1E 40 28 D7   D5 48 15 15 AA 60 8C 73  ..d..@(..H…`.s

0270: 6C 0F 36 97 46 B0 BE B6   EE C4 1F 6B 88 F5 1E F2  l.6.F……k….

0280: F8 93 04 B7 9C 1C 21 5D   D6 86 4D 45 DE 57 68 49  ……!]..ME.WhI

0290: 85 D1 A2 5F 0A 13 2E 33   DB A2 2E 90 9D 4F A2 E2  …_…3…..O..

02A0: 99 CB 3A BE 4E A1 C2 B3   D5 0F 7A 56 2E BA DB 15  ..:.N…..zV….

02B0: 2D 6F DA 51 B6 E8 B8 E7   19 94 A3 7D 0E B2 31 4F  -o.Q……….1O

02C0: E5 21 C4 D3 DC 11 64 DA   7E 37 89 E3 81 1F 24 C0  .!….d..7….$.

02D0: 4B 2C 11 46 A1 06 09 0B   0E 61 3A EA 59 09 83 8B  K,.F…..a:.Y…

02E0: A8 6F FC 78 14 41 EB 2F   32 81 AC CA 3B 3F 5E CF  .o.x.A./2…;?^.

02F0: 59 1F 48 C1 36 7A A2 82   84 6D D4 36 1F 54 9E 7C  Y.H.6z…m.6.T..

0300: 20 DC DA D2 68 19 B8 DF   14 25 25 C0 E9 DC 71 56   …h….%%…qV

0310: 51 73 63 44 86 A5 33 B6   25 0A 57 49 1F 0A 02 4F  QscD..3.%.WI…O

0320: 70 93 AC 52 8A BC 7A 7F   10 3C F1 D8 DB CE B7 CF  p..R..z..<……

0330: FA 84 FC C0 79 10 5F 25   3D 38 B7 4F D8 EA 4D 51  ….y._%=8.O..MQ

0340: 74 D3 C0 E1 04 D1 6E 34   C1 62 4F 4F 16 86 DE E8  t…..n4.bOO….

0350: B8 5B B1 50 13 05 30 09   59 58 A6 A9 E3 7E 9F 79  .[.P..0.YX…..y

0360: 8D E9 8B 25 D9 0E DB 17   26 FF A1 AD 8B 8A 55 52  …%….&…..UR

0370: 35 9F 3C 9E 56 7B 61 A1   AC 08 E0 37 9D EA 77 D9  5.<.V.a….7..w.

0380: 28 75 3D 39 C2 08 BF E9   8B 4A 8A 60 3A 69 3E 1C  (u=9…..J.`:i>.

0390: 6F E7 B9 92 96 F4 79 A5   87 0B 25 B2 11 5B FC ED  o…..y…%..[..

03A0: 19 36 D5 35 94 F5 74 90   82 70 CF F8 6A BB F2 80  .6.5..t..p..j…

03B0: FC 37 01 0A 41 AE 52 0C   A1 23 2F 62 4B 68 35 FF  .7..A.R..#/bKh5.

03C0: 3E DD 12 7E 27 5F 1D AF   66 52 D0 8C 06 60 46 5C  >…’_..fR…`F\

03D0: 08 BF EA 5A AD F1 BD 14   50 7A AF 48 1C 48 91 0D  …Z….Pz.H.H..

03E0: D8 0E 2E 44 9D 92 6E 0C   AA 0F C1 E5 6E 18 69 BB  …D..n…..n.i.

03F0: 4F 90 EE F7 22 9F F0 DE   F1 E9 69 4D              O…”…..iM

Client Principal = hdpuser3@FOO.COM

Server Principal = krbtgt/FOO.COM@FOO.COM

Session Key = EncryptionKey: keyType=18 keyBytes (hex dump)=

0000: 40 19 60 17 41 7F 9B A6   FA 6E D2 93 A6 F0 B3 2B  @.`.A….n…..+

0010: 20 BC 93 4D 72 E2 23 75   AF DA CF 5B C7 48 09 C6   ..Mr.#u…[.H..

Forwardable Ticket true

Forwarded Ticket false

Proxiable Ticket false

Proxy Ticket false

Postdated Ticket false

Renewable Ticket true

Initial Ticket true

Auth Time = Fri Apr 22 11:56:32 EDT 2016

Start Time = Fri Apr 22 11:58:12 EDT 2016

End Time = Fri Apr 22 21:56:32 EDT 2016

Renew Till = Fri Apr 29 11:58:12 EDT 2016

Client Addresses  Null

16/04/22 12:13:52 DEBUG security.UserGroupInformation: Current time is 1461341632663

16/04/22 12:13:52 DEBUG security.UserGroupInformation: Next refresh is 1461369412000

16/04/22 12:13:52 TRACE tracing.SpanReceiverHost: No span receiver names found in dfs.client.htrace.spanreceiver.classes.

16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false

16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = true

16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false

16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = /var/lib/hadoop-hdfs/dn_socket

16/04/22 12:13:53 DEBUG retry.RetryUtils: multipleLinearRandomRetry = null

16/04/22 12:13:53 DEBUG ipc.Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@18df8434

16/04/22 12:13:53 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@6c1a5b54

16/04/22 12:13:54 DEBUG unix.DomainSocketWatcher: org.apache.hadoop.net.unix.DomainSocketWatcher$2@2a2a8820: starting with interruptCheckPeriodMs = 60000

16/04/22 12:13:54 TRACE unix.DomainSocketWatcher: DomainSocketWatcher(1426041738): adding notificationSocket 351, connected to 350

16/04/22 12:13:54 DEBUG shortcircuit.DomainSocketFactory: The short-circuit local reads feature is enabled.

16/04/22 12:13:54 DEBUG sasl.DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection

16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Call -> rip1-horton2.foo.com/172.16.201.73:8020: getFileInfo {src: “/”}

16/04/22 12:13:54 DEBUG ipc.Client: The ping interval is 60000 ms.

16/04/22 12:13:54 DEBUG ipc.Client: Connecting to rip1-horton2.foo.com/172.16.201.73:8020

16/04/22 12:13:54 DEBUG security.UserGroupInformation: PrivilegedAction as:hdpuser3@FOO.COM (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:722)

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE

16/04/22 12:13:54 DEBUG azure.NativeAzureFileSystem: finalize() called.

16/04/22 12:13:54 DEBUG azure.NativeAzureFileSystem: finalize() called.

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: NEGOTIATE

auths {

method: “TOKEN”

mechanism: “DIGEST-MD5”

protocol: “default”

challenge: “nonce=\”p0L557HD2nzgyagR7F5o3hTweljDX9n6kd7M0YAT368=\”,realm=\”rip1-1\”,qop=\”auth,auth-int,auth-conf\”,cipher=\”rc4-40,rc4-56,rc4,des,3des\”,maxbuf=65536,charset=utf-8,algorithm=md5-sess”

}

auths {

method: “KERBEROS”

mechanism: “GSSAPI”

protocol: “hdfs”

serverId: “rip1.foo.com”

}

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:@org.apache.hadoop.security.token.TokenInfo(value=class org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSelector)

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=, serverPrincipal=dfs.namenode.kerberos.principal)

16/04/22 12:13:54 DEBUG security.SaslRpcClient: RPC Server’s Kerberos principal name for protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is hdfs/rip1.foo.com@FOO.COM

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Creating SASL GSSAPI(KERBEROS)  client to authenticate to service at rip1.foo.com

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Use KERBEROS authentication for protocol ClientNamenodeProtocolPB

Found ticket for hdpuser3@FOO.COM to go to krbtgt/FOO.COM@FOO.COM expiring on Fri Apr 22 21:56:32 EDT 2016

Entered Krb5Context.initSecContext with state=STATE_NEW

Found ticket for hdpuser3@FOO.COM to go to krbtgt/FOO.COM@FOO.COM expiring on Fri Apr 22 21:56:32 EDT 2016

Service ticket not found in the subject

>>> Credentials acquireServiceCreds: same realm

Using builtin default etypes for default_tgs_enctypes

default etypes for default_tgs_enctypes: 18 17 16 23.

>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType

>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType

>>> KdcAccessibility: reset

>>> KrbKdcReq send: kdc=rs-2008r2-dc.foo.com UDP:88, timeout=30000, number of retries =3, #bytes=1336

>>> KDCCommunication: kdc=rs-2008r2-dc.foo.com UDP:88, timeout=30000,Attempt =1, #bytes=1336

>>> KrbKdcReq send: #bytes read=1304

>>> KdcAccessibility: remove rs-2008r2-dc.foo.com

>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType

>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000

>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType

Krb5Context setting mySeqNumber to: 634089161

Created InitSecContextToken:

0000: 01 00 6E 82 04 D7 30 82   04 D3 A0 03 02 01 05 A1  ..n…0………

0010: 03 02 01 0E A2 07 03 05   00 20 00 00 00 A3 82 03  ……… ……

0020: EC 61 82 03 E8 30 82 03   E4 A0 03 02 01 05 A1 09  .a…0……….

0030: 1B 07 46 4F 4F 2E 43 4F   4D A2 1F 30 1D A0 03 02  ..FOO.COM..0….

0040: 01 00 A1 16 30 14 1B 04   68 64 66 73 1B 0C 72 69  ….0…hdfs..ri

0050: 70 31 2E 66 6F 6F 2E 63   6F 6D A3 82 03 AF 30 82  p1.foo.com….0.

0060: 03 AB A0 03 02 01 12 A1   03 02 01 04 A2 82 03 9D  …………….

0070: 04 82 03 99 DC D0 4F F8   8C 20 BB BC 88 B5 42 24  ……O.. ….B$

0080: E7 FD B3 D5 E3 D7 CA A6   62 8C B7 E6 88 61 A3 B5  ……..b….a..

0090: B6 ED E9 61 FC 19 00 B9   A6 8C 63 26 E6 F5 87 25  …a……c&…%

00A0: 85 B6 0F 31 9B A6 49 8D   B2 95 5B 74 39 9C 6C 0A  …1..I…[t9.l.

00B0: 2C 6D 1A E7 BC 4B 0F FA   4B DA C0 F7 81 83 C7 95  ,m…K..K…….

00C0: AD A8 66 33 18 30 3E 19   CC AB 7B F5 6D 18 EF CC  ..f3.0>…..m…

00D0: AB 05 8B F2 00 1A 3A 8E   B3 E0 40 34 AC 6C 63 E8  ……:…@4.lc.

00E0: 9A 9A 62 13 87 81 48 15   97 2A 7F 8D C3 04 2B D9  ..b…H..*….+.

00F0: C5 10 7C 40 13 18 5A 93   1F D4 1B 23 7E 9F 6A 6C  …@..Z….#..jl

0100: AE 05 10 FC 4C 52 5F BD   E9 4B 19 03 DE 08 AD 22  ….LR_..K…..”

0110: 7F 8E 95 D6 07 D5 65 FA   4A F2 9E 1B E8 08 20 D3  ……e.J….. .

0120: 54 A1 49 98 6C 92 4B B4   31 B5 32 6D 43 A1 0D 56  T.I.l.K.1.2mC..V

0130: 81 92 48 AA F4 4F F4 7D   9E 9C 7B 56 56 AD 61 89  ..H..O…..VV.a.

0140: 70 24 05 6C 75 00 F2 28   45 6A 80 A0 E6 0D 67 45  p$.lu..(Ej….gE

0150: A2 DD F9 AA 90 36 AB 5D   50 9A 58 F7 4A 15 0C 88  …..6.]P.X.J…

0160: 8C FF AB B9 36 1E 40 82   28 7B 88 BF 4C 7C A6 FE  ….6.@.(…L…

0170: F1 1C 61 E4 31 8A 5D 51   41 A8 78 B1 E6 36 74 65  ..a.1.]QA.x..6te

0180: D3 24 75 32 7F 42 58 37   B7 93 48 56 93 74 15 3F  .$u2.BX7..HV.t.?

0190: FF F5 94 7B 6D 19 15 DF   24 5A D4 75 12 BA BC 13  ….m…$Z.u….

01A0: 9E E7 DB 1C CD 36 90 A2   3B 48 D1 85 01 5B 5C CC  …..6..;H…[\.

01B0: 72 B0 37 9E 69 D6 C5 80   2E 12 FD FF 9D 9E 6D 8F  r.7.i………m.

01C0: 25 57 E5 E2 75 4E 89 54   8A A7 44 1D 11 8F D8 F5  %W..uN.T..D…..

01D0: C7 19 33 1E FD 15 8B 57   74 EE 53 77 C3 7D A5 3F  ..3….Wt.Sw…?

01E0: F7 39 76 2A A7 4F DF E5   88 8B E5 02 D1 E7 AF 7E  .9v*.O……….

01F0: 3A 00 7D 83 A9 34 68 5F   1F F7 87 86 D5 D3 9E C0  :….4h_……..

0200: F3 DA 3A C0 7F 46 5B CF   47 79 F0 7B 20 73 68 4D  ..:..F[.Gy.. shM

0210: 56 88 3E B4 76 25 FD 47   2C 4B 37 38 15 1A E5 BE  V.>.v%.G,K78….

0220: FC AB 48 C3 68 53 F7 C7   A1 D0 1D C2 D5 1A 38 2A  ..H.hS……..8*

0230: 4F 4B 71 34 FD 2B C6 69   57 4B 2A D8 32 1C 54 72  OKq4.+.iWK*.2.Tr

0240: CB C7 1B A0 EF D9 7B DE   75 7C AC C5 38 3A 4D 45  ……..u…8:ME

0250: 19 78 2E B5 C1 80 9B F0   A7 F1 AC 54 9B 43 7E 10  .x………T.C..

0260: 06 40 28 F1 76 AF 3E D8   4F 49 91 AF B0 FD A3 1F  .@(.v.>.OI……

0270: 4B CF 6D 64 91 78 F9 3F   CC 2B 68 B5 E2 8B 8C A5  K.md.x.?.+h…..

0280: 69 AA 29 67 3A 48 0E 4E   93 91 E9 B0 8A FE 00 F5  i.)g:H.N……..

0290: 12 B3 8C 07 FE 36 57 6C   68 D8 78 F5 F8 6B 73 0A  …..6Wlh.x..ks.

02A0: 7D 57 E6 ED E2 A4 6A 46   91 75 2C B3 D5 43 D7 81  .W….jF.u,..C..

02B0: 5B 77 D6 92 08 A9 F6 BB   B6 57 80 2E D6 37 F8 73  [w…….W…7.s

02C0: BC A3 5C 82 34 2A CD 03   97 74 FB 14 9A 18 B9 2F  ..\.4*…t…../

02D0: D4 01 37 83 1F 6A 99 31   7E E5 EE 7A 97 B8 D7 81  ..7..j.1…z….

02E0: 77 04 D5 81 01 09 96 53   97 22 CB 65 48 29 F7 18  w……S.”.eH)..

02F0: DE DA CB 55 3E BC D7 FC   BB 0B 49 91 BC 4E 10 21  …U>…..I..N.!

0300: F8 D4 4E D8 17 77 7B B6   BD 41 23 DB 38 5A 59 D9  ..N..w…A#.8ZY.

0310: E9 E8 E4 8E 23 C5 D6 56   3E 88 79 5B F4 24 63 96  ….#..V>.y[.$c.

0320: 19 76 F2 B4 23 97 E3 28   8C 9D F5 CB 37 6A 99 4C  .v..#..(….7j.L

0330: DB 3B 23 0D A8 F2 42 33   0C 35 5D D9 EE 5B 75 04  .;#…B3.5]..[u.

0340: A7 D4 94 5F 49 9A A3 06   15 53 E2 2C 21 1F F9 CC  …_I….S.,!…

0350: 51 83 58 18 8E CB BD 36   09 50 5B 90 22 8C 53 93  Q.X….6.P[.”.S.

0360: 87 B2 2E A5 55 0E 09 02   B7 5A 4D B7 53 58 19 BF  ….U….ZM.SX..

0370: 6C 6F B3 10 F3 E6 59 69   FF 66 C8 56 1A AA 53 22  lo….Yi.f.V..S”

0380: F0 5D B8 22 16 5E AD C0   B0 34 F1 04 9B 64 96 26  .].”.^…4…d.&

0390: 5A 04 7F 0A 9B 30 9E 0E   9B 83 27 83 18 0E E9 2E  Z….0….’…..

03A0: 42 67 B3 53 87 78 12 12   A9 14 5A 05 ED 6A 43 41  Bg.S.x….Z..jCA

03B0: BE 4E A9 91 68 6C 1B 89   74 14 C7 86 C2 47 FE 30  .N..hl..t….G.0

03C0: 6C 42 D9 59 8F A6 1A 50   40 07 70 11 DE B2 0E D1  lB.Y…P@.p…..

03D0: CE 3C 8F D1 DD F7 C4 12   CD B8 4B 0A 99 B4 EF BB  .<……..K…..

03E0: FB 4A 7B EE 9C E2 37 E0   06 EC 14 13 F1 F9 44 71  .J….7…….Dq

03F0: A3 F6 FE 6D 40 46 B4 4E   D2 25 5A E0 5E 9B 1B 28  …m@F.N.%Z.^..(

0400: 9D 91 C1 2E 07 E2 7B CE   45 D2 5B 84 56 A4 81 CD  ……..E.[.V…

0410: 30 81 CA A0 03 02 01 12   A2 81 C2 04 81 BF 1A 70  0…………..p

0420: A8 2D 8B 91 A7 12 8F 2C   1A B6 F1 E1 E2 F1 E4 C4  .-…..,……..

0430: F1 D1 14 77 8E 5A 73 DC   42 A9 F4 D5 5A 90 35 9A  …w.Zs.B…Z.5.

0440: 95 BD 99 B6 D7 7A 2D F8   3D 36 71 5C E3 11 76 A6  …..z-.=6q\..v.

0450: 0F 62 D8 30 8F E9 B1 58   EE 90 31 ED FF 3A BF 53  .b.0…X..1..:.S

0460: FA BE 45 AE A3 E6 9D 30   58 39 8E C3 11 B5 63 F0  ..E….0X9….c.

0470: 12 A7 B6 BC FA D9 75 D9   35 0A 81 AB 53 89 75 4C  ……u.5…S.uL

0480: EF 15 90 3D 4C F0 C5 D3   D2 A2 C8 0E 08 F7 0C 42  …=L……….B

0490: C3 91 64 F2 2B 35 59 02   34 80 64 44 38 3F 52 C7  ..d.+5Y.4.dD8?R.

04A0: 3C ED 01 07 6C FC 60 42   A4 8D D7 FD D0 83 31 FA  <…l.`B……1.

04B0: 54 57 09 BA 5F 05 EB E7   3E DA 0A 2A 0F 49 B1 4D  TW.._…>..*.I.M

04C0: C8 57 5E 5E 6F 57 32 B5   54 FC 55 B0 F1 06 16 32  .W^^oW2.T.U….2

04D0: B7 19 01 EE 66 EA 3F 52   07 9B 2B D9 AF           ….f.?R..+..

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: INITIATE

token: “`\202\004\350\006\t*\206H\206\367\022\001\002\002\001\000n\202\004\3270\202\004\323\240\003\002\001\005\241\003\002\001\016\242\a\003\005\000 \000\000\000\243\202\003\354a\202\003\3500\202\003\344\240\003\002\001\005\241\t\033\aFOO.COM\242\0370\035\240\003\002\001\000\241\0260\024\033\004hdfs\033\frip1.foo.com\243\202\003\2570\202\003\253\240\003\002\001\022\241\003\002\001\004\242\202\003\235\004\202\003\231\334\320O\370\214 \273\274\210\265B$\347\375\263\325\343\327\312\246b\214\267\346\210a\243\265\266\355\351a\374\031\000\271\246\214c&\346\365\207%\205\266\0171\233\246I\215\262\225[t9\234l\n,m\032\347\274K\017\372K\332\300\367\201\203\307\225\255\250f3\0300>\031\314\253{\365m\030\357\314\253\005\213\362\000\032:\216\263\340@4\254lc\350\232\232b\023\207\201H\025\227\215\303\004+\331\305\020|@\023\030Z\223\037\324\033#~\237jl\256\005\020\374LR_\275\351K\031\003\336\b\255\\216\225\326\a\325e\372J\362\236\033\350\b \323T\241I\230l\222K\2641\2652mC\241\rV\201\222H\252\364O\364}\236\234{VV\255a\211p$\005lu\000\362(Ej\200\240\346\rgE\242\335\371\252\2206\253]P\232X\367J\025\f\210\214\377\253\2716\036@\202({\210\277L|\246\376\361\034a\3441\212]QA\250x\261\3466te\323$uBX7\267\223HV\223t\025?\377\365\224{m\031\025\337$Z\324u\022\272\274\023\236\347\333\034\3156\220\242;H\321\205\001[\\\314r\2607\236i\326\305\200.\022\375\377\235\236m\217%W\345\342uN\211T\212\247D\035\021\217\330\365\307\0313\036\375\025\213Wt\356Sw\303}\245?\3679v*\247O\337\345\210\213\345\002\321\347\257~:\000}\203\2514h_\037\367\207\206\325\323\236\300\363\332:\30F[\317Gy\360{ shMV\210>\264v%\375G,K78\025\032\345\276\374\253H\303hS\367\307\241\320\035\302\325\0328*OKq4\375+\306iWK*\3302\034Tr\313\307\033\240\357\331{\336u|\254\3058:ME\031x.\265\301\200\233\360\247\361\254T\233C~\020\006@(\361v\257>\330OI\221\257\260\375\243\037K\317md\221x\371?\314+h\265\342\213\214\245i\252)g:H\016N\223\221\351\260\212\376\000\365\022\263\214\a\3766Wlh\330x\365\370ks\n}W\346\355\342\244jF\221u,\263\325C\327\201[w\326\222\b\251\366\273\266W\200.\3267\370s\274\243\\\2024*\315\003\227t\373\024\232\030\271/\324\0017\203\037j\2311~\345\356z\227\270\327\201w\004\325\201\001\t\226S\227\”\313eH)\367\030\336\332\313U>\274\327\374\273\vI\221\274N\020!\370\324N\330\027w{\266\275A#\3338ZY\331\351\350\344\216#\305\326V>\210y[\364$c\226\031v\362\264#\227\343(\214\235\365\3137j\231L\333;#\r\250\362B3\f5]\331\356[u\004\247\324\224_I\232\243\006\025S\342,!\037\371\314Q\203X\030\216\313\2756\tP[\220\”\214S\223\207\262.\245U\016\t\002\267ZM\267SX\031\277lo\263\020\363\346Yi\377f\310V\032\252S\”\360]\270\”\026^\255\300\2604\361\004\233d\226&Z\00\n\2330\236\016\233\203\’\203\030\016\351.Bg\263S\207x\022\022\251\024Z\005\355jCA\276N\251\221hl\033\211t\024\307\206\302G\3760lB\331Y\217\246\032P@\ap\021\336\262\016\321\316<\217\321\335\367\304\022\315\270K\n\231\264\357\273\373J{\356\234\3427\340\006\354\024\023\361\371Dq\243\366\376m@F\264N\322%Z\340^\233\033(\235\221\301.\a\342{\316E\322[\204V\244\201\3150\201\312\240\003\002\001\022\242\201\302\004\201\277\032p\250-\213\221\247\022\217,\032\266\361\341\342\361\344\304\361\321\024w\216Zs\334B\251\364\325Z\2205\232\225\275\231\266\327z-\370=6q\\\343\021v\246\017b\3300\217\351\261X\356\2201\355\377:\277S\372\276E\256\243\346\2350X9\216\303\021\265c\360\022\247\266\274\372\331u\3315\n\201\253S\211uL\357\025\220=L\360\305\323\322\242\310\016\b\367\fB\303\221d\362+5Y\0024\200dD8?R\307<\355\001\al\374`B\244\215\327\375\320\2031\372TW\t\272_\005\353\347>\332\n*\017I\261M\310W^^oW2\265T\374U\260\361\006\0262\267\031\001\356f\352?R\a\233+\331\257”

auths {

method: “KERBEROS”

mechanism: “GSSAPI”

protocol: “hdfs”

serverId: “rip1.foo.com”

}

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: CHALLENGE

token: “`\201\231\006\t*\206H\206\367\022\001\002\002\002\000o\201\2110\201\206\240\003\002\001\005\241\003\002\001\017\242z0x\240\003\002\001\022\242q\004o\t`\313Hu\3150\231z\345\333\3508\302x\n\337[H\357\226\b\r\377O\350\246A\240\221\223\355\243@\2651V\032\330\251A\314\361!\270UO\303L\030|z\342\341O\3050\000\310!\231V\316\204)>9c\345\326\346\300\253X\303b6\266-^\217%J\224\246W\270\330\313g/\205,h,c\037\306Z\273{B\003\260CZ\004\247S\252\331”

Entered Krb5Context.initSecContext with state=STATE_IN_PROCESS

>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType

Krb5Context setting peerSeqNumber to: 254781051

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE

token: “”

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: CHALLENGE

token: “\005\004\005\377\000\f\000\000\000\000\000\000\017/\246{\a\001\000\000\201`y\240\327\375\271K\033\027\227C”

Krb5Context.unwrap: token=[05 04 05 ff 00 0c 00 00 00 00 00 00 0f 2f a6 7b 07 01 00 00 81 60 79 a0 d7 fd b9 4b 1b 17 97 43 ]

Krb5Context.unwrap: data=[07 01 00 00 ]

Krb5Context.wrap: data=[01 01 00 00 ]

Krb5Context.wrap: token=[05 04 04 ff 00 0c 00 00 00 00 00 00 25 cb 6e c9 01 01 00 00 5b d4 02 3e c7 c8 71 42 27 10 b8 3d ]

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE

token: “\005\004\004\377\000\f\000\000\000\000\000\000%\313n\311\001\001\000\000[\324\002>\307\310qB\’\020\270=”

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: SUCCESS

16/04/22 12:13:54 DEBUG ipc.Client: Negotiated QOP is :auth

16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM: starting, having connections 1

16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM sending #0

16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM got value #0

16/04/22 12:13:54 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 691ms

16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Response <- rip1-horton2.foo.com/172.16.201.73:8020: getFileInfo {fs { fileType: IS_DIR path: “” length: 0 permission { perm: 509 } owner: “1002” group: “500” modification_time: 1461254680300 access_time: 1461254680300 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410103 childrenNum: -1 }}

16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Call -> rip1-horton2.foo.com/172.16.201.73:8020: getListing {src: “/” startAfter: “” needLocation: false}

16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM sending #1

16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM got value #1

16/04/22 12:13:54 DEBUG ipc.ProtobufRpcEngine: Call: getListing took 13ms

16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Response <- rip1-horton2.foo.com/172.16.201.73:8020: getListing {dirList { partialListing { fileType: IS_DIR path: “mr-history” length: 0 permission { perm: 511 } owner: “horton-mapred” group: “500” modification_time: 1418425889000 access_time: 1418425889000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410104 childrenNum: -1 } partialListing { fileType: IS_DIR path: “user” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1461340729681 access_time: 1461340729681 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410158 childrenNum: -1 } partialListing { fileType: IS_DIR path: “wiki” length: 0 permission { perm: 493 } owner: “508” group: “500” modification_time: 1418943800000 access_time: 1418943800000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410547 childrenNum: -1 } partialListing { fileType: IS_DIR path: “tmp” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1461251997618 access_time: 1461251997618 symlink: “” block_replication: 0 blocksize: 0 fileId: 4301390448 childrenNum: -1 } partialListing { fileType: IS_DIR path: “webhdfs” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1436279388000 access_time: 1436279388000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410575 childrenNum: -1 } partialListing { fileType: IS_FILE path: “ID.txt” length: 3082 permission { perm: 420 } owner: “root” group: “wheel” modification_time: 1461254430635 access_time: 1461254430635 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4303225170 childrenNum: 0 } partialListing { fileType: IS_DIR path: “mapred” length: 0 permission { perm: 509 } owner: “horton-mapred” group: “500” modification_time: 1446485241000 access_time: 1446485241000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410577 childrenNum: -1 } partialListing { fileType: IS_DIR path: “temp” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1421787820000 access_time: 1421787820000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296934115 childrenNum: -1 } partialListing { fileType: IS_FILE path: “USERS.txt” length: 17432 permission { perm: 420 } owner: “root” group: “wheel” modification_time: 1461092279350 access_time: 1461092279350 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4300669033 childrenNum: 0 } partialListing { fileType: IS_FILE path: “ID2.txt” length: 344 permission { perm: 420 } owner: “root” group: “wheel” modification_time: 1461255046159 access_time: 1461255046159 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4302307618 childrenNum: 0 } partialListing { fileType: IS_DIR path: “hdfs” length: 0 permission { perm: 493 } owner: “508” group: “500” modification_time: 1444892583000 access_time: 1444892583000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296934116 childrenNum: -1 } partialListing { fileType: IS_DIR path: “apps” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1436198205000 access_time: 1436198205000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296934022 childrenNum: -1 } partialListing { fileType: IS_DIR path: “app-logs” length: 0 permission { perm: 511 } owner: “507” group: “500” modification_time: 1461340151745 access_time: 1461340151745 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296279240 childrenNum: -1 } partialListing { fileType: IS_DIR path: “_tmp” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1461171839874 access_time: 1461171839874 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410553 childrenNum: -1 } partialListing { fileType: IS_FILE path: “ThisIsIsilon.txt” length: 0 permission { perm: 420 } owner: “root” group: “500” modification_time: 1418689161466 access_time: 1418689161466 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4296279310 childrenNum: 0 } partialListing { fileType: IS_DIR path: “hdp” length: 0 permission { perm: 493 } owner: “508” group: “500” modification_time: 1458852305525 access_time: 1458852305525 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410635 childrenNum: -1 } remainingEntries: 0 }}

Found 16 items

-rw-r–r–   3 root          wheel       3082 2016-04-21 12:00 /ID.txt

-rw-r–r–   3 root          wheel        344 2016-04-21 12:10 /ID2.txt

-rw-r–r–   3 root          500            0 2014-12-15 19:19 /ThisIsIsilon.txt

-rw-r–r–   3 root          wheel      17432 2016-04-19 14:57 /USERS.txt

drwxrwxrwx   – 508           500            0 2016-04-20 13:03 /_tmp

drwxrwxrwx   – 507           500            0 2016-04-22 11:49 /app-logs

drwxrwxrwx   – 508           500            0 2015-07-06 11:56 /apps

drwxr-xr-x   – 508           500            0 2015-10-15 03:03 /hdfs

drwxr-xr-x   – 508           500            0 2016-03-24 16:45 /hdp

drwxrwxr-x   – horton-mapred 500            0 2015-11-02 12:27 /mapred

drwxrwxrwx   – horton-mapred 500            0 2014-12-12 18:11 /mr-history

drwxrwxrwx   – 508           500            0 2015-01-20 16:03 /temp

drwxrwxrwx   – 508           500            0 2016-04-21 11:19 /tmp

drwxrwxrwx   – 508           500            0 2016-04-22 11:58 /user

drwxrwxrwx   – 508           500            0 2015-07-07 10:29 /webhdfs

drwxr-xr-x   – 508           500            0 2014-12-18 18:03 /wiki

16/04/22 12:13:54 DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@6c1a5b54

16/04/22 12:13:54 DEBUG ipc.Client: removing client from cache: org.apache.hadoop.ipc.Client@6c1a5b54

16/04/22 12:13:55 DEBUG ipc.Client: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@6c1a5b54

16/04/22 12:13:55 DEBUG ipc.Client: Stopping client

16/04/22 12:13:55 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM: closed

16/04/22 12:13:55 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM: stopped, remaining connections 0