Increase the kerberos logging level on a client to see what is actually going on

if you are seeing issues with kerberos based hdfs client access against an Isilon cluster, increasing the kerberos logging level on the client can show you a lot more information(a lot)

Change the logging level with the following:

export HADOOP_ROOT_LOGGER=”TRACE,console”

export HADOOP_OPTS=”-Dsun.security.krb5.debug=true”

quick example of a successful kerberized hadoop call from a kerberized hortoworks cluster against Isilon:

[root@hdp4 jvm]# su – hdpuser3

[hdpuser3@hdp4 ~]$ kinit

Password for hdpuser3@FOO.COM:

[hdpuser3@hdp4 ~]$ klist -e

Ticket cache: FILE:/tmp/krb5cc_49003

Default principal: hdpuser3@FOO.COM

Valid starting Expires Service principal

04/22/16 12:18:40 04/22/16 22:17:00 krbtgt/FOO.COM@FOO.COM

renew until 04/29/16 12:18:40, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

[hdpuser3@hdp4 ~]$ export HADOOP_ROOT_LOGGER=”TRACE,console”

[hdpuser3@hdp4 ~]$ export HADOOP_OPTS=”-Dsun.security.krb5.debug=true”

[hdpuser3@hdp4 ~]$ hadoop fs -ls /

16/04/22 12:13:50 DEBUG util.Shell: setsid exited with exit code 0

16/04/22 12:13:51 DEBUG conf.Configuration: parsing URL jar:file:/usr/hdp/2.3.4.0-3485/hadoop/hadoop-common-2.7.1.2.3.4.0-3485.jar!/core-default.xml

16/04/22 12:13:51 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@7cbd213e

16/04/22 12:13:51 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/2.3.4.0-3485/0/core-site.xml

16/04/22 12:13:51 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@5cdd8682

Java config name: null

Native config name: /etc/krb5.conf

Loaded from native config

16/04/22 12:13:52 DEBUG security.Groups: Creating new Groups object

16/04/22 12:13:52 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library…

16/04/22 12:13:52 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library

16/04/22 12:13:52 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution

16/04/22 12:13:52 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping

16/04/22 12:13:52 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000

>>>KinitOptions cache name is /tmp/krb5cc_49003

>>>DEBUG <CCacheInputStream> client principal is hdpuser3@FOO.COM

>>>DEBUG <CCacheInputStream> server principal is krbtgt/FOO.COM@FOO.COM

>>>DEBUG <CCacheInputStream> key type: 18

>>>DEBUG <CCacheInputStream> auth time: Fri Apr 22 11:56:32 EDT 2016

>>>DEBUG <CCacheInputStream> start time: Fri Apr 22 11:58:12 EDT 2016

>>>DEBUG <CCacheInputStream> end time: Fri Apr 22 21:56:32 EDT 2016

>>>DEBUG <CCacheInputStream> renew_till time: Fri Apr 29 11:58:12 EDT 2016

>>> CCacheInputStream: readFlags() FORWARDABLE; RENEWABLE; INITIAL; PRE_AUTH;

16/04/22 12:13:52 DEBUG security.UserGroupInformation: hadoop login

16/04/22 12:13:52 DEBUG security.UserGroupInformation: hadoop login commit

16/04/22 12:13:52 DEBUG security.UserGroupInformation: using kerberos user:hdpuser3@FOO.COM

16/04/22 12:13:52 DEBUG security.UserGroupInformation: Using user: “hdpuser3@FOO.COM” with name hdpuser3@FOO.COM

16/04/22 12:13:52 DEBUG security.UserGroupInformation: User entry: “hdpuser3@FOO.COM”

16/04/22 12:13:52 DEBUG security.UserGroupInformation: UGI loginUser:hdpuser3@FOO.COM (auth:KERBEROS)

16/04/22 12:13:52 DEBUG security.UserGroupInformation: Found tgt Ticket (hex) =

0000: 61 82 03 F8 30 82 03 F4 A0 03 02 01 05 A1 09 1B a…0………..

0010: 07 46 4F 4F 2E 43 4F 4D A2 1C 30 1A A0 03 02 01 .FOO.COM..0…..

0020: 02 A1 13 30 11 1B 06 6B 72 62 74 67 74 1B 07 46 …0…krbtgt..F

0030: 4F 4F 2E 43 4F 4D A3 82 03 C2 30 82 03 BE A0 03 OO.COM….0…..

0040: 02 01 12 A1 03 02 01 02 A2 82 03 B0 04 82 03 AC …………….

0050: 80 45 B8 57 D9 53 73 66 A9 20 92 30 46 2E 64 A6 .E.W.Ssf. .0F.d.

0060: 2B 42 0E 1E 71 6D 56 35 88 78 A5 CF 2D 3C 01 24 +B..qmV5.x..-<.$

0070: 49 5D FA 7C 88 31 03 0F 26 9A C1 29 63 30 67 B3 I]…1..&..)c0g.

0080: 20 AC 73 F8 35 D3 08 E9 FD 81 ED 66 C3 41 AD 4B .s.5……f.A.K

0090: 09 B6 DF B1 3B C6 FC 5F 1B A5 B5 FC A9 8A D3 A4 ….;.._……..

00A0: 5F 2B 31 73 0F 03 87 93 43 35 2E 6D 9D 4A 97 C8 _+1s….C5.m.J..

00B0: 3C 5C 45 B2 B0 21 D5 C8 B5 0A 73 10 76 92 AB 99 <\E..!….s.v…

00C0: B7 EE 63 C8 37 94 1B 57 E7 A0 57 7E 84 17 1E 04 ..c.7..W..W…..

00D0: 13 F9 5D 62 D6 0C 02 3D 46 EF 3C AE 27 D6 74 A1 ..]b…=F.<.’.t.

00E0: A9 A2 42 17 3A DA E3 DF F5 0B B0 F8 16 0F 77 59 ..B.:………wY

00F0: 84 61 DF 94 75 39 C8 6C AD 67 20 E2 EC 49 54 FE .a..u9.l.g ..IT.

0100: 14 4C 2E 9C 30 AC 1C 42 37 01 27 A0 C0 7F 7E 32 .L..0..B7.’….2

0110: DC D0 8F 14 36 A4 B7 D4 2B 0A ED DC BD AC 9E BD ….6…+…….

0120: 4A 43 C7 FB 60 B2 30 9F F7 E1 50 88 44 AC 07 06 JC..`.0…P.D…

0130: 4E 1C FE 3A 40 88 EF D3 20 82 7C 39 89 BC CA 4F N..:@… ..9…O

0140: 58 3B FD 3D A5 24 62 C9 75 C3 47 F3 E0 EA 83 B6 X;.=.$b.u.G…..

0150: 0C FA F3 25 8A BC 42 22 59 54 81 5D 25 64 E9 7E …%..B”YT.]%d..

0160: 9E 73 70 DD 24 35 E5 94 3D 71 B6 C2 14 C1 3F 5C .sp.$5..=q….?\

0170: 75 2B 0B 99 4E FC 29 40 8B B7 1D 09 A1 AA 52 41 u+..N.)@……RA

0180: BF 16 91 67 5F 21 53 68 3E 98 65 4E E2 AF FC 7C …g_!Sh>.eN….

0190: 96 70 13 74 FB B5 E5 9F 87 C5 39 49 B4 51 BA 8F .p.t……9I.Q..

01A0: F0 42 4B B4 4C F8 CF D3 5C 69 3C 53 8B 42 E8 F3 .BK.L…\i<S.B..

01B0: CC 71 F6 57 AF 5E 0F 1E 0C 63 49 0A 86 8B A6 00 .q.W.^…cI…..

01C0: 1A AF 8F B8 6E 2C 81 1E 8B B4 49 DF 61 95 77 3B ….n,….I.a.w;

01D0: A6 BB 48 4C A5 ED 03 F2 3D A2 BF 00 27 90 34 5A ..HL….=…’.4Z

01E0: 9E A9 57 D7 45 AB DC 85 71 37 D7 65 27 82 04 41 ..W.E…q7.e’..A

01F0: 5E 5D 14 40 94 AC 95 1E 4C F0 23 B1 94 96 A7 89 ^].@….L.#…..

0200: 03 6C 67 2D C0 08 18 22 2A DE FD DC 03 88 FD 3D .lg-…”*……=

0210: A4 95 C7 31 6C 28 F2 AD 2F C2 21 65 51 03 A9 8F …1l(../.!eQ…

0220: 85 C5 52 07 A3 81 05 E5 7F A4 EE 17 F5 71 C6 AC ..R……….q..

0230: F0 38 F3 EA FF 96 75 95 ED 03 2C 72 4C 3B C6 E1 .8….u…,rL;..

0240: 63 65 13 E6 45 69 F6 91 CD 39 B9 E8 CE F0 A6 96 ce..Ei…9……

0250: C0 AF 4A 43 FB 18 F8 E7 09 26 7C A5 0D 24 FB 7D ..JC…..&…$..

0260: CF C3 64 C4 1E 40 28 D7 D5 48 15 15 AA 60 8C 73 ..d..@(..H…`.s

0270: 6C 0F 36 97 46 B0 BE B6 EE C4 1F 6B 88 F5 1E F2 l.6.F……k….

0280: F8 93 04 B7 9C 1C 21 5D D6 86 4D 45 DE 57 68 49 ……!]..ME.WhI

0290: 85 D1 A2 5F 0A 13 2E 33 DB A2 2E 90 9D 4F A2 E2 …_…3…..O..

02A0: 99 CB 3A BE 4E A1 C2 B3 D5 0F 7A 56 2E BA DB 15 ..:.N…..zV….

02B0: 2D 6F DA 51 B6 E8 B8 E7 19 94 A3 7D 0E B2 31 4F -o.Q……….1O

02C0: E5 21 C4 D3 DC 11 64 DA 7E 37 89 E3 81 1F 24 C0 .!….d..7….$.

02D0: 4B 2C 11 46 A1 06 09 0B 0E 61 3A EA 59 09 83 8B K,.F…..a:.Y…

02E0: A8 6F FC 78 14 41 EB 2F 32 81 AC CA 3B 3F 5E CF .o.x.A./2…;?^.

02F0: 59 1F 48 C1 36 7A A2 82 84 6D D4 36 1F 54 9E 7C Y.H.6z…m.6.T..

0300: 20 DC DA D2 68 19 B8 DF 14 25 25 C0 E9 DC 71 56 …h….%%…qV

0310: 51 73 63 44 86 A5 33 B6 25 0A 57 49 1F 0A 02 4F QscD..3.%.WI…O

0320: 70 93 AC 52 8A BC 7A 7F 10 3C F1 D8 DB CE B7 CF p..R..z..<……

0330: FA 84 FC C0 79 10 5F 25 3D 38 B7 4F D8 EA 4D 51 ….y._%=8.O..MQ

0340: 74 D3 C0 E1 04 D1 6E 34 C1 62 4F 4F 16 86 DE E8 t…..n4.bOO….

0350: B8 5B B1 50 13 05 30 09 59 58 A6 A9 E3 7E 9F 79 .[.P..0.YX…..y

0360: 8D E9 8B 25 D9 0E DB 17 26 FF A1 AD 8B 8A 55 52 …%….&…..UR

0370: 35 9F 3C 9E 56 7B 61 A1 AC 08 E0 37 9D EA 77 D9 5.<.V.a….7..w.

0380: 28 75 3D 39 C2 08 BF E9 8B 4A 8A 60 3A 69 3E 1C (u=9…..J.`:i>.

0390: 6F E7 B9 92 96 F4 79 A5 87 0B 25 B2 11 5B FC ED o…..y…%..[..

03A0: 19 36 D5 35 94 F5 74 90 82 70 CF F8 6A BB F2 80 .6.5..t..p..j…

03B0: FC 37 01 0A 41 AE 52 0C A1 23 2F 62 4B 68 35 FF .7..A.R..#/bKh5.

03C0: 3E DD 12 7E 27 5F 1D AF 66 52 D0 8C 06 60 46 5C >…’_..fR…`F\

03D0: 08 BF EA 5A AD F1 BD 14 50 7A AF 48 1C 48 91 0D …Z….Pz.H.H..

03E0: D8 0E 2E 44 9D 92 6E 0C AA 0F C1 E5 6E 18 69 BB …D..n…..n.i.

03F0: 4F 90 EE F7 22 9F F0 DE F1 E9 69 4D O…”…..iM

Client Principal = hdpuser3@FOO.COM

Server Principal = krbtgt/FOO.COM@FOO.COM

Session Key = EncryptionKey: keyType=18 keyBytes (hex dump)=

0000: 40 19 60 17 41 7F 9B A6 FA 6E D2 93 A6 F0 B3 2B @.`.A….n…..+

0010: 20 BC 93 4D 72 E2 23 75 AF DA CF 5B C7 48 09 C6 ..Mr.#u…[.H..

Forwardable Ticket true

Forwarded Ticket false

Proxiable Ticket false

Proxy Ticket false

Postdated Ticket false

Renewable Ticket true

Initial Ticket true

Auth Time = Fri Apr 22 11:56:32 EDT 2016

Start Time = Fri Apr 22 11:58:12 EDT 2016

End Time = Fri Apr 22 21:56:32 EDT 2016

Renew Till = Fri Apr 29 11:58:12 EDT 2016

Client Addresses Null

16/04/22 12:13:52 DEBUG security.UserGroupInformation: Current time is 1461341632663

16/04/22 12:13:52 DEBUG security.UserGroupInformation: Next refresh is 1461369412000

16/04/22 12:13:52 TRACE tracing.SpanReceiverHost: No span receiver names found in dfs.client.htrace.spanreceiver.classes.

16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false

16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = true

16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false

16/04/22 12:13:53 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = /var/lib/hadoop-hdfs/dn_socket

16/04/22 12:13:53 DEBUG retry.RetryUtils: multipleLinearRandomRetry = null

16/04/22 12:13:53 DEBUG ipc.Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@18df8434

16/04/22 12:13:53 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@6c1a5b54

16/04/22 12:13:54 DEBUG unix.DomainSocketWatcher: org.apache.hadoop.net.unix.DomainSocketWatcher$2@2a2a8820: starting with interruptCheckPeriodMs = 60000

16/04/22 12:13:54 TRACE unix.DomainSocketWatcher: DomainSocketWatcher(1426041738): adding notificationSocket 351, connected to 350

16/04/22 12:13:54 DEBUG shortcircuit.DomainSocketFactory: The short-circuit local reads feature is enabled.

16/04/22 12:13:54 DEBUG sasl.DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection

16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Call -> rip1-horton2.foo.com/172.16.201.73:8020: getFileInfo {src: “/”}

16/04/22 12:13:54 DEBUG ipc.Client: The ping interval is 60000 ms.

16/04/22 12:13:54 DEBUG ipc.Client: Connecting to rip1-horton2.foo.com/172.16.201.73:8020

16/04/22 12:13:54 DEBUG security.UserGroupInformation: PrivilegedAction as:hdpuser3@FOO.COM (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:722)

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE

16/04/22 12:13:54 DEBUG azure.NativeAzureFileSystem: finalize() called.

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: NEGOTIATE

auths {

method: “TOKEN”

mechanism: “DIGEST-MD5”

protocol: “default”

challenge: “nonce=\”p0L557HD2nzgyagR7F5o3hTweljDX9n6kd7M0YAT368=\”,realm=\”rip1-1\”,qop=\”auth,auth-int,auth-conf\”,cipher=\”rc4-40,rc4-56,rc4,des,3des\”,maxbuf=65536,charset=utf-8,algorithm=md5-sess”

}

auths {

method: “KERBEROS”

mechanism: “GSSAPI”

protocol: “hdfs”

serverId: “rip1.foo.com”

}

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:@org.apache.hadoop.security.token.TokenInfo(value=class org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSelector)

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=, serverPrincipal=dfs.namenode.kerberos.principal)

16/04/22 12:13:54 DEBUG security.SaslRpcClient: RPC Server’s Kerberos principal name for protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is hdfs/rip1.foo.com@FOO.COM

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Creating SASL GSSAPI(KERBEROS) client to authenticate to service at rip1.foo.com

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Use KERBEROS authentication for protocol ClientNamenodeProtocolPB

Found ticket for hdpuser3@FOO.COM to go to krbtgt/FOO.COM@FOO.COM expiring on Fri Apr 22 21:56:32 EDT 2016

Entered Krb5Context.initSecContext with state=STATE_NEW

Found ticket for hdpuser3@FOO.COM to go to krbtgt/FOO.COM@FOO.COM expiring on Fri Apr 22 21:56:32 EDT 2016

Service ticket not found in the subject

>>> Credentials acquireServiceCreds: same realm

Using builtin default etypes for default_tgs_enctypes

default etypes for default_tgs_enctypes: 18 17 16 23.

>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType

>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType

>>> KdcAccessibility: reset

>>> KrbKdcReq send: kdc=rs-2008r2-dc.foo.com UDP:88, timeout=30000, number of retries =3, #bytes=1336

>>> KDCCommunication: kdc=rs-2008r2-dc.foo.com UDP:88, timeout=30000,Attempt =1, #bytes=1336

>>> KrbKdcReq send: #bytes read=1304

>>> KdcAccessibility: remove rs-2008r2-dc.foo.com

>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType

>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000

>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType

Krb5Context setting mySeqNumber to: 634089161

Created InitSecContextToken:

0000: 01 00 6E 82 04 D7 30 82 04 D3 A0 03 02 01 05 A1 ..n…0………

0010: 03 02 01 0E A2 07 03 05 00 20 00 00 00 A3 82 03 ……… ……

0020: EC 61 82 03 E8 30 82 03 E4 A0 03 02 01 05 A1 09 .a…0……….

0030: 1B 07 46 4F 4F 2E 43 4F 4D A2 1F 30 1D A0 03 02 ..FOO.COM..0….

0040: 01 00 A1 16 30 14 1B 04 68 64 66 73 1B 0C 72 69 ….0…hdfs..ri

0050: 70 31 2E 66 6F 6F 2E 63 6F 6D A3 82 03 AF 30 82 p1.foo.com….0.

0060: 03 AB A0 03 02 01 12 A1 03 02 01 04 A2 82 03 9D …………….

0070: 04 82 03 99 DC D0 4F F8 8C 20 BB BC 88 B5 42 24 ……O.. ….B$

0080: E7 FD B3 D5 E3 D7 CA A6 62 8C B7 E6 88 61 A3 B5 ……..b….a..

0090: B6 ED E9 61 FC 19 00 B9 A6 8C 63 26 E6 F5 87 25 …a……c&…%

00A0: 85 B6 0F 31 9B A6 49 8D B2 95 5B 74 39 9C 6C 0A …1..I…[t9.l.

00B0: 2C 6D 1A E7 BC 4B 0F FA 4B DA C0 F7 81 83 C7 95 ,m…K..K…….

00C0: AD A8 66 33 18 30 3E 19 CC AB 7B F5 6D 18 EF CC ..f3.0>…..m…

00D0: AB 05 8B F2 00 1A 3A 8E B3 E0 40 34 AC 6C 63 E8 ……:…@4.lc.

00E0: 9A 9A 62 13 87 81 48 15 97 2A 7F 8D C3 04 2B D9 ..b…H..*….+.

00F0: C5 10 7C 40 13 18 5A 93 1F D4 1B 23 7E 9F 6A 6C …@..Z….#..jl

0100: AE 05 10 FC 4C 52 5F BD E9 4B 19 03 DE 08 AD 22 ….LR_..K…..”

0110: 7F 8E 95 D6 07 D5 65 FA 4A F2 9E 1B E8 08 20 D3 ……e.J….. .

0120: 54 A1 49 98 6C 92 4B B4 31 B5 32 6D 43 A1 0D 56 T.I.l.K.1.2mC..V

0130: 81 92 48 AA F4 4F F4 7D 9E 9C 7B 56 56 AD 61 89 ..H..O…..VV.a.

0140: 70 24 05 6C 75 00 F2 28 45 6A 80 A0 E6 0D 67 45 p$.lu..(Ej….gE

0150: A2 DD F9 AA 90 36 AB 5D 50 9A 58 F7 4A 15 0C 88 …..6.]P.X.J…

0160: 8C FF AB B9 36 1E 40 82 28 7B 88 BF 4C 7C A6 FE ….6.@.(…L…

0170: F1 1C 61 E4 31 8A 5D 51 41 A8 78 B1 E6 36 74 65 ..a.1.]QA.x..6te

0180: D3 24 75 32 7F 42 58 37 B7 93 48 56 93 74 15 3F .$u2.BX7..HV.t.?

0190: FF F5 94 7B 6D 19 15 DF 24 5A D4 75 12 BA BC 13 ….m…$Z.u….

01A0: 9E E7 DB 1C CD 36 90 A2 3B 48 D1 85 01 5B 5C CC …..6..;H…[\.

01B0: 72 B0 37 9E 69 D6 C5 80 2E 12 FD FF 9D 9E 6D 8F r.7.i………m.

01C0: 25 57 E5 E2 75 4E 89 54 8A A7 44 1D 11 8F D8 F5 %W..uN.T..D…..

01D0: C7 19 33 1E FD 15 8B 57 74 EE 53 77 C3 7D A5 3F ..3….Wt.Sw…?

01E0: F7 39 76 2A A7 4F DF E5 88 8B E5 02 D1 E7 AF 7E .9v*.O……….

01F0: 3A 00 7D 83 A9 34 68 5F 1F F7 87 86 D5 D3 9E C0 :….4h_……..

0200: F3 DA 3A C0 7F 46 5B CF 47 79 F0 7B 20 73 68 4D ..:..F[.Gy.. shM

0210: 56 88 3E B4 76 25 FD 47 2C 4B 37 38 15 1A E5 BE V.>.v%.G,K78….

0220: FC AB 48 C3 68 53 F7 C7 A1 D0 1D C2 D5 1A 38 2A ..H.hS……..8*

0230: 4F 4B 71 34 FD 2B C6 69 57 4B 2A D8 32 1C 54 72 OKq4.+.iWK*.2.Tr

0240: CB C7 1B A0 EF D9 7B DE 75 7C AC C5 38 3A 4D 45 ……..u…8:ME

0250: 19 78 2E B5 C1 80 9B F0 A7 F1 AC 54 9B 43 7E 10 .x………T.C..

0260: 06 40 28 F1 76 AF 3E D8 4F 49 91 AF B0 FD A3 1F .@(.v.>.OI……

0270: 4B CF 6D 64 91 78 F9 3F CC 2B 68 B5 E2 8B 8C A5 K.md.x.?.+h…..

0280: 69 AA 29 67 3A 48 0E 4E 93 91 E9 B0 8A FE 00 F5 i.)g:H.N……..

0290: 12 B3 8C 07 FE 36 57 6C 68 D8 78 F5 F8 6B 73 0A …..6Wlh.x..ks.

02A0: 7D 57 E6 ED E2 A4 6A 46 91 75 2C B3 D5 43 D7 81 .W….jF.u,..C..

02B0: 5B 77 D6 92 08 A9 F6 BB B6 57 80 2E D6 37 F8 73 [w…….W…7.s

02C0: BC A3 5C 82 34 2A CD 03 97 74 FB 14 9A 18 B9 2F ..\.4*…t…../

02D0: D4 01 37 83 1F 6A 99 31 7E E5 EE 7A 97 B8 D7 81 ..7..j.1…z….

02E0: 77 04 D5 81 01 09 96 53 97 22 CB 65 48 29 F7 18 w……S.”.eH)..

02F0: DE DA CB 55 3E BC D7 FC BB 0B 49 91 BC 4E 10 21 …U>…..I..N.!

0300: F8 D4 4E D8 17 77 7B B6 BD 41 23 DB 38 5A 59 D9 ..N..w…A#.8ZY.

0310: E9 E8 E4 8E 23 C5 D6 56 3E 88 79 5B F4 24 63 96 ….#..V>.y[.$c.

0320: 19 76 F2 B4 23 97 E3 28 8C 9D F5 CB 37 6A 99 4C .v..#..(….7j.L

0330: DB 3B 23 0D A8 F2 42 33 0C 35 5D D9 EE 5B 75 04 .;#…B3.5]..[u.

0340: A7 D4 94 5F 49 9A A3 06 15 53 E2 2C 21 1F F9 CC …_I….S.,!…

0350: 51 83 58 18 8E CB BD 36 09 50 5B 90 22 8C 53 93 Q.X….6.P[.”.S.

0360: 87 B2 2E A5 55 0E 09 02 B7 5A 4D B7 53 58 19 BF ….U….ZM.SX..

0370: 6C 6F B3 10 F3 E6 59 69 FF 66 C8 56 1A AA 53 22 lo….Yi.f.V..S”

0380: F0 5D B8 22 16 5E AD C0 B0 34 F1 04 9B 64 96 26 .].”.^…4…d.&

0390: 5A 04 7F 0A 9B 30 9E 0E 9B 83 27 83 18 0E E9 2E Z….0….’…..

03A0: 42 67 B3 53 87 78 12 12 A9 14 5A 05 ED 6A 43 41 Bg.S.x….Z..jCA

03B0: BE 4E A9 91 68 6C 1B 89 74 14 C7 86 C2 47 FE 30 .N..hl..t….G.0

03C0: 6C 42 D9 59 8F A6 1A 50 40 07 70 11 DE B2 0E D1 lB.Y…P@.p…..

03D0: CE 3C 8F D1 DD F7 C4 12 CD B8 4B 0A 99 B4 EF BB .<……..K…..

03E0: FB 4A 7B EE 9C E2 37 E0 06 EC 14 13 F1 F9 44 71 .J….7…….Dq

03F0: A3 F6 FE 6D 40 46 B4 4E D2 25 5A E0 5E 9B 1B 28 …m@F.N.%Z.^..(

0400: 9D 91 C1 2E 07 E2 7B CE 45 D2 5B 84 56 A4 81 CD ……..E.[.V…

0410: 30 81 CA A0 03 02 01 12 A2 81 C2 04 81 BF 1A 70 0…………..p

0420: A8 2D 8B 91 A7 12 8F 2C 1A B6 F1 E1 E2 F1 E4 C4 .-…..,……..

0430: F1 D1 14 77 8E 5A 73 DC 42 A9 F4 D5 5A 90 35 9A …w.Zs.B…Z.5.

0440: 95 BD 99 B6 D7 7A 2D F8 3D 36 71 5C E3 11 76 A6 …..z-.=6q\..v.

0450: 0F 62 D8 30 8F E9 B1 58 EE 90 31 ED FF 3A BF 53 .b.0…X..1..:.S

0460: FA BE 45 AE A3 E6 9D 30 58 39 8E C3 11 B5 63 F0 ..E….0X9….c.

0470: 12 A7 B6 BC FA D9 75 D9 35 0A 81 AB 53 89 75 4C ……u.5…S.uL

0480: EF 15 90 3D 4C F0 C5 D3 D2 A2 C8 0E 08 F7 0C 42 …=L……….B

0490: C3 91 64 F2 2B 35 59 02 34 80 64 44 38 3F 52 C7 ..d.+5Y.4.dD8?R.

04A0: 3C ED 01 07 6C FC 60 42 A4 8D D7 FD D0 83 31 FA <…l.`B……1.

04B0: 54 57 09 BA 5F 05 EB E7 3E DA 0A 2A 0F 49 B1 4D TW.._…>..*.I.M

04C0: C8 57 5E 5E 6F 57 32 B5 54 FC 55 B0 F1 06 16 32 .W^^oW2.T.U….2

04D0: B7 19 01 EE 66 EA 3F 52 07 9B 2B D9 AF ….f.?R..+..

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: INITIATE

token: “`\202\004\350\006\t*\206H\206\367\022\001\002\002\001\000n\202\004\3270\202\004\323\240\003\002\001\005\241\003\002\001\016\242\a\003\005\000 \000\000\000\243\202\003\354a\202\003\3500\202\003\344\240\003\002\001\005\241\t\033\aFOO.COM\242\0370\035\240\003\002\001\000\241\0260\024\033\004hdfs\033\frip1.foo.com\243\202\003\2570\202\003\253\240\003\002\001\022\241\003\002\001\004\242\202\003\235\004\202\003\231\334\320O\370\214 \273\274\210\265B$\347\375\263\325\343\327\312\246b\214\267\346\210a\243\265\266\355\351a\374\031\000\271\246\214c&\346\365\207%\205\266\0171\233\246I\215\262\225[t9\234l\n,m\032\347\274K\017\372K\332\300\367\201\203\307\225\255\250f3\0300>\031\314\253{\365m\030\357\314\253\005\213\362\000\032:\216\263\340@4\254lc\350\232\232b\023\207\201H\025\227\215\303\004+\331\305\020|@\023\030Z\223\037\324\033#~\237jl\256\005\020\374LR_\275\351K\031\003\336\b\255\\216\225\326\a\325e\372J\362\236\033\350\b \323T\241I\230l\222K\2641\2652mC\241\rV\201\222H\252\364O\364}\236\234{VV\255a\211p$\005lu\000\362(Ej\200\240\346\rgE\242\335\371\252\2206\253]P\232X\367J\025\f\210\214\377\253\2716\036@\202({\210\277L|\246\376\361\034a\3441\212]QA\250x\261\3466te\323$uBX7\267\223HV\223t\025?\377\365\224{m\031\025\337$Z\324u\022\272\274\023\236\347\333\034\3156\220\242;H\321\205\001[\\\314r\2607\236i\326\305\200.\022\375\377\235\236m\217%W\345\342uN\211T\212\247D\035\021\217\330\365\307\0313\036\375\025\213Wt\356Sw\303}\245?\3679v*\247O\337\345\210\213\345\002\321\347\257~:\000}\203\2514h_\037\367\207\206\325\323\236\300\363\332:\30F[\317Gy\360{ shMV\210>\264v%\375G,K78\025\032\345\276\374\253H\303hS\367\307\241\320\035\302\325\0328*OKq4\375+\306iWK*\3302\034Tr\313\307\033\240\357\331{\336u|\254\3058:ME\031x.\265\301\200\233\360\247\361\254T\233C~\020\006@(\361v\257>\330OI\221\257\260\375\243\037K\317md\221x\371?\314+h\265\342\213\214\245i\252)g:H\016N\223\221\351\260\212\376\000\365\022\263\214\a\3766Wlh\330x\365\370ks\n}W\346\355\342\244jF\221u,\263\325C\327\201[w\326\222\b\251\366\273\266W\200.\3267\370s\274\243\\\2024*\315\003\227t\373\024\232\030\271/\324\0017\203\037j\2311~\345\356z\227\270\327\201w\004\325\201\001\t\226S\227\”\313eH)\367\030\336\332\313U>\274\327\374\273\vI\221\274N\020!\370\324N\330\027w{\266\275A#\3338ZY\331\351\350\344\216#\305\326V>\210y[\364$c\226\031v\362\264#\227\343(\214\235\365\3137j\231L\333;#\r\250\362B3\f5]\331\356[u\004\247\324\224_I\232\243\006\025S\342,!\037\371\314Q\203X\030\216\313\2756\tP[\220\”\214S\223\207\262.\245U\016\t\002\267ZM\267SX\031\277lo\263\020\363\346Yi\377f\310V\032\252S\”\360]\270\”\026^\255\300\2604\361\004\233d\226&Z\00\n\2330\236\016\233\203\’\203\030\016\351.Bg\263S\207x\022\022\251\024Z\005\355jCA\276N\251\221hl\033\211t\024\307\206\302G\3760lB\331Y\217\246\032P@\ap\021\336\262\016\321\316<\217\321\335\367\304\022\315\270K\n\231\264\357\273\373J{\356\234\3427\340\006\354\024\023\361\371Dq\243\366\376m@F\264N\322%Z\340^\233\033(\235\221\301.\a\342{\316E\322[\204V\244\201\3150\201\312\240\003\002\001\022\242\201\302\004\201\277\032p\250-\213\221\247\022\217,\032\266\361\341\342\361\344\304\361\321\024w\216Zs\334B\251\364\325Z\2205\232\225\275\231\266\327z-\370=6q\\\343\021v\246\017b\3300\217\351\261X\356\2201\355\377:\277S\372\276E\256\243\346\2350X9\216\303\021\265c\360\022\247\266\274\372\331u\3315\n\201\253S\211uL\357\025\220=L\360\305\323\322\242\310\016\b\367\fB\303\221d\362+5Y\0024\200dD8?R\307<\355\001\al\374`B\244\215\327\375\320\2031\372TW\t\272_\005\353\347>\332\n*\017I\261M\310W^^oW2\265T\374U\260\361\006\0262\267\031\001\356f\352?R\a\233+\331\257”

auths {

method: “KERBEROS”

mechanism: “GSSAPI”

protocol: “hdfs”

serverId: “rip1.foo.com”

}

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: CHALLENGE

token: “`\201\231\006\t*\206H\206\367\022\001\002\002\002\000o\201\2110\201\206\240\003\002\001\005\241\003\002\001\017\242z0x\240\003\002\001\022\242q\004o\t`\313Hu\3150\231z\345\333\3508\302x\n\337[H\357\226\b\r\377O\350\246A\240\221\223\355\243@\2651V\032\330\251A\314\361!\270UO\303L\030|z\342\341O\3050\000\310!\231V\316\204)>9c\345\326\346\300\253X\303b6\266-^\217%J\224\246W\270\330\313g/\205,h,c\037\306Z\273{B\003\260CZ\004\247S\252\331”

Entered Krb5Context.initSecContext with state=STATE_IN_PROCESS

>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType

Krb5Context setting peerSeqNumber to: 254781051

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE

token: “”

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: CHALLENGE

token: “\005\004\005\377\000\f\000\000\000\000\000\000\017/\246{\a\001\000\000\201`y\240\327\375\271K\033\027\227C”

Krb5Context.unwrap: token=[05 04 05 ff 00 0c 00 00 00 00 00 00 0f 2f a6 7b 07 01 00 00 81 60 79 a0 d7 fd b9 4b 1b 17 97 43 ]

Krb5Context.unwrap: data=[07 01 00 00 ]

Krb5Context.wrap: data=[01 01 00 00 ]

Krb5Context.wrap: token=[05 04 04 ff 00 0c 00 00 00 00 00 00 25 cb 6e c9 01 01 00 00 5b d4 02 3e c7 c8 71 42 27 10 b8 3d ]

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE

token: “\005\004\004\377\000\f\000\000\000\000\000\000%\313n\311\001\001\000\000[\324\002>\307\310qB\’\020\270=”

16/04/22 12:13:54 DEBUG security.SaslRpcClient: Received SASL message state: SUCCESS

16/04/22 12:13:54 DEBUG ipc.Client: Negotiated QOP is :auth

16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM: starting, having connections 1

16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM sending #0

16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM got value #0

16/04/22 12:13:54 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 691ms

16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Response <- rip1-horton2.foo.com/172.16.201.73:8020: getFileInfo {fs { fileType: IS_DIR path: “” length: 0 permission { perm: 509 } owner: “1002” group: “500” modification_time: 1461254680300 access_time: 1461254680300 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410103 childrenNum: -1 }}

16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Call -> rip1-horton2.foo.com/172.16.201.73:8020: getListing {src: “/” startAfter: “” needLocation: false}

16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM sending #1

16/04/22 12:13:54 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM got value #1

16/04/22 12:13:54 DEBUG ipc.ProtobufRpcEngine: Call: getListing took 13ms

16/04/22 12:13:54 TRACE ipc.ProtobufRpcEngine: 1: Response <- rip1-horton2.foo.com/172.16.201.73:8020: getListing {dirList { partialListing { fileType: IS_DIR path: “mr-history” length: 0 permission { perm: 511 } owner: “horton-mapred” group: “500” modification_time: 1418425889000 access_time: 1418425889000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410104 childrenNum: -1 } partialListing { fileType: IS_DIR path: “user” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1461340729681 access_time: 1461340729681 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410158 childrenNum: -1 } partialListing { fileType: IS_DIR path: “wiki” length: 0 permission { perm: 493 } owner: “508” group: “500” modification_time: 1418943800000 access_time: 1418943800000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410547 childrenNum: -1 } partialListing { fileType: IS_DIR path: “tmp” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1461251997618 access_time: 1461251997618 symlink: “” block_replication: 0 blocksize: 0 fileId: 4301390448 childrenNum: -1 } partialListing { fileType: IS_DIR path: “webhdfs” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1436279388000 access_time: 1436279388000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410575 childrenNum: -1 } partialListing { fileType: IS_FILE path: “ID.txt” length: 3082 permission { perm: 420 } owner: “root” group: “wheel” modification_time: 1461254430635 access_time: 1461254430635 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4303225170 childrenNum: 0 } partialListing { fileType: IS_DIR path: “mapred” length: 0 permission { perm: 509 } owner: “horton-mapred” group: “500” modification_time: 1446485241000 access_time: 1446485241000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410577 childrenNum: -1 } partialListing { fileType: IS_DIR path: “temp” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1421787820000 access_time: 1421787820000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296934115 childrenNum: -1 } partialListing { fileType: IS_FILE path: “USERS.txt” length: 17432 permission { perm: 420 } owner: “root” group: “wheel” modification_time: 1461092279350 access_time: 1461092279350 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4300669033 childrenNum: 0 } partialListing { fileType: IS_FILE path: “ID2.txt” length: 344 permission { perm: 420 } owner: “root” group: “wheel” modification_time: 1461255046159 access_time: 1461255046159 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4302307618 childrenNum: 0 } partialListing { fileType: IS_DIR path: “hdfs” length: 0 permission { perm: 493 } owner: “508” group: “500” modification_time: 1444892583000 access_time: 1444892583000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296934116 childrenNum: -1 } partialListing { fileType: IS_DIR path: “apps” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1436198205000 access_time: 1436198205000 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296934022 childrenNum: -1 } partialListing { fileType: IS_DIR path: “app-logs” length: 0 permission { perm: 511 } owner: “507” group: “500” modification_time: 1461340151745 access_time: 1461340151745 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296279240 childrenNum: -1 } partialListing { fileType: IS_DIR path: “_tmp” length: 0 permission { perm: 511 } owner: “508” group: “500” modification_time: 1461171839874 access_time: 1461171839874 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410553 childrenNum: -1 } partialListing { fileType: IS_FILE path: “ThisIsIsilon.txt” length: 0 permission { perm: 420 } owner: “root” group: “500” modification_time: 1418689161466 access_time: 1418689161466 symlink: “” block_replication: 3 blocksize: 134217728 fileId: 4296279310 childrenNum: 0 } partialListing { fileType: IS_DIR path: “hdp” length: 0 permission { perm: 493 } owner: “508” group: “500” modification_time: 1458852305525 access_time: 1458852305525 symlink: “” block_replication: 0 blocksize: 0 fileId: 4296410635 childrenNum: -1 } remainingEntries: 0 }}

Found 16 items

-rw-r–r– 3 root wheel 3082 2016-04-21 12:00 /ID.txt

-rw-r–r– 3 root wheel 344 2016-04-21 12:10 /ID2.txt

-rw-r–r– 3 root 500 0 2014-12-15 19:19 /ThisIsIsilon.txt

-rw-r–r– 3 root wheel 17432 2016-04-19 14:57 /USERS.txt

drwxrwxrwx – 508 500 0 2016-04-20 13:03 /_tmp

drwxrwxrwx – 507 500 0 2016-04-22 11:49 /app-logs

drwxrwxrwx – 508 500 0 2015-07-06 11:56 /apps

drwxr-xr-x – 508 500 0 2015-10-15 03:03 /hdfs

drwxr-xr-x – 508 500 0 2016-03-24 16:45 /hdp

drwxrwxr-x – horton-mapred 500 0 2015-11-02 12:27 /mapred

drwxrwxrwx – horton-mapred 500 0 2014-12-12 18:11 /mr-history

drwxrwxrwx – 508 500 0 2015-01-20 16:03 /temp

drwxrwxrwx – 508 500 0 2016-04-21 11:19 /tmp

drwxrwxrwx – 508 500 0 2016-04-22 11:58 /user

drwxrwxrwx – 508 500 0 2015-07-07 10:29 /webhdfs

drwxr-xr-x – 508 500 0 2014-12-18 18:03 /wiki

16/04/22 12:13:54 DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@6c1a5b54

16/04/22 12:13:54 DEBUG ipc.Client: removing client from cache: org.apache.hadoop.ipc.Client@6c1a5b54

16/04/22 12:13:55 DEBUG ipc.Client: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@6c1a5b54

16/04/22 12:13:55 DEBUG ipc.Client: Stopping client

16/04/22 12:13:55 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM: closed

16/04/22 12:13:55 DEBUG ipc.Client: IPC Client (1217875525) connection to rip1-horton2.foo.com/172.16.201.73:8020 from hdpuser3@FOO.COM: stopped, remaining connections 0

datathatisbigdotcom

my miscellaneous ramblings on technology focused on big data, isilon and scaleio, i publish content on the DELL EMC community and a lot of my posts link out to the original

Increase the kerberos logging level on a client to see what is actually going on

Leave a Reply Cancel reply

Share this:

Like this:

Leave a Reply Cancel reply