MAPREDUCE-3007. Fixed Yarn Mapreduce client to be able to connect to JobHistoryServer in secure mode. Contributed by Vinod Kumar Vavilapalli.

git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1171051 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Vinod Kumar Vavilapalli 2011-09-15 10:53:05 +00:00
parent be2b0921fa
commit cb48bc1c93
3 changed files with 76 additions and 65 deletions

View File

@ -1243,89 +1243,92 @@ Release 0.23.0 - Unreleased
MAPREDUCE-2897. Javadoc for ClientRMProtocol protocol and related records. MAPREDUCE-2897. Javadoc for ClientRMProtocol protocol and related records.
(acmurthy) (acmurthy)
MAPREDUCE-2916. Ivy build for MRv1 fails with bad organization for MAPREDUCE-2916. Ivy build for MRv1 fails with bad organization for
common daemon. (mahadev) common daemon. (mahadev)
MAPREDUCE-2917. Fixed corner case in container reservation which led to MAPREDUCE-2917. Fixed corner case in container reservation which led to
starvation and hung jobs. (acmurthy) starvation and hung jobs. (acmurthy)
MAPREDUCE-2756. Better error handling in JobControl for failed jobs. MAPREDUCE-2756. Better error handling in JobControl for failed jobs.
(Robert Evans via acmurthy) (Robert Evans via acmurthy)
MAPREDUCE-2716. MRReliabilityTest job fails because of missing MAPREDUCE-2716. MRReliabilityTest job fails because of missing
job-file. (Jeffrey Naisbitt via vinodkv) job-file. (Jeffrey Naisbitt via vinodkv)
MAPREDUCE-2882. TestLineRecordReader depends on ant jars. (todd) MAPREDUCE-2882. TestLineRecordReader depends on ant jars. (todd)
MAPREDUCE-2687. Fix NodeManager to use the right version of MAPREDUCE-2687. Fix NodeManager to use the right version of
LocalDirAllocator.getLocalPathToWrite. (mahadev & acmurthy) LocalDirAllocator.getLocalPathToWrite. (mahadev & acmurthy)
MAPREDUCE-2800. Set final progress for tasks to ensure all task information MAPREDUCE-2800. Set final progress for tasks to ensure all task information
is correctly logged to JobHistory. (Siddharth Seth via acmurthy) is correctly logged to JobHistory. (Siddharth Seth via acmurthy)
MAPREDUCE-2938. Log application submission failure in CapacityScheduler. MAPREDUCE-2938. Log application submission failure in CapacityScheduler.
(acmurthy) (acmurthy)
MAPREDUCE-2948. Hadoop streaming test failure, post MR-2767 (mahadev) MAPREDUCE-2948. Hadoop streaming test failure, post MR-2767 (mahadev)
MAPREDUCE-2908. Fix all findbugs warnings. (vinodkv via acmurthy) MAPREDUCE-2908. Fix all findbugs warnings. (vinodkv via acmurthy)
MAPREDUCE-2942. TestNMAuditLogger.testNMAuditLoggerWithIP failing (Thomas Graves MAPREDUCE-2942. TestNMAuditLogger.testNMAuditLoggerWithIP failing (Thomas Graves
via mahadev) via mahadev)
MAPREDUCE-2947. Fixed race condition in AuxiliaryServices. (vinodkv via MAPREDUCE-2947. Fixed race condition in AuxiliaryServices. (vinodkv via
acmurthy) acmurthy)
MAPREDUCE-2844. Fixed display of nodes in UI. (Ravi Teja Ch N V via MAPREDUCE-2844. Fixed display of nodes in UI. (Ravi Teja Ch N V via
acmurthy) acmurthy)
MAPREDUCE-2677. Fixed 404 for some links from HistoryServer. (Robert Evans MAPREDUCE-2677. Fixed 404 for some links from HistoryServer. (Robert Evans
via acmurthy) via acmurthy)
MAPREDUCE-2937. Ensure reason for application failure is displayed to the MAPREDUCE-2937. Ensure reason for application failure is displayed to the
user. (mahadev via acmurthy) user. (mahadev via acmurthy)
MAPREDUCE-2953. Fix a race condition on submission which caused client to MAPREDUCE-2953. Fix a race condition on submission which caused client to
incorrectly assume application was gone by making submission synchronous incorrectly assume application was gone by making submission synchronous
for RMAppManager. (Thomas Graves via acmurthy) for RMAppManager. (Thomas Graves via acmurthy)
MAPREDUCE-2963. Fix hang in TestMRJobs. (Siddharth Seth via acmurthy) MAPREDUCE-2963. Fix hang in TestMRJobs. (Siddharth Seth via acmurthy)
MAPREDUCE-2954. Fixed a deadlock in NM caused due to wrong synchronization MAPREDUCE-2954. Fixed a deadlock in NM caused due to wrong synchronization
in protocol buffer records. (Siddharth Seth via vinodkv) in protocol buffer records. (Siddharth Seth via vinodkv)
MAPREDUCE-2975. Fixed YARNRunner to use YarnConfiguration rather than MAPREDUCE-2975. Fixed YARNRunner to use YarnConfiguration rather than
Configuration. (mahadev via acmurthy) Configuration. (mahadev via acmurthy)
MAPREDUCE-2971. ant build mapreduce fails protected access jc.displayJobList MAPREDUCE-2971. ant build mapreduce fails protected access jc.displayJobList
(jobs) (Thomas Graves via mahadev) (jobs) (Thomas Graves via mahadev)
MAPREDUCE-2691. Finishing up the cleanup of distributed cache file resources MAPREDUCE-2691. Finishing up the cleanup of distributed cache file resources
and related tests. (Siddharth Seth via vinodkv) and related tests. (Siddharth Seth via vinodkv)
MAPREDUCE-2749. Ensure NM registers with RM after starting all its services MAPREDUCE-2749. Ensure NM registers with RM after starting all its services
correctly. (Thomas Graves via acmurthy) correctly. (Thomas Graves via acmurthy)
MAPREDUCE-2979. Removed the needless ClientProtocolProvider configuration MAPREDUCE-2979. Removed the needless ClientProtocolProvider configuration
from the hadoop-mapreduce-client-core module. (Siddharth Seth via vinodkv) from the hadoop-mapreduce-client-core module. (Siddharth Seth via vinodkv)
MAPREDUCE-2985. Fixed findbugs warnings in ResourceLocalizationService. MAPREDUCE-2985. Fixed findbugs warnings in ResourceLocalizationService.
(Thomas Graves via acmurthy) (Thomas Graves via acmurthy)
MAPREDUCE-2874. Fix formatting of ApplicationId in web-ui. (Eric Payne via MAPREDUCE-2874. Fix formatting of ApplicationId in web-ui. (Eric Payne via
acmurthy) acmurthy)
MAPREDUCE-2995. Better handling of expired containers in MapReduce MAPREDUCE-2995. Better handling of expired containers in MapReduce
ApplicationMaster. (vinodkv via acmurthy) ApplicationMaster. (vinodkv via acmurthy)
MAPREDUCE-2995. Fixed race condition in ContainerLauncher. (vinodkv via MAPREDUCE-2995. Fixed race condition in ContainerLauncher. (vinodkv via
acmurthy) acmurthy)
MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service
startup fails. (Ravi Teja via vinodkv) startup fails. (Ravi Teja via vinodkv)
MAPREDUCE-3005. Fix both FifoScheduler and CapacityScheduler to correctly MAPREDUCE-3005. Fix both FifoScheduler and CapacityScheduler to correctly
enforce locality constraints. (acmurthy) enforce locality constraints. (acmurthy)
MAPREDUCE-3007. Fixed Yarn Mapreduce client to be able to connect to
JobHistoryServer in secure mode. (vinodkv)
Release 0.22.0 - Unreleased Release 0.22.0 - Unreleased

View File

@ -19,6 +19,7 @@
package org.apache.hadoop.mapred; package org.apache.hadoop.mapred;
import java.io.IOException; import java.io.IOException;
import java.security.PrivilegedAction;
import java.util.HashMap; import java.util.HashMap;
import java.util.Map; import java.util.Map;
@ -28,12 +29,13 @@
import org.apache.hadoop.mapreduce.JobID; import org.apache.hadoop.mapreduce.JobID;
import org.apache.hadoop.mapreduce.v2.api.MRClientProtocol; import org.apache.hadoop.mapreduce.v2.api.MRClientProtocol;
import org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig; import org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig;
import org.apache.hadoop.mapreduce.v2.security.client.ClientHSSecurityInfo;
import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.net.NetUtils;
import org.apache.hadoop.security.SecurityInfo; import org.apache.hadoop.security.SecurityInfo;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.yarn.YarnException; import org.apache.hadoop.yarn.YarnException;
import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.ipc.YarnRPC; import org.apache.hadoop.yarn.ipc.YarnRPC;
import org.apache.hadoop.yarn.security.client.ClientRMSecurityInfo;
public class ClientCache { public class ClientCache {
@ -72,16 +74,21 @@ synchronized ClientServiceDelegate getClient(JobID jobId) {
private MRClientProtocol instantiateHistoryProxy() private MRClientProtocol instantiateHistoryProxy()
throws IOException { throws IOException {
String serviceAddr = conf.get(JHAdminConfig.MR_HISTORY_ADDRESS, final String serviceAddr = conf.get(JHAdminConfig.MR_HISTORY_ADDRESS,
JHAdminConfig.DEFAULT_MR_HISTORY_ADDRESS); JHAdminConfig.DEFAULT_MR_HISTORY_ADDRESS);
LOG.info("Connecting to HistoryServer at: " + serviceAddr); LOG.info("Connecting to HistoryServer at: " + serviceAddr);
Configuration myConf = new Configuration(conf); final Configuration myConf = new Configuration(conf);
//TODO This should ideally be using it's own class (instead of ClientRMSecurityInfo)
myConf.setClass(YarnConfiguration.YARN_SECURITY_INFO, myConf.setClass(YarnConfiguration.YARN_SECURITY_INFO,
ClientRMSecurityInfo.class, SecurityInfo.class); ClientHSSecurityInfo.class, SecurityInfo.class);
YarnRPC rpc = YarnRPC.create(myConf); final YarnRPC rpc = YarnRPC.create(myConf);
LOG.info("Connected to HistoryServer at: " + serviceAddr); LOG.info("Connected to HistoryServer at: " + serviceAddr);
return (MRClientProtocol) rpc.getProxy(MRClientProtocol.class, UserGroupInformation currentUser = UserGroupInformation.getCurrentUser();
NetUtils.createSocketAddr(serviceAddr), myConf); return currentUser.doAs(new PrivilegedAction<MRClientProtocol>() {
@Override
public MRClientProtocol run() {
return (MRClientProtocol) rpc.getProxy(MRClientProtocol.class,
NetUtils.createSocketAddr(serviceAddr), myConf);
}
});
} }
} }

View File

@ -0,0 +1 @@
org.apache.hadoop.mapreduce.v2.security.client.ClientHSSecurityInfo