MAPREDUCE-3007. Fixed Yarn Mapreduce client to be able to connect to JobHistoryServer in secure mode. Contributed by Vinod Kumar Vavilapalli.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1171051 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
parent
be2b0921fa
commit
cb48bc1c93
@ -1243,89 +1243,92 @@ Release 0.23.0 - Unreleased
|
|||||||
MAPREDUCE-2897. Javadoc for ClientRMProtocol protocol and related records.
|
MAPREDUCE-2897. Javadoc for ClientRMProtocol protocol and related records.
|
||||||
(acmurthy)
|
(acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2916. Ivy build for MRv1 fails with bad organization for
|
MAPREDUCE-2916. Ivy build for MRv1 fails with bad organization for
|
||||||
common daemon. (mahadev)
|
common daemon. (mahadev)
|
||||||
|
|
||||||
MAPREDUCE-2917. Fixed corner case in container reservation which led to
|
MAPREDUCE-2917. Fixed corner case in container reservation which led to
|
||||||
starvation and hung jobs. (acmurthy)
|
starvation and hung jobs. (acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2756. Better error handling in JobControl for failed jobs.
|
MAPREDUCE-2756. Better error handling in JobControl for failed jobs.
|
||||||
(Robert Evans via acmurthy)
|
(Robert Evans via acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2716. MRReliabilityTest job fails because of missing
|
MAPREDUCE-2716. MRReliabilityTest job fails because of missing
|
||||||
job-file. (Jeffrey Naisbitt via vinodkv)
|
job-file. (Jeffrey Naisbitt via vinodkv)
|
||||||
|
|
||||||
MAPREDUCE-2882. TestLineRecordReader depends on ant jars. (todd)
|
MAPREDUCE-2882. TestLineRecordReader depends on ant jars. (todd)
|
||||||
|
|
||||||
MAPREDUCE-2687. Fix NodeManager to use the right version of
|
MAPREDUCE-2687. Fix NodeManager to use the right version of
|
||||||
LocalDirAllocator.getLocalPathToWrite. (mahadev & acmurthy)
|
LocalDirAllocator.getLocalPathToWrite. (mahadev & acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2800. Set final progress for tasks to ensure all task information
|
MAPREDUCE-2800. Set final progress for tasks to ensure all task information
|
||||||
is correctly logged to JobHistory. (Siddharth Seth via acmurthy)
|
is correctly logged to JobHistory. (Siddharth Seth via acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2938. Log application submission failure in CapacityScheduler.
|
MAPREDUCE-2938. Log application submission failure in CapacityScheduler.
|
||||||
(acmurthy)
|
(acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2948. Hadoop streaming test failure, post MR-2767 (mahadev)
|
MAPREDUCE-2948. Hadoop streaming test failure, post MR-2767 (mahadev)
|
||||||
|
|
||||||
MAPREDUCE-2908. Fix all findbugs warnings. (vinodkv via acmurthy)
|
MAPREDUCE-2908. Fix all findbugs warnings. (vinodkv via acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2942. TestNMAuditLogger.testNMAuditLoggerWithIP failing (Thomas Graves
|
MAPREDUCE-2942. TestNMAuditLogger.testNMAuditLoggerWithIP failing (Thomas Graves
|
||||||
via mahadev)
|
via mahadev)
|
||||||
|
|
||||||
MAPREDUCE-2947. Fixed race condition in AuxiliaryServices. (vinodkv via
|
MAPREDUCE-2947. Fixed race condition in AuxiliaryServices. (vinodkv via
|
||||||
acmurthy)
|
acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2844. Fixed display of nodes in UI. (Ravi Teja Ch N V via
|
MAPREDUCE-2844. Fixed display of nodes in UI. (Ravi Teja Ch N V via
|
||||||
acmurthy)
|
acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2677. Fixed 404 for some links from HistoryServer. (Robert Evans
|
MAPREDUCE-2677. Fixed 404 for some links from HistoryServer. (Robert Evans
|
||||||
via acmurthy)
|
via acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2937. Ensure reason for application failure is displayed to the
|
MAPREDUCE-2937. Ensure reason for application failure is displayed to the
|
||||||
user. (mahadev via acmurthy)
|
user. (mahadev via acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2953. Fix a race condition on submission which caused client to
|
MAPREDUCE-2953. Fix a race condition on submission which caused client to
|
||||||
incorrectly assume application was gone by making submission synchronous
|
incorrectly assume application was gone by making submission synchronous
|
||||||
for RMAppManager. (Thomas Graves via acmurthy)
|
for RMAppManager. (Thomas Graves via acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2963. Fix hang in TestMRJobs. (Siddharth Seth via acmurthy)
|
MAPREDUCE-2963. Fix hang in TestMRJobs. (Siddharth Seth via acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2954. Fixed a deadlock in NM caused due to wrong synchronization
|
MAPREDUCE-2954. Fixed a deadlock in NM caused due to wrong synchronization
|
||||||
in protocol buffer records. (Siddharth Seth via vinodkv)
|
in protocol buffer records. (Siddharth Seth via vinodkv)
|
||||||
|
|
||||||
MAPREDUCE-2975. Fixed YARNRunner to use YarnConfiguration rather than
|
MAPREDUCE-2975. Fixed YARNRunner to use YarnConfiguration rather than
|
||||||
Configuration. (mahadev via acmurthy)
|
Configuration. (mahadev via acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2971. ant build mapreduce fails protected access jc.displayJobList
|
MAPREDUCE-2971. ant build mapreduce fails protected access jc.displayJobList
|
||||||
(jobs) (Thomas Graves via mahadev)
|
(jobs) (Thomas Graves via mahadev)
|
||||||
|
|
||||||
MAPREDUCE-2691. Finishing up the cleanup of distributed cache file resources
|
MAPREDUCE-2691. Finishing up the cleanup of distributed cache file resources
|
||||||
and related tests. (Siddharth Seth via vinodkv)
|
and related tests. (Siddharth Seth via vinodkv)
|
||||||
|
|
||||||
MAPREDUCE-2749. Ensure NM registers with RM after starting all its services
|
MAPREDUCE-2749. Ensure NM registers with RM after starting all its services
|
||||||
correctly. (Thomas Graves via acmurthy)
|
correctly. (Thomas Graves via acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2979. Removed the needless ClientProtocolProvider configuration
|
MAPREDUCE-2979. Removed the needless ClientProtocolProvider configuration
|
||||||
from the hadoop-mapreduce-client-core module. (Siddharth Seth via vinodkv)
|
from the hadoop-mapreduce-client-core module. (Siddharth Seth via vinodkv)
|
||||||
|
|
||||||
MAPREDUCE-2985. Fixed findbugs warnings in ResourceLocalizationService.
|
MAPREDUCE-2985. Fixed findbugs warnings in ResourceLocalizationService.
|
||||||
(Thomas Graves via acmurthy)
|
(Thomas Graves via acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2874. Fix formatting of ApplicationId in web-ui. (Eric Payne via
|
MAPREDUCE-2874. Fix formatting of ApplicationId in web-ui. (Eric Payne via
|
||||||
acmurthy)
|
acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2995. Better handling of expired containers in MapReduce
|
MAPREDUCE-2995. Better handling of expired containers in MapReduce
|
||||||
ApplicationMaster. (vinodkv via acmurthy)
|
ApplicationMaster. (vinodkv via acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2995. Fixed race condition in ContainerLauncher. (vinodkv via
|
MAPREDUCE-2995. Fixed race condition in ContainerLauncher. (vinodkv via
|
||||||
acmurthy)
|
acmurthy)
|
||||||
|
|
||||||
MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service
|
MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service
|
||||||
startup fails. (Ravi Teja via vinodkv)
|
startup fails. (Ravi Teja via vinodkv)
|
||||||
|
|
||||||
MAPREDUCE-3005. Fix both FifoScheduler and CapacityScheduler to correctly
|
MAPREDUCE-3005. Fix both FifoScheduler and CapacityScheduler to correctly
|
||||||
enforce locality constraints. (acmurthy)
|
enforce locality constraints. (acmurthy)
|
||||||
|
|
||||||
|
MAPREDUCE-3007. Fixed Yarn Mapreduce client to be able to connect to
|
||||||
|
JobHistoryServer in secure mode. (vinodkv)
|
||||||
|
|
||||||
Release 0.22.0 - Unreleased
|
Release 0.22.0 - Unreleased
|
||||||
|
|
||||||
|
@ -19,6 +19,7 @@
|
|||||||
package org.apache.hadoop.mapred;
|
package org.apache.hadoop.mapred;
|
||||||
|
|
||||||
import java.io.IOException;
|
import java.io.IOException;
|
||||||
|
import java.security.PrivilegedAction;
|
||||||
import java.util.HashMap;
|
import java.util.HashMap;
|
||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
|
|
||||||
@ -28,12 +29,13 @@
|
|||||||
import org.apache.hadoop.mapreduce.JobID;
|
import org.apache.hadoop.mapreduce.JobID;
|
||||||
import org.apache.hadoop.mapreduce.v2.api.MRClientProtocol;
|
import org.apache.hadoop.mapreduce.v2.api.MRClientProtocol;
|
||||||
import org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig;
|
import org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig;
|
||||||
|
import org.apache.hadoop.mapreduce.v2.security.client.ClientHSSecurityInfo;
|
||||||
import org.apache.hadoop.net.NetUtils;
|
import org.apache.hadoop.net.NetUtils;
|
||||||
import org.apache.hadoop.security.SecurityInfo;
|
import org.apache.hadoop.security.SecurityInfo;
|
||||||
|
import org.apache.hadoop.security.UserGroupInformation;
|
||||||
import org.apache.hadoop.yarn.YarnException;
|
import org.apache.hadoop.yarn.YarnException;
|
||||||
import org.apache.hadoop.yarn.conf.YarnConfiguration;
|
import org.apache.hadoop.yarn.conf.YarnConfiguration;
|
||||||
import org.apache.hadoop.yarn.ipc.YarnRPC;
|
import org.apache.hadoop.yarn.ipc.YarnRPC;
|
||||||
import org.apache.hadoop.yarn.security.client.ClientRMSecurityInfo;
|
|
||||||
|
|
||||||
public class ClientCache {
|
public class ClientCache {
|
||||||
|
|
||||||
@ -72,16 +74,21 @@ synchronized ClientServiceDelegate getClient(JobID jobId) {
|
|||||||
|
|
||||||
private MRClientProtocol instantiateHistoryProxy()
|
private MRClientProtocol instantiateHistoryProxy()
|
||||||
throws IOException {
|
throws IOException {
|
||||||
String serviceAddr = conf.get(JHAdminConfig.MR_HISTORY_ADDRESS,
|
final String serviceAddr = conf.get(JHAdminConfig.MR_HISTORY_ADDRESS,
|
||||||
JHAdminConfig.DEFAULT_MR_HISTORY_ADDRESS);
|
JHAdminConfig.DEFAULT_MR_HISTORY_ADDRESS);
|
||||||
LOG.info("Connecting to HistoryServer at: " + serviceAddr);
|
LOG.info("Connecting to HistoryServer at: " + serviceAddr);
|
||||||
Configuration myConf = new Configuration(conf);
|
final Configuration myConf = new Configuration(conf);
|
||||||
//TODO This should ideally be using it's own class (instead of ClientRMSecurityInfo)
|
|
||||||
myConf.setClass(YarnConfiguration.YARN_SECURITY_INFO,
|
myConf.setClass(YarnConfiguration.YARN_SECURITY_INFO,
|
||||||
ClientRMSecurityInfo.class, SecurityInfo.class);
|
ClientHSSecurityInfo.class, SecurityInfo.class);
|
||||||
YarnRPC rpc = YarnRPC.create(myConf);
|
final YarnRPC rpc = YarnRPC.create(myConf);
|
||||||
LOG.info("Connected to HistoryServer at: " + serviceAddr);
|
LOG.info("Connected to HistoryServer at: " + serviceAddr);
|
||||||
return (MRClientProtocol) rpc.getProxy(MRClientProtocol.class,
|
UserGroupInformation currentUser = UserGroupInformation.getCurrentUser();
|
||||||
NetUtils.createSocketAddr(serviceAddr), myConf);
|
return currentUser.doAs(new PrivilegedAction<MRClientProtocol>() {
|
||||||
|
@Override
|
||||||
|
public MRClientProtocol run() {
|
||||||
|
return (MRClientProtocol) rpc.getProxy(MRClientProtocol.class,
|
||||||
|
NetUtils.createSocketAddr(serviceAddr), myConf);
|
||||||
|
}
|
||||||
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@ -0,0 +1 @@
|
|||||||
|
org.apache.hadoop.mapreduce.v2.security.client.ClientHSSecurityInfo
|
Loading…
Reference in New Issue
Block a user