big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,011 Commits 2 Branches 2 Tags 264 MiB
d72cdf7205
Commit Graph

27011 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
xuzq
b1d4af2492
HDFS-16704. Datanode return empty response instead of NPE for GetVolumeInfo during restarting (#4661). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-08-15 20:57:27 +08:00
Hui Fei
86cc96c493
Revert "HDFS-16689. NameNode may crash when transitioning to Active with in-progress tailer if there are some abnormal JNs. (#4628)" (#4743) 2022-08-15 19:48:05 +08:00
Steve Loughran
906ae5138e
HADOOP-18402. S3A committer NPE in spark job abort (#4735) 
JobID.toString() and TaskID.toString() to only be called
when the IDs are not null.

This doesn't surface in MapReduce, but Spark SQL can trigger
in job abort, where it may invok abortJob() with an
incomplete TaskContext.

This patch MUST be applied to branches containing
HADOOP-17833. "Improve Magic Committer Performance."

Contributed by Steve Loughran.
 2022-08-15 11:20:59 +01:00
Steve Loughran
eee59a8372
Revert "HADOOP-18402. S3A committer NPE in spark job abort (#4735)" 
(managed to commit through the github ui before I'd got the message done)

This reverts commit ad83e95046.
 2022-08-15 11:20:36 +01:00
Steve Loughran
ad83e95046
HADOOP-18402. S3A committer NPE in spark job abort (#4735) 
jobId.toString() to only be called when the ID isn't null.

this doesn't surface in MR, but spark seems to manage it

Change-Id: I06692ef30a4af510c660d7222292932a8d4b5147
 2022-08-15 11:18:47 +01:00
slfan1989
ab88e4b65d
YARN-11223. [Federation] Add getAppPriority, updateApplicationPriority REST APIs for Router. (#4733) 2022-08-14 19:22:16 -07:00
Paul King
d0fdb1d6e0
HADOOP-18404. Fix broken link to wiki help page in org.apache.hadoop.util.Shell (#4718). Contributed by Paul King. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-14 19:28:22 +05:30
slfan1989
d383cc4525
YARN-11236. Implement FederationReservationHomeSubClusterStore With MemoryStore. (#4711) 2022-08-13 10:37:20 -07:00
zhengchenyu
9f6bbc90a8
YARN-11148. In federation and security mode, nm recover may fail. (#4308) 2022-08-13 10:33:16 -07:00
kevins-29
b737869e01
HADOOP-18383. Codecs with @DoNotPool annotation are not closed causing memory leak (#4585) 2022-08-12 16:05:13 -07:00
xuzq
e0c8c6eed4
HDFS-16678. RBF should supports disable getNodeUsage() in RBFMetrics (#4606) 2022-08-12 12:01:58 -07:00
xuzq
521e65acfe
HDFS-16723. Replace incorrect SafeModeException with StandbyException in RouterRpcServer.class (#4716). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-13 00:21:11 +05:30
Viraj Jasani
8c9533a0f8
HADOOP-18397. Shutdown AWSSecurityTokenService when its resources are no longer in use (#4722) 
Contributed by Viraj Jasani.
 2022-08-12 11:59:15 +01:00
xuzq
59619ad247
HDFS-16689. NameNode may crash when transitioning to Active with in-progress tailer if there are some abnormal JNs. (#4628) 2022-08-12 12:19:28 +08:00
Steve Vaughan
2005582a28
HDFS-16702. MiniDFSCluster should report cause of exception in assert error (#4680) 
When the MiniDFSClsuter detects that an exception caused an exit, it should include that exception as the cause for the AssertionError that it throws. The current AssertError simply reports the message "Test resulted in an unexpected exit" and provides a stack trace to the location of the check for an exit exception. This patch adds the original exception as the cause of the AssertError.
 2022-08-11 13:52:39 -07:00
slfan1989
6ca2d3f848
YARN-6539. Create SecureLogin inside Router. (#4712) 2022-08-11 13:25:51 -07:00
xuzq
09cabaad68
HDFS-13274. RBF: Extend RouterRpcClient to use multiple sockets (#4531) 2022-08-11 13:23:32 -07:00
Mukund Thakur
b28e4c6904
HADOOP-18392. Propagate vectored s3a input stream stats to file system stats. (#4704) 
part of HADOOP-18103.

Contributed By: Mukund Thakur
 2022-08-12 01:42:00 +05:30
huaxiangsun
e9509ac467
HADOOP-18340. deleteOnExit does not work with S3AFileSystem (#4608) 
Contributed by Huaxiang Sun
 2022-08-11 20:25:13 +01:00
Yubi Lee
c0bbdca97e
HADOOP-18398. Prevent AvroRecord*.class from being included non-test jar (#4727) 
Contributed by Yubi Lee.
 2022-08-11 20:12:41 +01:00
slfan1989
133e8aabf0
YARN-11227. [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST APIs for Router. (#4715) 2022-08-10 14:53:46 -07:00
slfan1989
ffa9ed93a4
YARN-6572. Refactoring Router services to use common util classes for pipeline creations. (#4594) 2022-08-09 14:44:29 -07:00
Ashutosh Gupta
92abd99450
YARN-11237. Fix Bug while disabling proxy failover with Federation (#4658) 2022-08-08 13:29:27 -07:00
slfan1989
977f4b6165
MAPREDUCE-7385. impove JobEndNotifier#httpNotification With recommended methods. (#4403). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-09 00:59:03 +05:30
xuzq
895f7c51fd
HDFS-16709. Remove redundant cast in FSEditLogOp.class (#4667). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-09 00:54:01 +05:30
Viraj Jasani
06f0f7db79
HADOOP-18373. IOStatisticsContext tuning (#4705) 
The name of the option to enable/disable thread level statistics is
"fs.iostatistics.thread.level.enabled";

There is also an enabled() probe in IOStatisticsContext which can
be used to see if the thread level statistics is active.

Contributed by Viraj Jasani
 2022-08-08 10:42:57 +01:00
slfan1989
d8d3325d2f
HADOOP-18387. Fix incorrect placeholder in hadoop-common (#4679). Contributed by fanshilun. 2022-08-08 02:35:39 +05:30
Ashutosh Gupta
1cda2dcb6e
YARN-10793. Upgrade Junit from 4 to 5 in hadoop-yarn-server-applicationhistoryservice (#4603) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-07 16:15:47 +09:00
slfan1989
52c2d99889
YARN-11228. [Federation] Add getAppAttempts, getAppAttempt REST APIs for Router. (#4695) 2022-08-06 09:36:26 -07:00
xuzq
25ccdc77af
HDFS-16648. Add isDebugEnabled check for debug blockLogs in some classes (#4529) 2022-08-06 21:34:01 +08:00
Ashutosh Gupta
bd0f9a46e1
HADOOP-18390. Fix out of sync import for HADOOP-18321 (#4694) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-06 21:51:23 +09:00
ahmarsuhail
b5642c5638
HADOOP-18366. ITestS3Select.testSelectSeekFullLandsat is timing out. (#4702) 
Reduces size of data read to 1 MB

Contributed by Ahmar Suhail
 2022-08-05 14:13:04 +01:00
Steve Loughran
62dbefd8f2
HADOOP-18305. Release Hadoop 3.3.4: upstream changelog and jdiff files 
Add the r3.3.4 changelog, release notes and jdiff xml files.
 2022-08-05 14:06:22 +01:00
Ayush Saxena
080e67039d
HADOOP-17234. Addendum. Add .asf.yaml to allow github and jira integration. (#4686). Contributed by Ayush Saxena. 
Reviewed-by: Mingliang Liu <liuml07@apache.org>
 2022-08-05 08:34:56 +05:30
slfan1989
6f7c4c74ea
YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster (#4656) 2022-08-04 10:16:08 -07:00
Ashutosh Gupta
0aa08ef543
HADOOP-18363. Fix bug preventing hadoop-metrics2 from emitting metrics to > 1 Ganglia servers (#4627) 
* HADOOP-18363. Fix bug preventing hadoop-metrics2 from emitting metrics to > 1 Ganglia servers
 2022-08-04 18:26:38 +05:30
zhangshuyan0
dbf73e16b1
HADOOP-18364. All method metrics related to the RPC protocol should be initialized. (#4624). Contributed by Shuyan Zhang. 
Reviewed-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-08-04 13:00:37 +08:00
xuzq
8eebf40b1a
HDFS-16642. Moving the selecting inputstream from journalnode in EditLogTailer outof FSNLock (#4497) 2022-08-04 11:04:28 +08:00
Mukund Thakur
66dec9d322
HADOOP-18355. Update previous index properly while validating overlapping ranges. (#4647) 
part of HADOOP-18103.

Contributed By: Mukund Thakur
 2022-08-04 04:08:04 +05:30
slfan1989
c5ec727435
YARN-11230. [Federation] Add getContainer, signalToContainer REST APIs for Router. (#4689) 2022-08-03 11:21:48 -07:00
slfan1989
6463f86f83
YARN-11029. Refactor AMRMProxy Service code and Added Some Metrics. (#4650) 2022-08-03 09:38:00 -07:00
slfan1989
c5eba323bc
YARN-6972. Adding RM ClusterId in AppInfo. (#4673) 2022-08-03 09:35:40 -07:00
Steve Vaughan
0fc7dd8228
HDFS-16687. RouterFsckServlet replicates code from DfsServlet base class (#4681) 2022-08-03 09:14:11 -07:00
xuzq
0f36539d60
HDFS-16712. Fix incorrect placeholder in DataNode.java (#4672). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-03 13:01:41 +05:30
Ashutosh Gupta
69f6fdb757
HADOOP-18301. Upgrade commons-io to 2.11.0 (#4455) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-03 10:44:39 +09:00
slfan1989
1f0a71a92b
YARN-8973. [Router] Add missing methods in RMWebProtocol. (#4664) 2022-08-02 14:07:09 -07:00
slfan1989
57da4bb0a1
YARN-11220. [Federation] Add getLabelsToNodes, getClusterNodeLabels, getLabelsOnNode REST APIs for Router (#4657) 2022-08-02 12:09:55 -07:00
ahmarsuhail
123d1aa884
HADOOP-18368. Fixes ITestCustomSigner for access point names with '-' (#4634) 
Contributed By: Ahmar Suhail <ahmarsu@amazon.co.uk>
 2022-08-02 01:49:42 +05:30
slfan1989
13fbfd5dea
HADOOP-18358. Update commons-math3 from 3.1.1 to 3.6.1. (#4619). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-02 01:48:47 +05:30
slfan1989
2680f17eb4
YARN-11180. Refactor some code of getNewApplication, submitApplication etc. (#4618) 2022-07-29 08:23:11 -07:00