big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,382 Commits 2 Branches 2 Tags
Commit Graph

24382 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
zhangshuyan0
9f2db2c9fd
HDFS-15963. Unreleased volume references cause an infinite loop. (#2941) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-21 18:32:56 +08:00
litao
29414871bd
HDFS-15879. Exclude slow nodes when choose targets for blocks (#2928) 2021-04-21 10:41:13 +09:00
billierinaldi
8170a7bb60 HADOOP-16948. Support infinite lease dirs (#1925). Contributed by Billie Rinaldi. 
(cherry picked from commit c1fde4fe94f268c6d5515b421ac47345dca8163d)
 2021-04-20 14:36:54 -04:00
Eric Badger
1960924d07 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 
(cherry picked from commit 6cb90005a7d0651474883ac4e1b6961ef74fe513)
 2021-04-20 17:46:05 +00:00
hemanthboyina
90f63cb12c HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina. 
(cherry picked from commit 16a20503cacf12c3d8e27ba90820384f58bed06f)
 2021-04-20 16:21:57 +01:00
Steve Loughran
f30a0debae HADOOP-17641. ITestWasbUriAndConfiguration failing. (#2937) 
This moves the mock account name --which is required to never exist-- from
"mockAccount"  to an account name containing a static UUID.

Contributed by Steve Loughran.
 2021-04-20 15:37:18 +01:00
Ayush Saxena
c93a2f6789 HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena. 
(cherry picked from commit c84e6beada4e604175f7f138c9878a29665a8c47)
 2021-04-20 12:13:39 +01:00
litao
dc830cf277 HDFS-15970. Print network topology on the web (#2896) 
(cherry picked from commit 85a3532849d81ca929bc52cb7ca905c4f53652ec)
 2021-04-19 21:22:47 +09:00
Eric Badger
f7e382c62e YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 
(cherry picked from commit 31477a224f2eddc78e19a9426f5ec6182193cc00)
 2021-04-16 20:21:01 +00:00
Viraj Jasani
49f6326a9f HADOOP-17633. Bump json-smart to 2.4.2 and nimbus-jose-jwt to 9.8 due to CVEs (#2895). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-04-16 12:41:06 +05:30
sumangala-patki
8daa26d2e5
HADOOP-17576. ABFS: Disable throttling update for auth failures (#2761) (#2885) 
Contributed by Sumangala Patki

(cherry picked from commit 6f640abbaf14efa98d6c599e5fff95647730ad42)
 2021-04-16 10:47:11 +05:30
Hui Fei
4c0b833bab HDFS-15898. Test case TestOfflineImageViewer fails (#2779) 
(cherry picked from commit 0d4210411946d7f214d90e047b9f7bb9b7b82dfb)
 2021-04-16 14:07:15 +09:00
Masatake Iwasaki
a3405cb56a HADOOP-17569. Building native code fails on Fedora 33. (#2886) 
(cherry picked from commit 2717203f858ff654de0fc01cfb9afef2e705e33c)
 2021-04-15 12:39:07 +00:00
He Xiaoqiao
a3cff8776a
HADOOP-17613. Log not flushed fully when daemon shutdown. Contributed by Renukaprasad C. 2021-04-14 12:28:34 +08:00
Akira Ajisaka
72355c7b6e
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be3ae1f42bde9c22ab5ba96cf60df3c6)
 2021-04-13 17:10:00 +09:00
Wei-Chiu Chuang
bfba6f1f3c
HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun. (#2882) 
(cherry picked from commit e391844e8e414abf8c94f7bd4719053efa3b538a)

Co-authored-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-04-12 23:55:32 -07:00
Akira Ajisaka
77315abe47
HADOOP-17608. Fix TestKMS failure (#2880) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2bd810a5075e6b9c15fde22e43ac9bcf2a6c22f9)
 2021-04-12 12:54:59 +09:00
dependabot[bot]
d0b02a23f3 Bump y18n 
Bumps [y18n](https://github.com/yargs/y18n) from 3.2.1 to 3.2.2.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
(cherry picked from commit 1a90fc4b50068865359809cd437e45595c397195)
 2021-04-11 01:31:27 +09:00
Eric Badger
1658a5140a YARN-10503. Support queue capacity in terms of absolute resources with custom 
resourceType. Contributed by Qi Zhu.
 2021-04-09 17:51:01 +00:00
Stephen O'Donnell
0e6cd352b5
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. (#2813) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2021-04-09 10:25:14 +01:00
lichaojacobs
068f114066
MAPREDUCE-7329: HadoopPipes task may fail when linux kernel version change from 3.x to 4.x (#2775) 
(cherry picked from commit 663ca14a769bd8fa124c1aff4ac6630491dbb425)
 2021-04-09 12:00:38 +09:00
Viraj Jasani
4994b73eeb HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874) 
(cherry picked from commit 56bd968fb434c7d2ae7ac9e7659d0ca682340b2a)
 2021-04-09 11:13:39 +09:00
He Xiaoqiao
df99ac0399 HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 
(cherry picked from commit 459df4199791a9093324e089b4a5cc880be8d6ff)
 2021-04-08 13:15:37 +01:00
Stephen O'Donnell
ef95f7a963 HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 
(cherry picked from commit 4c567fcff7af45c75117ee4a75c087aa454a89e5)
 2021-04-08 12:15:31 +01:00
touchida
dca2bf9dd5 HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 
(cherry picked from commit 95e68926750b55196cf9da53c25359c98ef58a4f)
 2021-04-08 17:20:08 +08:00
Wei-Chiu Chuang
7f7535534d HADOOP-17625. Update to Jetty 9.4.39. (#2870) 
Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99aae5e29c17b03fddb0a805da8fcae8)
 2021-04-08 14:28:03 +08:00
Brahma Reddy Battula
236a9a7713 HADOOP-17617. Incorrect representation of RESPONSE for Get Key Version in KMS index.md.vm file. Contributed by Ravuri Sushma sree 
(cherry picked from commit ae88174c29ae02b6cf48785ecb3432a2698944bb)
 2021-04-07 23:50:02 +05:30
Gautham B A
b59bde61f6
HDFS-15948. Fix test4tests for libhdfspp (#2873) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 02b9506c5a7be214c185fe1fb8aad214b3210bb8)
 2021-04-07 19:58:03 +09:00
Wei-Chiu Chuang
1110b03752 HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282013d7776667415a429e7ca35efba66)
 2021-04-07 13:17:40 +08:00
Wei-Chiu Chuang
44365244e6 HADOOP-17624. Remove any rocksdb exclusion code. (#2867) 
Reviewed-by: Márton Elek <elek@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit f608cda72976d62354e063472d9b985eed48ebae)
 2021-04-07 12:53:34 +08:00
Eric Badger
fb5809984e YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-06 23:34:35 +00:00
Gautham B A
d7b31acbba
HDFS-15949. Fix integer overflow (#2857) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 36014b8282579ac3d89b62a05586206fb6e0adcf)
 2021-04-06 18:23:38 +09:00
Viraj Jasani
8b4b3d6fe6 HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 3f2682b92b540be3ce15642ab8be463df87a4e4e)
 2021-04-06 14:18:31 +09:00
Viraj Jasani
2b5fd341b9 HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844). Contributed by Viraj Jasani 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 00:20:56 +05:30
litao
7a7e194228 HDFS-15951. Remove unused parameters in NameNodeProxiesClient (#2859) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9e2d5d6d05a2d8b657d06fdf3db49d588394a579)
 2021-04-05 17:49:27 +09:00
Gautham B A
e4d04a640b
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca5e13fd74838fca3598587eb4770b797f436832)
 2021-04-05 11:42:52 +09:00
Wei-Chiu Chuang
cc0cc7ab5f HADOOP-17614. Bump netty to the latest 4.1.61. (#2850) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 29a6769b68f32fc3eb0686a955040b9412a24a66)
 2021-04-05 08:37:10 +08:00
Surendra Singh Lilhore
e079aaa820 MAPREDUCE-7199. HsJobsBlock reuse JobACLsManager for checkAccess. Contributed by Bilwa S T 
(cherry picked from commit a1b0697d379d33223ec1a46dfef31d6d226169bb)
 2021-04-02 21:31:45 +05:30
Inigo Goiri
d973f37054 HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina. 
(cherry picked from commit 2148a8fe645333444c4e8110bb56acf0fb8e41b4)
 2021-04-02 17:14:34 +08:00
Inigo Goiri
887948d127 HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina. 
(cherry picked from commit 719b53a79dc169a8c52229831dcb011935a8a151)
 2021-04-02 17:14:24 +08:00
Surendra Singh Lilhore
02d37c7d22 HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina 
(cherry picked from commit 743c2e9071f4a73e0196ad4ca005b767758642b9)
 2021-04-02 17:14:14 +08:00
Takanobu Asanuma
14fddba715 HDFS-15252. HttpFS: setWorkingDirectory should not accept invalid paths. Contributed by hemanthboyina. 
(cherry picked from commit 736659e0e1ab2882313e3a41d9a20d4b0f5b0816)
 2021-04-02 17:13:59 +08:00
Ayush Saxena
2da3356428 HDFS-15591. RBF: Fix webHdfs file display error. Contributed by wangzhaohui. 
(cherry picked from commit fc8a6dd8a92f8e18436d1a73d6e8ef808c583aee)
 2021-04-02 17:13:00 +08:00
Takanobu Asanuma
f5cc1540b4 HDFS-15510. RBF: Quota and Content Summary was not correct in Multiple Destinations. Contributed by Hemanth Boyina. 
(cherry picked from commit ca8e7a77256003e11ab7e3d079ee4cf9f50080dd)
 2021-04-02 17:11:33 +08:00
Ayush Saxena
a549b4a82e HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq. 
(cherry picked from commit 936bf09c3745cfec26fa9cfa0562f88b1f8be133)
 2021-04-02 17:11:16 +08:00
Ayush Saxena
5bfc3a4c3a HDFS-15198. RBF: Add test for MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by zhengchenyu. 
(cherry picked from commit 8a9a674ef10a951c073ef17ba6db1ff07cff52cd)
 2021-04-02 17:10:56 +08:00
Inigo Goiri
327488d688 HDFS-15196. RBF: RouterRpcServer getListing cannot list large dirs correctly. Contributed by Fengnan Li. 
(cherry picked from commit 80b877a72f52ef0f4acafe15db55b8ed61fbe6d2)
 2021-04-02 14:12:17 +08:00
Inigo Goiri
57d2fff84b HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina. 
(cherry picked from commit 8b7695bb2628574b4450bac19c12b29db9ee0628)
 2021-04-02 14:06:52 +08:00
Inigo Goiri
1148d3895f HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina. 
(cherry picked from commit 130f89e068ca346a44fa6619ae0726c1e5cc5d06)
 2021-04-02 14:06:52 +08:00
Inigo Goiri
41fad5056c HDFS-15265. HttpFS: validate content-type in HttpFSUtils. Contributed by hemanthboyina. 
(cherry picked from commit 31b2f687eff5a77ea83c2279cc94c4c7158160dc)
 2021-04-02 14:06:52 +08:00