big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Projects Wiki Activity
24,321 Commits 2 Branches 2 Tags
Commit Graph

24321 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
He Xiaoqiao
ab451b8e42 HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa12a012709f43aa35cc606db6fb137f9)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
 2021-03-30 22:29:49 +08:00
Wanqiang Ji
872839416b HDFS-15331. Remove invalid exclusions that minicluster dependency on HDFS (#1996). Contributed by Wanqiang Ji 
(cherry picked from commit e877118454b97ae72d3b52dd256999ca073df91d)
 2021-03-30 22:29:49 +08:00
Akira Ajisaka
53b024a2b7 HDFS-15508. [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module 
(cherry picked from commit 9062814bab7d6087c584e03b0b16eaa60b31f60a)
 2021-03-30 22:29:49 +08:00
Akira Ajisaka
a407f7de5a HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li. 
(cherry picked from commit 32895f4f7ea51522e779e68a030bfc9515f40798)
 2021-03-30 22:29:49 +08:00
Akira Ajisaka
c4f7b8578b HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li. 
(cherry picked from commit 7938ebfb9dd12b874ced1798f0c3ad0c160cb465)
 2021-03-30 22:29:49 +08:00
Karthik Palanisamy
769df1e964 HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366) 
(cherry picked from commit df4006eb8139a08b92b5a94adfde544de978137c)
 2021-03-30 22:29:49 +08:00
Ayush Saxena
7f20fad419 HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena. 
(cherry picked from commit 2e8cafac3b071fe5b943542827fd8a496b137fa9)
 2021-03-30 22:29:48 +08:00
Stephen O'Donnell
56ef16468a
HADOOP-17222. Create socket address leveraging URI cache (#2817) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-30 11:59:44 +01:00
Akira Ajisaka
c96fbb9199
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9f674e734e7f4c917ee2d51c77e527d)
 2021-03-30 17:47:54 +09:00
Brahma Reddy Battula
5358313f97 YARN-10466.Fix NullPointerException in yarn-services Component.java. Contributed by D M Murali Krishna Reddy 
(cherry picked from commit 413a4c3c05d317090c706385c51e4cabcfd92b0e)
 2021-03-30 13:52:35 +05:30
Brahma Reddy Battula
bac1326e4e YARN-10439. addendum fix for shaded guva. 
(cherry picked from commit 6577bf1891b11c9271d73491b311059677dfb376)
 2021-03-30 13:49:23 +05:30
Brahma Reddy Battula
616a41ee32 YARN-10441. Add support for hadoop.http.rmwebapp.scheduler.page.class. Contributed by D M Murali Krishna Reddy 
(cherry picked from commit b61f52ec565b84306ec8d9e0b53f5d0390e1b597)
 2021-03-30 09:52:28 +05:30
Brahma Reddy Battula
5181b2004b YARN-10439. Yarn Service AM listens on all IP's on the machine. Contributed by D M Murali Krishna Reddy 
(cherry picked from commit d0dcfc405c624f73ed1af9527bbf456a10337a6d)
 2021-03-30 09:46:52 +05:30
Brahma Reddy Battula
e9d8f16a70 YARN-10437. Destroy yarn service if any YarnException occurs during submitApp.Contributed by D M Murali Krishna Reddy 
(cherry picked from commit 2d62dced4b60938cab630321830a0510d5391338)
 2021-03-30 09:40:06 +05:30
Viraj Jasani
d6aaf516d9 HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-29 23:25:14 +05:30
Harunobu Daikoku
db8af483d1 HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
(cherry picked from commit ea6595d3b68ac462aec0d493718d7a10fbda0b6d)
 2021-03-29 11:47:16 +09:00
Ayush Saxena
9c9b16c957
HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2808). Contributed by Ayush Saxena. 
* HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).

* HADOOP-17531.Addendum: DistCp: Reduce memory usage on copying huge directories. (#2820)

Signed-off-by: Steve Loughran <stevel@apache.org>
 2021-03-27 09:25:25 +05:30
Eric Badger
65bba8c3ed YARN-10713. ClusterMetrics should support custom resource capacity related metrics. Contributed by Qi Zhu. 
(cherry picked from commit 19e418c10d8fbd994e5ca326bf894cab706b8a4f)
 2021-03-25 22:35:19 +00:00
Xiaoyu Yao
67d52af225 HADOOP-16828. Zookeeper Delegation Token Manager fetch sequence number by batch. Contributed by Fengnan Li. 
(cherry picked from commit 6288e15118fab65a9a1452898e639313c6996769)
 2021-03-25 14:44:02 +00:00
Steve Loughran
a07e3c41ca
HADOOP-13551. AWS metrics wire-up (#2778) 
Moves to the builder API for AWS S3 client creation, and
offers a similar style of API to the S3A FileSystem and tests, hiding
the details of which options are client, which are in AWS Conf,
and doing the wiring up of S3A statistics interfaces to the AWS
SDK internals. S3A Statistics, including IOStatistics, should now
count throttling events handled in the AWS SDK itself.

This patch restores endpoint determination by probes to US-East-1
if the client isn't configured with fs.s3a.endpoint.

Explicitly setting the endpoint will save the cost of these probe
HTTP requests.

Contributed by Steve Loughran.

Change-Id: Ifa6caa8ff56369ad30e4fd01a42bc74f7b8b3d6b
 2021-03-25 13:59:33 +00:00
Mukund Thakur
4c3324ca1a
HADOOP-17305. Fix ITestCustomSigner to work with s3 compatible endpoints (#2395) 
Contributed by Mukund Thakur

Change-Id: Ia5def405056691c349cf05530fd3172047d2058b
 2021-03-25 13:59:33 +00:00
leosunli
b3e968d21e HDFS-13639. SlotReleaser is not fast enough (#1885) 
(cherry picked from commit be374faf429d28561dd9c582f5c55451213d89a4)
 2021-03-25 13:25:17 +00:00
Akira Ajisaka
80d4636067
HADOOP-17602. Upgrade JUnit to 4.13.1. Contributed by Ahmed Hussein. 
(cherry picked from commit c5929c00a35a8e58bcf1c80d9d414c3cbdce4483)

 Conflicts:
	hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml
 2021-03-25 19:54:51 +09:00
He Xiaoqiao
bf018e7237
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:25:22 +08:00
LeonGao91
4db5643ff9 HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299) 
(cherry picked from commit de055ec8aad1b5a131d366aa5395f2904f0f6ecd)
 2021-03-25 05:09:11 +08:00
Steve Loughran
9bbf1f87c9
HADOOP-17476. ITestAssumeRole.testAssumeRoleBadInnerAuth failure. (#2777) 
Contributed by Steve Loughran.

Change-Id: Ie96ca99f5d91e5a6aaea4cae4c2e850de9fddb01
 2021-03-24 16:49:41 +00:00
Takanobu Asanuma
9a0c5d0ca2 HDFS-15275. HttpFS: Response of Create was not correct with noredirect and data are true. Contributed by hemanthboyina. 
(cherry picked from commit 1fdfaebd98d9b8f40f616d03fa2f3b6b5c71b11b)
 2021-03-24 17:44:29 +08:00
Inigo Goiri
813aafcee2 HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun. 
(cherry picked from commit 6e04b00df1bf4f0a45571c9fc4361e4e8a05f7ee)
 2021-03-24 17:44:29 +08:00
Ayush Saxena
e4217be749 HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut. 
(cherry picked from commit 5ab1db9e218e11d00035e234344ef90228d8898d)
 2021-03-24 17:44:29 +08:00
Ayush Saxena
580634882e HDFS-14546. Document block placement policies. Contributed by Amithsha. 
(cherry picked from commit 17ffcab5f621400bd8bb47dc8fb29365f6e24ebd)
 2021-03-24 17:44:29 +08:00
Karthik Palanisamy
c96821ece8 HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 (#2365) 
(cherry picked from commit 735e85a1d2036585075b81d1fb2fd302e89eb363)
 2021-03-24 17:44:29 +08:00
Takanobu Asanuma
9e1e89b1b9 HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel. 
(cherry picked from commit b4d97a8dc75e4dc823d092f5bac37d6096dc6dc4)
 2021-03-24 17:53:19 +09:00
Masatake Iwasaki
2d83369f6f HDFS-15743. Fix -Pdist build failure of hadoop-hdfs-native-client. (#2569) 
(cherry picked from commit 9aefe72ed68fdf3ae6015a6fed15d8bb20b0c6fb)
 2021-03-24 14:04:47 +08:00
sunlisheng
a8c0083b1b HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun. 
(cherry picked from commit 7025f39944e628345109b43cba2cd4d49ca8cc6b)
 2021-03-24 14:04:47 +08:00
Ayush Saxena
94766fdb13 HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun. 
(cherry picked from commit ff84a57483320abf81bf941b81d13d5bee14084b)
 2021-03-24 14:04:47 +08:00
imbajin
4f79df4da1 HDFS-15551. Tiny Improve for DeadNode detector (#2265) 
Contributed by imbajin.

Reviewed-by: leosunli <lisheng.sun08@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 89428f142fe7cee17bd1a0f5f207b6952ec79d32)
 2021-03-24 14:04:47 +08:00
sunlisheng
3ecd02ac78 HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun. 
(cherry picked from commit f8769e0f4b917d9fda8ff7a9fddb4d755d246a1e)
 2021-03-24 14:04:47 +08:00
Narges Shadab
f07bde90c9 HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652) 
(cherry picked from commit 115623a6ee4eb7391c7c26e1728fa153c0698803)
 2021-03-24 14:04:47 +08:00
Narges Shadab
3a355282c9 HDFS-15868. Possible Resource Leak in EditLogFileOutputStream (#2736). Contributed by Narges Shadab. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit f5f3fc6fe90e3bb2ee783f956432db11e655b9e3)
 2021-03-24 14:04:47 +08:00
Narges Shadab
4abe6e6ac1 HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal (#2790). Contributed by Narges Shadab. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit d77c7ab4e0604484ec3c48ac2acb32ea71a53d9a)
 2021-03-24 14:04:47 +08:00
Gautham B A
a216ee3b85 HDFS-15910. Improve security with explicit_bzero (#2793) 
(cherry picked from commit 714427c41401eacf3cab7c087d9e5d61a658716b)
 2021-03-24 14:04:47 +08:00
Toshihiro Suzuki
e9b06d942c HDFS-15249 ThrottledAsyncChecker is not thread-safe. (#1922) 
(cherry picked from commit c12ddbd1de40b32bbe2f6a3e484abf843d6d92ae)
 2021-03-24 14:04:47 +08:00
Viraj Jasani
6d3f5e844b
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2796) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 22:51:21 -07:00
Ayush Saxena
76c40a52d7 HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell. 2021-03-24 02:13:52 +05:30
Jim Brennan
78bddd0d9f YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 
(cherry picked from commit 174f3a96b10a0ab0fd8aed1b0f904ca5f0c3f268)
 2021-03-23 18:23:50 +00:00
Ayush Saxena
b26d75ce37 HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena. 
(cherry picked from commit e0ae232f669b2e2a6654cfacff22a090c462effc)
 2021-03-23 15:34:48 +00:00
Ayush Saxena
968c95bfba HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit ae13a5ccbea10fe86481adbbff574c528e03c7f6)
 2021-03-23 15:34:36 +00:00
Jim Brennan
91d229bf35 MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein 
(cherry picked from commit ede490d13182f0fd3831afe6db97e01917add5b4)
 2021-03-22 18:44:41 +00:00
Ayush Saxena
d8314fffc0 HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team. 
(cherry picked from commit c2a17659d1d48407edf437131783b653dbc87bc1)
 2021-03-22 12:38:09 +00:00
Ayush Saxena
793ab9acf3 HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 
(cherry picked from commit e3d1966f58ad473b8e852aa2b11c8ed2b434d9e4)
 2021-03-22 12:35:11 +00:00