big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Projects Wiki Activity
25,267 Commits 2 Branches 2 Tags
Commit Graph

25267 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Bobby Wang
904cdd0b00
HADOOP-17812. NPE in S3AInputStream read() after failure to reconnect to store (#3222) 
This improves error handling after multiple failures reading data
-when the read fails and attempts to reconnect() also fail.

Contributed by Bobby Wang.

Change-Id: If17dee395ad6b9b7c738021bad20d0a13eb4011e
 2021-08-02 12:58:25 +01:00
Petre Bogdan Stolojan
f2cec5cb88
HADOOP-17139 Re-enable optimized copyFromLocal implementation in S3AFileSystem (#3101) 
This work
* Defines the behavior of FileSystem.copyFromLocal in filesystem.md
* Implements a high performance implementation of copyFromLocalOperation
  for S3
* Adds a contract test for the operation: AbstractContractCopyFromLocalTest
* Implements the contract tests for Local and S3A FileSystems

Contributed by: Bogdan Stolojan

Change-Id: I25d502102775c3626c4264e5a14c649879730050
 2021-08-02 11:58:36 +01:00
Brian Loss
37e0828e76
HADOOP-17811: ABFS ExponentialRetryPolicy doesn't pick up configuration values (#3221) 
Contributed by Brian Loss.

Change-Id: I5f24196d1d02de91336c3679abaf8d55cfaed746
 2021-08-02 11:37:33 +01:00
jenny
b8a8821735
MAPREDUCE-7258. HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's itemName from taskcounterGroup to taskCounterGroup (#1808) 
Co-authored-by: chenjuanni <chenjuanni@inspur.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e31169c8641384fa4b45d8a27c4b7a152c1ea35b)
 2021-08-02 15:39:38 +09:00
Szilard Nemeth
0a0628b1cb YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-31 14:53:30 +02:00
Szilard Nemeth
a272adc5fa YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi 2021-07-29 19:21:58 +02:00
He Xiaoqiao
797c595ba5
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
(cherry picked from commit 683feaa1d4863499207733a9f4c8c3cd2ec9c9d2)
 2021-07-29 19:04:08 +08:00
jianghuazhu
43f15578fc
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 6f730fd25c9845fea09b98c64b5d59bee5576db2)
 2021-07-29 15:44:02 +08:00
hchaverr
6cc1426b63 HADOOP-17819. Add extensions to ProtobufRpcEngine RequestHeaderProto. Contributed by Hector Sandoval Chaverri. (#3242) 
(cherry picked from commit 3c8a48e681babe4038e49205624e7be183b3466e)
 2021-07-28 15:48:51 -07:00
9uapaw
f001595b47 YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
(cherry picked from commit b4a524722a2466ede935aefd7b24a1e08fdbf076)
 2021-07-28 17:12:40 +02:00
S O'Donnell
d661afc06f HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell 
(cherry picked from commit 4eae284827c040cd9fd0eab2e7fd8aaae721326e)
 2021-07-28 15:02:16 +01:00
Szilard Nemeth
72801be13a YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori 2021-07-28 14:55:19 +02:00
bshashikant
18bd66e5b0 HDFS-16145. CopyListing fails with FNF exception with snapshot diff. (#3234) 
(cherry picked from commit dac10fcc202ed6d1fe4bd852f57a6bbcbadd90fe)
 2021-07-28 09:38:06 +01:00
Akira Ajisaka
9e72b697a2
HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit d710ec8d85692e4c2f915db1474b15dba84af666)
 2021-07-26 16:47:26 +09:00
dependabot[bot]
e533efc58e [UI2] Bump http-proxy to 1.18.1 (#2891) 
Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases)
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/http-party/node-http-proxy/compare/1.18.0...1.18.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b7431c3ed5e1fff947e84f3a4847b6d2864658d8)
 2021-07-26 02:17:36 +09:00
Akira Ajisaka
3f573b6751 HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227) 
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"

This reverts commit b6d839a60ceed733bfacb791fc5ed06116720dd0.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 2f2f822488c8da0227b95548c1fe40823b16a44d)
 2021-07-26 01:54:32 +09:00
Petre Bogdan Stolojan
e89d30b6b7
HADOOP-17458. S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException (#3040) 
Some network exceptions can raise SdkClientException with message
`Data read has a different length than the expected`.

These should be recoverable.

Contributed by Bogdan Stolojan

Change-Id: Ia22fd77d90971e9e02b4f947398a4749eebe5909
 2021-07-23 14:46:59 +01:00
Viraj Jasani
5ea1337167 HADOOP-17808. ipc.Client to set interrupt flag after catching InterruptedException (#3219) 
(cherry picked from commit 3a52bfc5db1b13a71d5a5c6ee0cd689fb8b27895)
 2021-07-23 12:42:52 +08:00
Viraj Jasani
92158cf84b HADOOP-17813. Checkstyle - Allow line length: 100 
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 98412ce2e1453308655d4f43e7f8c922960e6955)
 2021-07-22 15:02:58 -05:00
Renukaprasad C
d37192a613
HADOOP-17796. Upgrade jetty version to 9.4.43 (#3208) 
(cherry picked from commit dbd255f4a96474b31af2bd45c952ac7151a265b9)
 2021-07-22 16:37:39 +08:00
zhuqi-lucas
c31618e6b9 YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger. 2021-07-22 10:12:32 +08:00
Viraj Jasani
ec3311975c
HADOOP-16290. Enable RpcMetrics units to be configurable (#3198) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e1d00addb5b6d7240884536aaa57846af34a0dd5)
 2021-07-20 14:56:28 +08:00
Masatake Iwasaki
177569f1af
YARN-10858. [UI2] YARN-10826 breaks Queue view. (#3213) 
(cherry picked from commit 17bf2fcbc5142f7d860207c137fcf1d4d4edb699)
 2021-07-20 14:16:05 +09:00
Kihwal Lee
95a70c0fdf HDFS-16127. Improper pipeline close recovery causes a permanent write failure or data loss. Contributed by Kihwal Lee. 
(cherry picked from commit 47002719f2aa4ff58378d528d38b0f0962a45c25)
 2021-07-16 14:23:36 -05:00
Mehakmeet Singh
14a3e74c5c
HADOOP-17801. No error message reported when bucket doesn't exist in S3AFS (#3202) 
Contributed by: Mehakmeet Singh.

Change-Id: I26c2a85ef6bbfd1b8269a23fc44d9a55d7fa091c
 2021-07-16 15:36:54 +01:00
Mehakmeet Singh
cd15b0cb8a HADOOP-17803. Remove WARN logging from LoggingAuditor when executing a request outside an audit span (#3207) 
Followup to HADOOP-17511. "Add audit/telemetry logging to S3A connector"

Contributed by Mehakmeet Singh
 2021-07-16 11:52:37 +01:00
crossfire
3faae2e06d
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414) 
(cherry picked from commit 4bb25c810beb7abb8540c051114488e69a128ea3)
 2021-07-16 11:07:27 +08:00
Jim Brennan
0f2b89b791 HDFS-16042. DatanodeAdminMonitor scan should be delay based. Contributed by Ahmed Hussein. 2021-07-15 20:47:02 +00:00
Jim Brennan
b3481062e0 YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne. 
(cherry picked from commit 632f64cadb1dfd8f0940e350b9314b4d4f8eda4b)
 2021-07-15 14:26:03 +00:00
Abhishek Das
450dae7383 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 
(cherry picked from commit 1dd03cc4b573270dc960117c3b6c74bb78215caa)
 2021-07-13 18:23:27 -07:00
snehavarma
11825d30e8
HADOOP-17714 ABFS: testBlobBackCompatibility, testRandomRead & WasbAbfsCompatibility tests fail when triggered with default configs (#3035) (#3126) 
(cherry picked from commit 35e4c31fff9946d35a3543481585031558e9f5d5)
 2021-07-12 11:53:46 +05:30
snehavarma
ab3809cf8d
HADOOP-17715 ABFS: Append blob tests with non HNS accounts fail (#3028) (#3125) 
(cherry picked from commit 4c039fafebfe7b4d68b60c5ec6075d889ab1c40b)
 2021-07-12 11:51:41 +05:30
Akira Ajisaka
a1f90883f0
HADOOP-12665. Document hadoop.security.token.service.use_ip. (#3187) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit c81f82e21d137b6d45c20c481e7bbc6160d72f94)
 2021-07-12 10:18:19 +09:00
sumangala-patki
aa6a9cac72
HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const (#3106) 
* HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const (#2795)
. Contributed by Sumangala Patki.

(cherry picked from commit 76d92eb2a22c71b5fcde88a9b4d2faec81a1cb9f)
 2021-07-10 15:09:59 +05:30
litao
7cb91db575 HDFS-16122. Fix DistCpContext#toString() (#3191). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-10 13:56:36 +05:30
Artem Smotrakov
c18ad9a007 HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada573291266c4d6447862072fdf88af5)
 2021-07-10 13:43:31 +09:00
Akira Ajisaka
6cc9549499 HADOOP-17794. Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS (#3190) 
(cherry picked from commit 6ac5d8a1ed6ec3f6de7af3b8a79869830a4377ee)
 2021-07-09 22:50:18 +09:00
S O'Donnell
705884b87a HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma 
(cherry picked from commit b6c06c4b76ca5c206639e2ddea4f3bef393dbb11)
 2021-07-09 10:19:47 +01:00
bshashikant
ab939814a3 HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 
(cherry picked from commit 6450c1ea9ab884015290876798a734cbe5c0038b)
 2021-07-09 09:17:10 +01:00
Rafal Wojdyla
e3fb63f33f
HADOOP-17402. Add GCS config to the core-site (#2638) 
Contributed by Rafal Wojdyla
 2021-07-07 22:43:31 +01:00
Eric Payne
e395711164 MAPREDUCE-7353: Mapreduce job fails when NM is stopped. Contributed by Bilwa S T (BilwaST) 
(cherry picked from commit 7581413156da396db218e36a966c5749589b31a7)
 2021-07-07 20:57:32 +00:00
Masatake Iwasaki
99cff01e38
HADOOP-17775. Remove JavaScript package from Docker environment. (#3182) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-07 17:49:19 +09:00
liangxs
24b780820c HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout (#3080) 
(cherry picked from commit a5db6831bc674a24a3251cf1b20f22a4fd4fac9f)
 2021-07-07 09:41:11 +08:00
Mukund Thakur
e8f9af6f2a
HADOOP-17250 Lot of short reads can be merged with readahead. (#3110) 
Introducing fs.azure.readahead.range parameter which can be set by the user.
Data will be populated in buffer for random reads as well which leads to fewer
remote calls.

This patch also changes the seek implementation to perform a lazy seek. The
actual seek is done when a read is initiated and data is not present in the buffer else
data is returned from the buffer thus reducing the number of remote storage calls.

Contributed By: Mukund Thakur

Change-Id: Ib920eedd0087caa150afa4d4c23e89df56b29e83
 2021-07-05 11:23:32 +01:00
Viraj Jasani
2d56335a3d
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169) 
(cherry picked from commit da575a53cf521dc9c9660b7dd836fce04d05bab5)
 2021-07-05 13:42:51 +09:00
litao
aec8c9188a
HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b87bac13e43c7a0b4f16002f5aff9d8c596413a9)
 2021-07-05 08:15:10 +09:00
Mehakmeet Singh
f1a14df9e6
HADOOP-17774. S3A bytesRead FS statistic showing twice the correct value (#3144) 
Contributed by: Mehakmeet Singh

Change-Id: I3302654ca36474a5f399aa848f88bce4587022d8
 2021-07-02 14:13:26 +01:00
Viraj Jasani
bf6ed21104 HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
    Signed-off-by: stack Chuang <stack@apache.org>
    Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-30 12:59:25 -07:00
Wanqiang Ji
628b08db63
HADOOP-17777. Update clover-maven-plugin version from 3.3.0 to 4.4.1 (#3152) 
(cherry picked from commit c005195b783514e26f734e694beedd38ef8f7954)
 2021-06-30 09:35:36 +08:00
litao
4ee7bd345f
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146) 
(cherry picked from commit 95454d821c6e4c08d49df195d7c1e0ea1aacafb9)
 2021-06-29 18:15:56 +08:00