big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,275 Commits 2 Branches 2 Tags 264 MiB
4df4389325
Commit Graph

25275 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
jianghuazhu
53a2c65694
HDFS-16162.Improve DFSUtil#checkProtectedDescendants() related parameter comments. (#3295) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-17 16:47:10 +08:00
Liang-Chi Hsieh
6014a089fd
HADOOP-17825. Add BuiltInGzipCompressor (#3250) 
Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is not loaded. So, without Hadoop native codec installed, saving SequenceFile using GzipCodec will throw exception like "SequenceFile doesn't work with GzipCodec without native-hadoop code!"

Same as other codecs which we migrated to using prepared packages (lz4, snappy), it will be better if we support GzipCodec generally without Hadoop native codec installed. Similar to BuiltInGzipDecompressor, we can use Java Deflater to support BuiltInGzipCompressor.
 2021-08-16 10:08:03 -07:00
Viraj Jasani
6342d5e523
HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-16 14:54:25 +09:00
Viraj Jasani
919a99bbb4
HDFS-16163. Avoid locking entire blockPinningFailures map (#3296) 2021-08-16 13:40:05 +08:00
Siyao Meng
3aaac8a1f6
HADOOP-17834. Bump aliyun-sdk-oss to 3.13.0 (#3261) 
Change-Id: I335d4a2cb08c75dc24ef36bdfab51111f87e0762
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-14 21:19:10 +09:00
Akira Ajisaka
3ec53a33ba
HADOOP-17799. Improve the GitHub pull request template (#3277) 
Reviewed-by: Yi Sheng <yisheng@apache.org>
 2021-08-14 21:16:15 +09:00
Renukaprasad C
b90389ae98
HADOOP-17844. Upgrade JSON smart to 2.4.7 (#3299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-14 19:55:32 +09:00
Gautham B A
77383a4d55
HDFS-15976. Make mkdtemp cross platform (#2908) 2021-08-10 09:23:28 -07:00
Viraj Jasani
6a7883431f
HADOOP-17841. Remove ListenerHandle from Hadoop registry (#3278) 2021-08-09 16:57:53 +08:00
jianghuazhu
0c7b951e03
HDFS-16151. Improve the parameter comments related to ProtobufRpcEngine2#Server(). (#3256) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-08 14:55:55 +09:00
Akira Ajisaka
3565c9477d
HADOOP-17370. Upgrade commons-compress to 1.21 (#3274) 2021-08-08 11:23:28 +09:00
Akira Ajisaka
4972e7a246
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 2021-08-07 11:27:25 +09:00
Viraj Jasani
23e2a0b202
HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache (#3266) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-07 11:20:35 +09:00
Gautham B A
4fd97e01e5
HADOOP-17787. Refactor fetching of credentials in Jenkins (#3167) 2021-08-06 11:56:27 -07:00
Bryan Beaudreault
b0b867e977
HADOOP-17837: Add unresolved endpoint value to UnknownHostException (ADDENDUM) (#3276) 2021-08-06 21:54:07 +05:30
wangzhaohui
e85c44657c
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-06 18:23:00 +08:00
Bryan Beaudreault
5e54d92e6e
HADOOP-17837: Add unresolved endpoint value to UnknownHostException (#3272) 2021-08-06 17:00:20 +08:00
wangzhaohui
a73b64f86b
HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 2021-08-06 09:27:05 +01:00
Viraj Jasani
9fe1f24ec1
HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM) (#3267) 2021-08-06 09:27:30 +08:00
Gautham B A
e80b5a08e2
HADOOP-17816. Run optional CI for changes in C (#3238) 
* HADOOP-17816. Run optional CI for changes in C

* We need to ensure that we run
  the CI for all the platforms
  when there are changes in C files.

* Trigger CI

* Revert "Trigger CI"

This reverts commit ecb88350b91c692cac128fc6f72dcd8971932041.
 2021-08-05 12:13:33 +00:00
Mehakmeet Singh
8d6a686953
HADOOP-17823. S3A S3Guard tests to skip if S3-CSE are enabled (#3263) 
Follow on to
* HADOOP-13887. Encrypt S3A data client-side with AWS SDK (S3-CSE)
* HADOOP-17817. S3A to raise IOE if both S3-CSE and S3Guard enabled

If the S3A bucket is set up to use S3-CSE encryption, all tests which turn
on S3Guard are skipped, so they don't raise any exceptions about
incompatible configurations.

Contributed by: Mehakmeet Singh
 2021-08-05 11:46:17 +01:00
Steve Loughran
a67a0fd37a
YARN-10878. move TestNMSimulator off com.google (#3268) 
Converting from a class to a lambda-expression removes all need to reference the google stuff

Contributed by Steve Loughran
 2021-08-05 11:34:10 +01:00
sumangala-patki
3450522c2f
HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs (#2845) 
Contributed by Sumangala Patki
 2021-08-04 19:45:57 +01:00
Gergely Pollak
44bab51be4 YARN-10849 Clarify testcase documentation for TestServiceAM#testContainersReleasedWhenPreLaunchFails. Contributed by Szilard Nemeth 2021-08-04 15:43:57 +02:00
Tamas Domok
0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-04 15:27:46 +02:00
zhangshuyan0
10a2526b0b
HDFS-16146. All three replicas are lost due to not adding a new DataN… (#3247) Contributed by Shuyan Zhang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-04 00:22:21 +08:00
Tamas Domok
a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests (#3248) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-03 16:19:04 +02:00
jianghuazhu
8616591b0c
HDFS-16149.Improve the parameter annotation in FairCallQueue#priorityLevels. (#3255) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
 2021-08-03 16:53:24 +08:00
Viraj Jasani
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-03 14:44:00 +09:00
Steve Loughran
4627e9c7ef
HADOOP-17822. fs.s3a.acl.default not working after S3A Audit feature (#3249) 
Fixes the regression caused by HADOOP-17511 by moving where the
option  fs.s3a.acl.default is read -doing it before the RequestFactory
is created.

Adds

* A unit test in TestRequestFactory to verify the ACLs are set
  on all file write operations.
* A new ITestS3ACannedACLs test which verifies that ACLs really
  do get all the way through.
* S3A Assumed Role delegation tokens to include the IAM permission
  s3:PutObjectAcl in the generated role.

Contributed by Steve Loughran
 2021-08-02 15:26:56 +01:00
Steve Loughran
ee466d4b40
HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out. (#3240) 
This patch cuts down the size of directory trees used for
distcp contract tests against object stores, so making
them much faster against distant/slow stores.

On abfs, the test only runs with -Dscale (as was the case for s3a already),
and has the larger scale test timeout.

After every test case, the FileSystem IOStatistics are logged,
to provide information about what IO is taking place and
what it's performance is.

There are some test cases which upload files of 1+ MiB; you can
increase the size of the upload in the option
"scale.test.distcp.file.size.kb" 
Set it to zero and the large file tests are skipped.

Contributed by Steve Loughran.
 2021-08-02 11:36:43 +01:00
zhuqi-lucas
efb3fa2bf5 YARN-10854. Support marking inactive node as untracked without configured include path. Contributed by Tao Yang. 2021-08-02 18:23:33 +08:00
jenny
e31169c864
MAPREDUCE-7258. HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's itemName from taskcounterGroup to taskCounterGroup (#1808) 
Co-authored-by: chenjuanni <chenjuanni@inspur.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-02 15:36:53 +09:00
Bobby Wang
266b1bd1bb
HADOOP-17812. NPE in S3AInputStream read() after failure to reconnect to store (#3222) 
This improves error handling after multiple failures reading data
-when the read fails and attempts to reconnect() also fail.

Contributed by Bobby Wang.
 2021-07-30 20:04:11 +01:00
Petre Bogdan Stolojan
a218038960
HADOOP-17139 Re-enable optimized copyFromLocal implementation in S3AFileSystem (#3101) 
This work
* Defines the behavior of FileSystem.copyFromLocal in filesystem.md
* Implements a high performance implementation of copyFromLocalOperation
  for S3 
* Adds a contract test for the operation: AbstractContractCopyFromLocalTest
* Implements the contract tests for Local and S3A FileSystems

Contributed by: Bogdan Stolojan
 2021-07-30 19:42:08 +01:00
Wei-Chiu Chuang
6d77f3b6cd
HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found (#3243) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-30 08:29:56 -07:00
Tamas Domok
798a0837c1
YARN-10814. Fallback to RandomSecretProvider if the secret file is empty (#3206) 
The rest endpoint would be unusable with an empty secret file
(throwing IllegalArgumentExceptions).

Any IO error would have resulted in the same fallback path.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-07-30 12:16:46 +02:00
Gautham B A
13467f4524
HADOOP-17815. Run CI for Centos 7 (#3231) 2021-07-29 10:50:57 -07:00
Benjamin Teke
8f750c54aa
YARN-10856. Prevent ATS v2 health check REST API call if the ATS service itself is disabled. (#3236) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 19:15:27 +02:00
Benjamin Teke
ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues (#3225) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 17:56:14 +02:00
Szilard Nemeth
74770c8a16 YARN-10663. Add runningApps stats in SLS. Contributed by Vadaga Ananyo Rao 2021-07-29 17:37:40 +02:00
Szilard Nemeth
54f9fff218 YARN-10628. Add node usage metrics in SLS. Contributed by Vadaga Ananyo Rao 2021-07-29 13:43:40 +02:00
minni31
d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. (#3194) 
YARN-10841. Fix token reset synchronization for UAM response token.  Contributed by Minni Mittal
 2021-07-29 14:55:39 +05:30
jianghuazhu
6f730fd25c
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-07-28 21:57:28 -07:00
He Xiaoqiao
683feaa1d4
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
 2021-07-29 11:19:17 +08:00
hchaverr
3c8a48e681 HADOOP-17819. Add extensions to ProtobufRpcEngine RequestHeaderProto. Contributed by Hector Sandoval Chaverri. (#3242) 2021-07-28 15:37:56 -07:00
Brian Loss
1d03c69963
HADOOP-17811: ABFS ExponentialRetryPolicy doesn't pick up configuration values (#3221) 
Contributed by Brian Loss.
 2021-07-28 20:22:58 +01:00
Viraj Jasani
e001f8ee39
HADOOP-17814. Provide fallbacks for identity/cost providers and backoff enable (#3230) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-29 02:10:07 +09:00
Szilard Nemeth
f2b6c03fc1 YARN-6272. TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently. Contributed by Andras Gyory & Prabhu Joseph 2021-07-28 17:02:15 +02:00
Szilard Nemeth
1b9efe58c9 YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori 2021-07-28 16:50:14 +02:00