big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,512 Commits 2 Branches 2 Tags 264 MiB
39e6f2d191
Commit Graph

24512 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eric Badger
cd417f17ae YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu. 
(cherry picked from commit 49f89f1d3d)
 2021-03-17 18:16:59 +00:00
Viraj Jasani
17307ee88d HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348)
 2021-03-17 15:13:27 +09:00
Takanobu Asanuma
aec9d2cff7 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8)
 2021-03-17 14:15:53 +09:00
touchida
f4a9327002
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc7)
 2021-03-17 12:01:15 +09:00
Jim Brennan
ad74038e02 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 
(cherry picked from commit 299b8062f1)
 2021-03-15 20:17:02 +00:00
Eric Badger
1022ed91ac YARN-10495. make the rpath of container-executor configurable. Contributed by 
angerszhu.

(cherry picked from commit 7d3c8ef606)
 2021-03-15 20:01:24 +00:00
Eric Payne
f5810ea83c YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7)
 2021-03-15 19:14:19 +00:00
Chao Sun
e2aaa10512 HADOOP-17532. Yarn Job execution get failed when LZ4 Compression Codec is used. Contributed Bhavik Patel. 2021-03-14 21:20:44 -07:00
He Xiaoqiao
7fb49a48d1 HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 
(cherry picked from commit b1dc6c40a0)
 2021-03-14 14:56:16 +00:00
Ayush Saxena
28e89509a3 HADOOP-17586. Upgrade org.codehaus.woodstox:stax2-api to 4.2.1. (#2769). Contributed by Ayush Saxena. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-03-13 15:10:27 +05:30
Jim Brennan
2204bafbfb [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f3)
 2021-03-12 17:20:37 +00:00
Akira Ajisaka
d2723b21b9
HADOOP-17582. Replace GitHub App Token with GitHub OAuth token (#2766) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b62b12d0d)
 2021-03-12 18:05:00 +09:00
Viraj Jasani
36313b38cb
HADOOP-17571 : Bump up woodstox-core to 5.3.0 due to security concerns (#2757) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-11 11:55:09 -08:00
Akira Ajisaka
0c1ba49c37
HADOOP-17570. Apply YETUS-1102 to re-enable GitHub comments (#2745) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9b78de2550)
 2021-03-11 23:26:36 +09:00
Steve Loughran
469fcdaf8f HADOOP-16721. Improve S3A rename resilience (#2742) 
The S3A connector's rename() operation now raises FileNotFoundException if
the source doesn't exist; a FileAlreadyExistsException if the destination
exists and is unsuitable for the source file/directory.

When renaming to a path which does not exist, the connector no longer checks
for the destination parent directory existing -instead it simply verifies
that there is no file immediately above the destination path.
This is needed to avoid race conditions with delete() and rename()
calls working on adjacent subdirectories.

Contributed by Steve Loughran.
 2021-03-11 12:54:15 +00:00
Akira Ajisaka
de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-11 14:57:03 +09:00
Hui Fei
34f7562abe HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee4)
 2021-03-11 10:07:48 +08:00
sumangala-patki
cdaa64458d
HADOOP-17191. ABFS: Run the tests with various combinations of configurations and publish a consolidated results (#2597) 
Contributed by Bilahari T H and Sumangala Patki
 2021-03-10 18:25:41 +00:00
Pierrick Hymbert
0e82d42d2c [HADOOP-17567] typo in MagicCommitTracker (#2749) 
Contributed by Pierrick Hymbert
 2021-03-10 15:41:21 +00:00
Mike
5ffcee8979
HADOOP-17528. SFTP File System: close the connection pool when closing a FileSystem (#2701) 
Contributed by Mike Pryakhin.

Change-Id: I59ef67c38c313f30c5e000b2fe41fcf715cf3a4b
 2021-03-09 19:58:11 +00:00
Brahma Reddy Battula
f12293fba2 YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy. 
(cherry picked from commit b2a565629d)
 2021-03-09 20:27:07 +05:30
Akira Ajisaka
2da6a96821
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein. 
(cherry picked from commit 63aa142b77)
 2021-03-09 11:10:53 +09:00
Peter Bacsko
eb05d49349 YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-03-08 12:36:08 +01:00
Peter Bacsko
066f89af01 YARN-10672. All testcases in TestReservations are flaky. Contributed by Szilard Nemeth. 2021-03-08 11:42:59 +01:00
tomscut
c498ef4b6d HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106)
 2021-03-06 14:00:25 -08:00
Eric Badger
1a33878279 YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim 
Brennan.

(cherry picked from commit 9d088639bf)
 2021-03-05 19:49:49 +00:00
Takanobu Asanuma
e607d03995 HADOOP-17563. Update Bouncy Castle to 1.68. (#2740) 
(cherry picked from commit 0774116756)
 2021-03-05 22:59:09 +09:00
Neil
0396a721e3 YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie 
(cherry picked from commit d615e2d3bd)
 2021-03-04 14:54:28 +05:30
Akira Ajisaka
4462da0a84
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa)
 2021-03-04 14:56:54 +09:00
litao
852adff194 HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380)
 2021-03-03 22:35:45 +09:00
Eric Badger
138021ce23 [MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file. 
Contributed by Eric Badger.

(cherry picked from commit a17dc4e821)
 2021-03-02 21:49:09 +00:00
Konstantin V Shvachko
56679e83bf HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4)
 2021-03-01 17:20:40 -08:00
S O'Donnell
24a0304159 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee)
 2021-03-01 11:50:23 +00:00
Ahmed Hussein
792329fde9 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit e04bcb3a06)
 2021-02-26 19:56:07 +00:00
Eric Badger
e4dcc31114 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec)
 2021-02-26 19:07:59 +00:00
Surendra Singh Lilhore
a6f86af39f HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3)
 2021-02-26 10:49:16 +08:00
Jonathan Hung
be6e99963d YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
 2021-02-25 15:09:33 -08:00
Jim Brennan
db457b056a [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 
(cherry picked from commit c373da9f88)
 2021-02-25 16:48:46 +00:00
S O'Donnell
e3ef326663 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5)
 2021-02-24 11:04:52 +00:00
Renukaprasad C
ced08fba7c HADOOP-17510. Hadoop prints sensitive Cookie information. (#2673) 
(cherry picked from commit 924fa87010)
 2021-02-24 17:34:07 +08:00
Masatake Iwasaki
cc42f5dafd HADOOP-17538. Add kms-default.xml and httpfs-default.xml to site index. (#2716) 
(cherry picked from commit e9a3c2950e)
 2021-02-23 22:55:10 +00:00
Ayush Saxena
6ff5da931f HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel. 2021-02-23 19:12:42 +05:30
Eric Badger
305c4e72c2 YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.

(cherry picked from commit 47420ae3ed)
 2021-02-22 18:59:45 +00:00
Akira Ajisaka
3339ff5966
MAPREDUCE-7323. Remove job_history_summary.py. (#2712) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 8bca105655)
 2021-02-22 23:16:05 +09:00
sumangala-patki
7642ddcd6f HADOOP-17537. ABFS: Correct assertion reversed in HADOOP-13327 
Contributed Sumangala Patki.
 2021-02-22 11:49:25 +00:00
Anoop Sam John
5857b781a3
HADOOP-17038 Support disabling buffered reads in ABFS positional reads. (#2646) 
- Contributed by @anoopsjohn

Change-Id: Ibd11cc9d7aed0c2cc831a01e07d0a1595f7026fb
 2021-02-22 11:46:35 +00:00
Wang Yuxuan
52a46f064a HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2f)
 2021-02-22 18:14:18 +08:00
Akira Ajisaka
852db790d3
HADOOP-17534. Update Jackson to 2.10.5 and Jackson databind to 2.10.5.1 (#2708) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e119063a07)
 2021-02-22 17:34:43 +09:00
jianghuazhu
ee0e32abdc HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8b)
 2021-02-22 15:24:37 +08:00
Akira Ajisaka
ff8b79f890
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (branch-3.3) (#2709) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed956)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64
 2021-02-22 15:19:05 +09:00