big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,252 Commits 2 Branches 2 Tags
Commit Graph

24252 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jim Brennan
2204bafbfb [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f304df673d709bbc6faa9a635651c404)
 2021-03-12 17:20:37 +00:00
Akira Ajisaka
d2723b21b9
HADOOP-17582. Replace GitHub App Token with GitHub OAuth token (#2766) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b62b12d0d77de469384edd19d67092ae140faf0)
 2021-03-12 18:05:00 +09:00
Viraj Jasani
36313b38cb
HADOOP-17571 : Bump up woodstox-core to 5.3.0 due to security concerns (#2757) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-11 11:55:09 -08:00
Akira Ajisaka
0c1ba49c37
HADOOP-17570. Apply YETUS-1102 to re-enable GitHub comments (#2745) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9b78de25507bc22ff5a44da4b2a112fbccc64500)
 2021-03-11 23:26:36 +09:00
Steve Loughran
469fcdaf8f HADOOP-16721. Improve S3A rename resilience (#2742) 
The S3A connector's rename() operation now raises FileNotFoundException if
the source doesn't exist; a FileAlreadyExistsException if the destination
exists and is unsuitable for the source file/directory.

When renaming to a path which does not exist, the connector no longer checks
for the destination parent directory existing -instead it simply verifies
that there is no file immediately above the destination path.
This is needed to avoid race conditions with delete() and rename()
calls working on adjacent subdirectories.

Contributed by Steve Loughran.
 2021-03-11 12:54:15 +00:00
Akira Ajisaka
de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed17119b905dfa61ab1ce32ab554bfb57)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-11 14:57:03 +09:00
Hui Fei
34f7562abe HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee46f61686bd7fd8b62d141399e9af4dc)
 2021-03-11 10:07:48 +08:00
sumangala-patki
cdaa64458d
HADOOP-17191. ABFS: Run the tests with various combinations of configurations and publish a consolidated results (#2597) 
Contributed by Bilahari T H and Sumangala Patki
 2021-03-10 18:25:41 +00:00
Pierrick Hymbert
0e82d42d2c [HADOOP-17567] typo in MagicCommitTracker (#2749) 
Contributed by Pierrick Hymbert
 2021-03-10 15:41:21 +00:00
Mike
5ffcee8979
HADOOP-17528. SFTP File System: close the connection pool when closing a FileSystem (#2701) 
Contributed by Mike Pryakhin.

Change-Id: I59ef67c38c313f30c5e000b2fe41fcf715cf3a4b
 2021-03-09 19:58:11 +00:00
Brahma Reddy Battula
f12293fba2 YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy. 
(cherry picked from commit b2a565629dba125be5b330e84c313ba26b50e80f)
 2021-03-09 20:27:07 +05:30
Akira Ajisaka
2da6a96821
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein. 
(cherry picked from commit 63aa142b770eda31e83ca6c6e5e96cb802b8b6a9)
 2021-03-09 11:10:53 +09:00
Peter Bacsko
eb05d49349 YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-03-08 12:36:08 +01:00
Peter Bacsko
066f89af01 YARN-10672. All testcases in TestReservations are flaky. Contributed by Szilard Nemeth. 2021-03-08 11:42:59 +01:00
tomscut
c498ef4b6d HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106802c78b1400fba9f1d1c7e772dd5e7)
 2021-03-06 14:00:25 -08:00
Eric Badger
1a33878279 YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim 
Brennan.

(cherry picked from commit 9d088639bfe1556669f39f01816d6f1703b77f4f)
 2021-03-05 19:49:49 +00:00
Takanobu Asanuma
e607d03995 HADOOP-17563. Update Bouncy Castle to 1.68. (#2740) 
(cherry picked from commit 077411675679900f94adba5329d0e33a4a528793)
 2021-03-05 22:59:09 +09:00
Neil
0396a721e3 YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie 
(cherry picked from commit d615e2d3bd35780fbae379030a8231c301bda035)
 2021-03-04 14:54:28 +05:30
Akira Ajisaka
4462da0a84
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa58610c04bdc1ac7570476accdc25b7)
 2021-03-04 14:56:54 +09:00
litao
852adff194 HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380bce2ce7d8918e0772a5762fdba3122)
 2021-03-03 22:35:45 +09:00
Eric Badger
138021ce23 [MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file. 
Contributed by Eric Badger.

(cherry picked from commit a17dc4e8218b77e7fb5b5e253cb2d20014073b49)
 2021-03-02 21:49:09 +00:00
Konstantin V Shvachko
56679e83bf HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4789188f744f3a1fba7f1a9bb9b8aa4)
 2021-03-01 17:20:40 -08:00
S O'Donnell
24a0304159 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee475e5f37f85946be1a42a1b1e622a5)
 2021-03-01 11:50:23 +00:00
Ahmed Hussein
792329fde9 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit e04bcb3a061f7d89fb1353cd5ef6f550c049f36c)
 2021-02-26 19:56:07 +00:00
Eric Badger
e4dcc31114 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec45ded55e8b7fdc29f8caf85547cca3)
 2021-02-26 19:07:59 +00:00
Surendra Singh Lilhore
a6f86af39f HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3959220c6f985f940e2e731f99d8e30)
 2021-02-26 10:49:16 +08:00
Jonathan Hung
be6e99963d YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
 2021-02-25 15:09:33 -08:00
Jim Brennan
db457b056a [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 
(cherry picked from commit c373da9f883e0f58c1fff0d2213ae973d1233e14)
 2021-02-25 16:48:46 +00:00
S O'Donnell
e3ef326663 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5d0024ab052429ef2f0c0b20e1ea258)
 2021-02-24 11:04:52 +00:00
Renukaprasad C
ced08fba7c HADOOP-17510. Hadoop prints sensitive Cookie information. (#2673) 
(cherry picked from commit 924fa87010ac193a8242d59b14f9fa9d17957fe5)
 2021-02-24 17:34:07 +08:00
Masatake Iwasaki
cc42f5dafd HADOOP-17538. Add kms-default.xml and httpfs-default.xml to site index. (#2716) 
(cherry picked from commit e9a3c2950e869482a8f1a9d7027c1b0b0f1bbd0d)
 2021-02-23 22:55:10 +00:00
Ayush Saxena
6ff5da931f HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel. 2021-02-23 19:12:42 +05:30
Eric Badger
305c4e72c2 YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.

(cherry picked from commit 47420ae3edb9076d3aac7775de8712a7685dc135)
 2021-02-22 18:59:45 +00:00
Akira Ajisaka
3339ff5966
MAPREDUCE-7323. Remove job_history_summary.py. (#2712) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 8bca105655d91d949f8aa8daae656f34ed950e78)
 2021-02-22 23:16:05 +09:00
sumangala-patki
7642ddcd6f HADOOP-17537. ABFS: Correct assertion reversed in HADOOP-13327 
Contributed Sumangala Patki.
 2021-02-22 11:49:25 +00:00
Anoop Sam John
5857b781a3
HADOOP-17038 Support disabling buffered reads in ABFS positional reads. (#2646) 
- Contributed by @anoopsjohn

Change-Id: Ibd11cc9d7aed0c2cc831a01e07d0a1595f7026fb
 2021-02-22 11:46:35 +00:00
Wang Yuxuan
52a46f064a HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2fa83ad4119954b32a3a8daac9d08134)
 2021-02-22 18:14:18 +08:00
Akira Ajisaka
852db790d3
HADOOP-17534. Update Jackson to 2.10.5 and Jackson databind to 2.10.5.1 (#2708) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e119063a078f06523158dba3b9e16ca04768e920)
 2021-02-22 17:34:43 +09:00
jianghuazhu
ee0e32abdc HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8bf96cf5a1ec75134fb67fdeb955e5b6)
 2021-02-22 15:24:37 +08:00
Akira Ajisaka
ff8b79f890
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (branch-3.3) (#2709) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed956e6fa9b55758f3d2c1b92ae2597cdbb)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64
 2021-02-22 15:19:05 +09:00
zhaorenhai
c7bab342ee
HADOOP-17084 Update Dockerfile_aarch64 to use Bionic (#2103). Contributed by zhaorenhai. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 6c57be48973e182f8141d166102bdc513b944900)
 2021-02-22 13:49:51 +09:00
Akira Ajisaka
118a5747f8
HADOOP-16054. Update Dockerfile to use Bionic (#1966) 
(cherry picked from commit 81d8b71534645a2109a037115fb955351edfbf64)
 2021-02-22 13:49:29 +09:00
Eric Badger
27f00c7cd4 YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b2bdf36589da1174d4d8ee29acc57f6)
 2021-02-19 23:25:25 +00:00
Takanobu Asanuma
1125e3f0c0 HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel. 
(cherry picked from commit 25af8901a978e1b5220a961813d0e0c834010553)
 2021-02-19 14:47:16 +09:00
Hui Fei
a9d3500894 HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:14:14 +08:00
Inigo Goiri
8c8ef2f444 YARN-9017. PlacementRule order is not maintained in CS. Contributed by Bilwa S T. 
(cherry picked from commit 35010120fbbcad8618f99abf7130e53f98879a33)
 2021-02-18 20:42:26 +05:30
Inigo Goiri
97171b9b18 YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value. Contributed by Bilwa S T. 
(cherry picked from commit 108ecf992f0004dd64a7143d1c400de1361b13f3)
 2021-02-18 20:32:32 +05:30
Inigo Goiri
5cb9657320 YARN-9301. Too many InvalidStateTransitionException with SLS. Contributed by Bilwa S T. 
(cherry picked from commit 96bbc3bc972619bd830b2f935c06a1585a5470c6)
 2021-02-18 20:29:57 +05:30
Inigo Goiri
5be3a1dc7b YARN-9301. Too many InvalidStateTransitionException with SLS. Contributed by Bilwa S T. 
(cherry picked from commit 9cbd0cd2a9268ff2e8fed0af335e9c4f91c5f601)
 2021-02-18 20:29:42 +05:30
bibinchundatt
a1bebfd85e YARN-10359. Log container report only if list is not empty. Contributed by Bilwa S T. 
(cherry picked from commit 5323e83edfe63355ec38ffdaacc0c27d14cad31c)
 2021-02-18 20:23:56 +05:30