big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Projects Wiki Activity
23,788 Commits 2 Branches 2 Tags
Commit Graph

23788 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ayush Saxena
6b7040a1cb HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 2020-05-28 13:04:22 +05:30
Wilfred Spiegelenburg
586d9427e3
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false 
Contributed by Tarun Parimi.

(cherry picked from commit 10db97df1c8562a9e29b00e60d5bde1773c09188)
 2020-05-28 13:36:21 +10:00
Szilard Nemeth
94358dbc3c YARN-10108. FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 
(cherry picked from commit ac4540dd8e2872b113a61107b0a4a66beeec8649)
 2020-05-27 13:07:17 +02:00
Szilard Nemeth
2c60bcb1c2 Revert "FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak" 
This reverts commit 9ce913a7bf720944d0305960bbf1899461b8628b.
Reverting because of missing Jira-ID in the commit message
 2020-05-27 13:07:01 +02:00
Ayush Saxena
718c8a5868 HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun. 2020-05-24 18:06:19 +05:30
Takanobu Asanuma
c44d8b8bdf HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f0eb093775f0fe4f39ef4be639ad752)
 2020-05-23 17:33:39 +09:00
Szilard Nemeth
9ce913a7bf FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 
(cherry picked from commit ac4540dd8e2872b113a61107b0a4a66beeec8649)
 2020-05-22 14:48:53 +02:00
Masatake Iwasaki
4d30c395f7 HADOOP-17040. Fix intermittent failure of ITestBlockingThreadPoolExecutorService. (#2020) 
(cherry picked from commit 968531463375ebf29ba3186c13b5f8685df10d25)
 2020-05-22 18:53:53 +09:00
Akira Ajisaka
6b54f259e7
HADOOP-17049. javax.activation-api and jakarta.activation-api define overlapping classes (#2027) 
* Removed javax.activation-api from dependency

(cherry picked from commit 52b21de1d8857151461341fbcf2fdb38c64485b1)
 2020-05-22 11:20:16 +09:00
Eric Yang
d7cf19d7c0 YARN-10228. Relax restriction of file path character in yarn.service.am.java.opts. 
Contributed by Bilwa S T via eyang
 2020-05-20 09:31:40 -07:00
Chen Liang
e452163a06
HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 
(cherry picked from commit 7bb902bc0d0c62d63a8960db444de3abb0a6ad22)
 2020-05-19 09:52:25 +09:00
He Xiaoqiao
eb045ea056 HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a3f44dacc1fa19acc4eefd1e2505e54f8629e603)
 2020-05-18 09:59:12 -07:00
Ayush Saxena
3915d1afc7 HDFS-15356. Unify configuration dfs.ha.allow.stale.reads to DFSConfigKeys. Contributed by Xiaoqiao He. 
(cherry picked from commit 178336f8a8bb291eb355bede729082f2f0382216)
 2020-05-18 09:58:59 -07:00
Wei-Chiu Chuang
240cba7e6d HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
(cherry picked from commit 2abcf7762ae74b936e1cedb60d5d2b4cc4ee86ea)
 2020-05-18 09:41:11 -07:00
Wei-Chiu Chuang
289f8acc64 Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)" 
This reverts commit ad9a6a0ee3d6cb3bde5e23c73151c0857d47ffd4.
 2020-05-18 09:41:03 -07:00
Wei-Chiu Chuang
53d22fdb88 Revert "HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He." 
This reverts commit acae31aa2824bf50fe9291148357b4f5c76a9329.
 2020-05-18 09:39:57 -07:00
Wei-Chiu Chuang
032ccba67c HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun. 
(cherry picked from commit 50caba1a92cb36ce78307d47ed7624ce216562fc)
 2020-05-18 08:42:20 -07:00
He Xiaoqiao
acae31aa28 HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a3f44dacc1fa19acc4eefd1e2505e54f8629e603)
 2020-05-18 07:10:27 -07:00
pustota2009
ad9a6a0ee3 HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.

(cherry picked from commit 86e6aa8eec538e142044e2b6415ec1caff5e9cbd)
 2020-05-18 07:05:03 -07:00
Akira Ajisaka
77587ffb1e
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 
(cherry picked from commit 27601fc79ed053ce978ac18a2c5706d32e58019f)
 2020-05-18 15:37:22 +09:00
Steve Loughran
f2f727359b Revert "HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default)." 
This reverts commit 44350fdf495f5cf1bb15b1fe6f6e9587d3de0a59.

It is related to the rollback of HADOOP-8143.

Change-Id: If48e3dd670c920ada702dc36461ff398fe9d35cc
 2020-05-14 19:20:34 +01:00
Steve Loughran
7f2cf334a8
Revert "HADOOP-8143. Change distcp to have -pb on by default." 
This reverts commit dd65eea74b1f9dde858ff34df8111e5340115511.

Change-Id: I74180cf59d5bbad8c9f66cb331535addcbea863e
 2020-05-14 19:14:39 +01:00
Surendra Singh Lilhore
7cef7109de YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng 2020-05-14 15:17:13 +05:30
Ayush Saxena
74887ab2ad YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng. 2020-05-14 00:37:09 +05:30
Akira Ajisaka
ee58c112d0
Revert "SPNEGO TLS verification" 
This reverts commit 0f27c04c23c53f625ecea75ed914375eaf6b9538.
 2020-05-13 17:14:51 +09:00
Jonathan Hung
564d3211f2 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c1226ec23841b04dd478e8b97f31abbeba)
 2020-05-12 10:49:10 -07:00
S O'Donnell
433aaeefa4 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:25:05 +01:00
Ayush Saxena
ef41229f70 HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:43:37 +05:30
Szilard Nemeth
95e4ed18a6 YARN-10160. Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2020-05-12 09:03:50 +02:00
Babneet Singh
d1e5e393c3 HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951) 
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b7be6866eda73223ab6ef1ec78d30cd)
 2020-05-11 10:45:57 -07:00
Ayush Saxena
61a4cd5539 HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun. 2020-05-11 16:52:23 +05:30
Akira Ajisaka
282427f6d1
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003) 
(cherry picked from commit 328eae9a146b2dd9857a17a0db6fcddb1de23a0d)
 2020-05-11 14:46:43 +09:00
Ayush Saxena
6ccb1cd5f1 HDFS-15250. Setting dfs.client.use.datanode.hostname to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:44:02 +05:30
Akira Ajisaka
763a79916d
HDFS-15343. TestConfiguredFailoverProxyProvider is failing. (#2001) 
(cherry picked from commit c784ba370eeb3e48c1c2c471ca72674c62a4edb7)
 2020-05-08 17:19:48 +09:00
Ayush Saxena
817814105a HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:57:06 +05:30
Masatake Iwasaki
6a0aeea60f HADOOP-17025. Fix invalid metastore configuration in S3GuardTool tests. (#1994) 
(cherry picked from commit 99840aaba662e7bb6187206b74e35132102a3b38)
 2020-05-07 12:02:15 +09:00
Uma Maheswara Rao G
edf52d29f1 HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04dfc3d4cefe7a00b63625f60ee232cfe)
 2020-05-06 15:13:33 -07:00
Ahmed Hussein
7aade932c7 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb40199aec0890b868b8d6bf2d7fe90abbf)
 2020-05-06 12:28:04 -05:00
Ayush Saxena
4e0d99c4d6 HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 17:02:40 +05:30
Wei-Chiu Chuang
78c97907cb HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998) 
Reviewed-by: Arpit Agarwal <arp@apache.org>
(cherry picked from commit adecdb8b534c536354f4f47789467ffb82dd1496)
 2020-05-05 21:25:56 -07:00
Konstantin V Shvachko
8809b43fb9 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ebb878bab991c242b5089a18881aa10abf318ea0)
 2020-05-04 10:45:18 -07:00
Ahmed Hussein
1d8350055e YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d51fe781da97f07ac4dca8dd49c98e2)
 2020-05-04 10:24:01 -05:00
Ayush Saxena
df16146e7f Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:17:14 +05:30
Akira Ajisaka
e32e1384d9
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992) 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-05-02 13:59:58 -07:00
Eric Yang
0f27c04c23
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454a5f6ea84f2cf7ac0082c555e2954a7)
 2020-05-02 21:51:12 +09:00
Ayush Saxena
ccebc9d9d0 HADOOP-16952. Add .diff to gitignore. Contributed by Ayush Saxena. 2020-05-01 14:43:34 +05:30
Akira Ajisaka
dfa7f160a5
Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
S O'Donnell
a34174fea3 HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 
(cherry picked from commit 9ca6298a9ac1ffd5c87222d5b81a561a86913840)
 2020-04-29 16:01:48 +01:00
Szilard Nemeth
8ffe1f313c YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G 
(cherry picked from commit 410c605aec308a2ccd903f60aade3aaeefcaa610)
 2020-04-29 15:54:32 +02:00
Szilard Nemeth
75433a658d YARN-10237. Add isAbsoluteResource config for queue in scheduler response. Contributed by Prabhu Joseph 2020-04-29 09:36:37 +02:00