big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,223 Commits 2 Branches 2 Tags 264 MiB
34cb595d2e
Commit Graph

23223 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ayush Saxena
34cb595d2e HDFS-14967. TestWebHDFS fails in Windows. Contributed by Renukaprasad C. 2019-11-18 18:49:58 +05:30
Gabor Bota
cad540819f
HADOOP-16484. S3A to warn or fail if S3Guard is disabled - addendum: silent for S3GuardTool (#1714). Contributed by Gabor Bota. 
Change-Id: I63b928ef5da425ef982dd4100a426fc23f64bac1
 2019-11-18 13:56:37 +01:00
Ayush Saxena
12617fad2e HDFS-14955. RBF: getQuotaUsage() on mount point should return global quota. Contributed by Jinglun. 2019-11-18 16:04:52 +05:30
Ayush Saxena
3b5a0e86c1 HDFS-14974. RBF: Make tests use free ports. Contributed by Inigo Goiri. 2019-11-18 15:49:36 +05:30
prabhujoseph
bd454348b0 YARN-9982. Fix Container API example link in NodeManager REST API doc. Contributed by Charan Hebri. 2019-11-18 15:19:47 +05:30
Andras Bokor
96c4520f89
HADOOP-16710. Testing_azure.md documentation is misleading. 
Contributed by Andras Bokor.

Change-Id: Icf07a53145936953629c7dace2e9648b7b21588d
 2019-11-17 17:04:29 +00:00
Yiqun Lin
b3119b9ab6 HDFS-14648. Implement DeadNodeDetector basic model. Contributed by Lisheng Sun. 2019-11-16 11:32:41 +08:00
Hui Fei
67f2c491fe HDFS-14802. The feature of protect directories should be used in RenameOp (#1669) 2019-11-15 12:35:37 -08:00
He Xiaoqiao
c892a879dd HDFS-14882. Consider DataNode load when #getBlockLocation. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Istvan Fajth <pifta@cloudera.com>
 2019-11-15 12:16:31 -08:00
Sandeep Nemuri
9f0610fb83 HADOOP-16654:Delete hadoop-ozone and hadoop-hdds subprojects from apache trunk 
Signed-off-by: Dinesh Chitlangia <dineshchitlangia@gmail.com>
 2019-11-15 14:53:28 -05:00
Erik Krogen
b2cc8b6b4a HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 2019-11-15 10:10:37 -08:00
Akira Ajisaka
92c28c100e
HADOOP-15097. AbstractContractDeleteTest::testDeleteNonEmptyDirRecursive with misleading path. Contributed by Xieming Li. 2019-11-15 13:49:57 +09:00
belugabehr
d0302d3849 HADOOP-16691. Unify Logging in UserGroupInformation. 
Unify Logging in UserGroupInformation.
 2019-11-14 10:28:26 -08:00
Lukas Majercak
c73334a924 HADOOP-16705. MBeanInfoBuilder puts unnecessary memory pressure on the system with a debug log. 
MBeanInfoBuilder's get() method DEBUG logs all the MBeanAttributeInfo attributes that it gathered. This can have a high memory churn that can be easily avoided.
 2019-11-14 09:54:02 -08:00
Steve Loughran
a139a87610
HADOOP-16708. HadoopExecutors cleanup to only log at debug. 
Contributed by David Mollitor.

Change-Id: Ib501a3ed6ed299ccd3c773c4dc086441a5728e73
 2019-11-14 17:51:39 +00:00
Surendra Singh Lilhore
73a386ab76 HADOOP-16585. [Tool:NNloadGeneratorMR] Multiple threads are using same id for creating file LoadGenerator#write. Contributed by Ranith Sardar. 2019-11-14 01:45:46 +05:30
Erik Krogen
586defe711 HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen. 2019-11-13 11:46:35 -08:00
Xudong Cao
df6b3162c1 HADOOP-16677. Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout. (#1687) 2019-11-12 14:19:39 -08:00
Chao Sun
97ec34e117 HDFS-14959: [SBNN read] access time should be turned off (#1706) Contributed by Chao Sun. 2019-11-12 11:35:50 -08:00
Siyao Meng
3ab98eefa5 HADOOP-16656. Document FairCallQueue configs in core-default.xml. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-11-12 11:34:24 -08:00
Inigo Goiri
40150da1e1 HDFS-14922. Prevent snapshot modification time got change on startup. Contributed by hemanthboyina. 2019-11-12 10:53:54 -08:00
Steve Loughran
990063d2af
HADOOP-16665. Filesystems to be closed if they failed during initialize(). 
Contributed by Steve Loughran.

This FileSystem instantiation so if an IOException or RuntimeException is
raised in the invocation of FileSystem.initialize() then a best-effort
attempt is made to close the FS instance; exceptions raised that there
are swallowed.

The S3AFileSystem is also modified to do its own cleanup if an
IOException is raised during its initialize() process, it being the
FS we know has the "potential" to leak threads, especially in
extension points (e.g AWS Authenticators) which spawn threads.

Change-Id: Ib84073a606c9d53bf53cbfca4629876a03894f04
 2019-11-12 18:17:21 +00:00
Yufei Gu
b83b9ab418 YARN-9537. Add configuration to disable AM preemption. Contributed by ZhouKang 2019-11-12 09:40:16 -08:00
Steve Loughran
f6697aa82b
HADOOP-16477. S3A delegation token tests fail if fs.s3a.encryption.key set. 
Contributed by Steve Loughran.

Change-Id: I843989f32472bbdefbd4fa504b26c7a614ab1cee
 2019-11-12 15:31:53 +00:00
Abhishek Modi
fb512f5087 YARN-9697. Efficient allocation of Opportunistic containers. Contributed by Abhishek Modi. 2019-11-12 16:34:04 +05:30
Masatake Iwasaki
a98c4dfc47 HADOOP-16702. Move documentation of hadoop-cos to under src directory. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-11-12 17:47:17 +09:00
Masatake Iwasaki
b988487cae HADOOP-16701. Fix broken links in site index. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-11-12 16:43:52 +09:00
Szilard Nemeth
30b93f914b YARN-9865. Capacity scheduler: add support for combined %user + %secondary_group mapping. Contributed by Manikandan R 2019-11-11 13:28:02 +01:00
Abhishek Modi
516377bfa6 YARN-9965. Fix NodeManager failing to start when Hdfs Auxillary Jar is set. Contributed by Prabhu Joseph. 2019-11-11 16:01:44 +05:30
Ayush Saxena
77934bc07b HDFS-14975. Add CR for SetECPolicyCommand usage. Contributed by Fei Hui. 2019-11-11 11:05:34 +05:30
Wang Yuxuan
b25a37c322 HDFS-14962. RBF: ConnectionPool#newConnection() error log wrong protocol class (#1699). Contributed by Yuxuan Wang. 2019-11-11 10:48:38 +05:30
Takanobu Asanuma
6663d6a5c2 HDFS-14928. UI: unifying the WebUI across different components. Contributed by Xieming Li 2019-11-11 14:04:57 +09:00
Surendra Singh Lilhore
320008bb7c HDFS-14720. DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE. Contributed by hemanthboyina. 2019-11-09 22:14:35 +05:30
Szilard Nemeth
3d249301f4 HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal 2019-11-09 15:43:53 +01:00
Szilard Nemeth
ceb9c6175e YARN-9890. [UI2] Add Application tag to the app table and app detail page. Contributed by Julia Kinga Marton 2019-11-09 14:12:04 +01:00
Szilard Nemeth
31f172fd96 YARN-9677. Make FpgaDevice and GpuDevice classes more similar to each other. Contributed by kevin su 2019-11-09 14:04:14 +01:00
Ayush Saxena
42fc8884ab HDFS-14815. RBF: Update the quota in MountTable when calling setQuota on a MountTable src. Contributed by Jinglun. 2019-11-08 08:48:52 +05:30
Ayush Saxena
247584eb63 HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan. 2019-11-07 22:49:01 +05:30
Chen Liang
dd900259c4 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 09:56:19 -08:00
Chen Liang
9d0d580031 HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 09:30:04 -08:00
Ayush Saxena
9e287054a8 HDFS-14942. Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex. Contributed by Lisheng Sun. 2019-11-06 22:02:03 +05:30
Ayush Saxena
dcf55838ae HDFS-14880. Correct the sequence of statistics & exit message in balencer. Contributed by Renukaprasad C. 2019-11-06 21:45:03 +05:30
Surendra Singh Lilhore
c36014165c HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 2019-11-06 19:28:55 +05:30
Wei-Chiu Chuang
ee8addbec4 Revert "HADOOP-16656. Document FairCallQueue configs in core-default.xml. Contributed by Siyao Meng." 
This reverts commit f9b99d2f24.
 2019-11-05 15:41:52 -08:00
Ahmed Hussein
ed302f1fed MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2019-11-05 14:55:20 -06:00
Inigo Goiri
bfb8f28cc9 HDFS-14775. Add Timestamp for longest FSN write/read lock held log. Contributed by Chen Zhang. 2019-11-05 09:19:52 -08:00
Takanobu Asanuma
d17ba85482 HADOOP-16681. mvn javadoc:javadoc fails in hadoop-aws. Contributed by Xieming Li 2019-11-05 15:24:59 +09:00
Takanobu Asanuma
7901062707 HDFS-14891. RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme. Contributed by Xieming Li 2019-11-05 15:03:09 +09:00
Ayush Saxena
b643a1cbe8 HDFS-14938. Add check if excludedNodes contain scope in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Lisheng Sun. 2019-11-05 09:23:20 +05:30
zhoukang
7d0adddf09 YARN-9689: Support proxy user for Router to support kerberos (#1654) 2019-11-04 17:13:29 -08:00