big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,980 Commits 2 Branches 2 Tags 264 MiB
e664f81ce7
Commit Graph

9672 Commits

Author SHA1 Message Date
Gautham B A
6415eb04e8
HDFS-16665. Fix duplicate sources for HDFS test (#4573) 
* The library target hdfspp_test_shim_static is
  built using the following sources, which
  causes duplicate symbols to be defined -
  - hdfs_shim.c
  - ${LIBHDFSPP_BINDING_C}/hdfs.cc
* ${LIBHDFSPP_BINDING_C}/hdfs.cc is redundant
  and removing this fixes the issue.
 2022-07-19 21:39:06 +05:30
Gautham B A
4fb799e6c5
HDFS-16464. Create only libhdfspp static libraries for Windows (#4571) 
* Added the appropriate CMake flags and
  commands to enable only statically
  linked libraries and executables to
  be built on Windows.
 2022-07-19 21:37:22 +05:30
Gautham B A
21b8952125
HDFS-16666. Pass CMake args for Windows in pom.xml (#4574) 
* This PR passes the necessary CMake args in the
  pom.xml needed for building HDFS native client
  on Windows.
* These arguments are exposed as maven options
  and can be passed from the command-line.
 2022-07-19 10:45:59 +05:30
Gautham B A
440f4c2b28
HDFS-16654. Link OpenSSL lib for CMake deps check (#4538) 
* The check_c_source_compiles fails on Windows
  while linking with an "unable to resolve
  external symbol" error.
* This PR links OpenSSL lib for this check to
  fix this issue.
 2022-07-17 20:47:30 +05:30
RuinanGu
9376b65989
HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits (#4252) 2022-07-16 04:52:12 +08:00
Murali Krishna
2835174a4c
HDFS-16652. Upgrade jquery datatable version references to v1.10.19 (#4562) 2022-07-14 18:27:07 +05:30
xuzq
6f9c4359ec
HDFS-16283. RBF: reducing the load of renewLease() RPC (#4524). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-14 07:26:40 +05:30
Gautham B A
8e39e35bea
HDFS-16466. Implement Linux permission flags on Windows (#4526) 
* HDFS-16466. Implement Linux permission flags on Windows

* statinfo.cc uses POSIX permission flags.
  These flags aren't available for Windows.
* This PR implements the equivalent flags
  on Windows to make this cross platform
  compatible.
 2022-07-08 09:29:13 +05:30
cxzl25
ea46f49b04
HDFS-16638. Add isDebugEnabled check for debug blockLogs in BlockManager. (#4480). Contributed by dzcxzl. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-04 02:36:29 +05:30
jianghuazhu
a333674785
HDFS-16647. Delete unused NameNode#FS_HDFS_IMPL_KEY. (#4525). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-03 01:47:02 +05:30
slfan1989
073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' (#4422). Contributed by fanshilun. 2022-06-29 15:20:41 +05:30
jianghuazhu
321a4844ad
HADOOP-18314. Add some description for PowerShellFencer. (#4505) 2022-06-28 19:06:39 -07:00
Ashutosh Gupta
b7edc6c60c
HDFS-16633. Fixing when Reserved Space For Replicas is not released on some cases (#4452) 
* HDFS-16633.Reserved Space For Replicas is not released on some cases

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-24 18:35:00 +05:30
Samrat
e8fd914c58
HDFS-16616. remove use of org.apache.hadoop.util.Sets (#4400) 
Co-Authored by: Samrat Deb
 2022-06-22 10:17:36 +05:30
Viraj Jasani
7a1d811197
HDFS-16637. TestHDFSCLI#testAll consistently failing (#4466). Contributed by Viraj Jasani. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-21 13:42:43 +05:30
Ashutosh Gupta
efc2761d32
HDFS-16635.Fixed javadoc error in Java 11 (#4451) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-20 11:14:31 +05:30
KevinWikant
cfceaebde6
HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-20 11:20:24 +09:00
Viraj Jasani
cb0421095b
HDFS-16634. Dynamically adjust slow peer report size on JMX metrics (#4448) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-20 09:21:00 +08:00
Viraj Jasani
e38e13be03
HADOOP-18288. Total requests and total requests per sec served by RPC servers (#4431) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-18 12:17:20 +08:00
xuzq
4893f00395
HDFS-16600. Fix deadlock of fine-grain lock for FsDatastImpl of DataNode. (#4367). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-06-17 22:05:33 +08:00
caozhiqiang
9e3fc40ecb
HDFS-16613. EC: Improve performance of decommissioning dn with many ec blocks (#4398) 2022-06-17 02:11:25 +08:00
jianghuazhu
6cbeae2e52
HDFS-16581.Print node status when executing printTopology. (#4321) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-16 19:18:58 +08:00
章锡平
f8c7e67fdc
HDFS-16628 RBF: Correct target directory when move to trash for kerberos login user. (#4424). Contributed by Xiping Zhang. 2022-06-15 21:16:24 +08:00
Gautham B A
5a40224b53
HDFS-16469. Locate protoc-gen-hrpc across platforms (#4434) 
* We use the TARGET_FILE CMake
  generator expression to get
  the location of the
  protoc-gen-hrpc CMake target.
 2022-06-15 15:28:10 +05:30
xuzq
d0715b1024
HDFS-16598. Fix DataNode FsDatasetImpl lock issue without GS checks. (#4366). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-06-14 21:48:03 +08:00
Steve Vaughan
bebf03a66e
HDFS-16625. Check assumption about PMDK availability (#4414) 
Signed-off-by: Ashutosh Gupta <ashutosh.gupta@st.niituniversity.in>
Signed-off-by: stack <stack@apache.org>
 2022-06-13 23:42:18 -07:00
slfan1989
170668b994
HDFS-16609. Fix Flakes Junit Tests that often report timeouts. (#4382). Contributed by fanshilun. 2022-06-11 20:11:21 +08:00
slfan1989
02c21ef8c3
HDFS-16627. Improve BPServiceActor#register log to add NameNode address. (#4419). Contributed by fanshilun. 2022-06-11 20:04:39 +08:00
xuzq
af5003a473
HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-06-10 12:00:56 -07:00
jianghuazhu
7f5a34dfaa
HDFS-16621.Remove unused JNStorage#getCurrentDir() (#4404). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-10 12:12:41 +05:30
Gautham B A
d557c44d94
HDFS-16463. Make dirent cross platform compatible (#4370) 
* jnihelper.c in HDFS native client uses
  dirent.h. This header file isn't available
  on Windows.
* This PR provides a cross platform
  compatible implementation for dirent
  under the XPlatform library.
 2022-06-10 09:59:07 +05:30
Ashutosh Gupta
6e11c94170
HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-09 22:39:00 +09:00
slfan1989
2ac01b273a
HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig (#4412) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-09 09:34:25 +08:00
Viraj Jasani
0a65883c50
HDFS-16618. sync_file_range error should include more volume/file info (#4402) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-07 16:53:45 +08:00
Samrat
7f08ed0d1d
HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery (#4379) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-06 17:58:49 +09:00
Viraj Jasani
25591ef51b
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits (#4357) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-06-04 06:10:14 +08:00
Gautham B A
a406f49bc0
HDFS-16602. Use "defined" directive along with #if (#4371) 
* The #if directive expects a
  boolean expression. This PR
  uses the more appropriate
  "defined" directive.
 2022-06-03 22:05:20 +05:30
Stephen O'Donnell
34a973a90e
HDFS-16610. Make fsck read timeout configurable (#4384) 2022-06-01 20:36:01 +01:00
Gautham B A
4496e2daf6
HDFS-16604. Install gtest via FetchContent_Declare in CMake (#4374) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-01 13:45:44 +09:00
slfan1989
e85f827f0a
HDFS-16603. Improve DatanodeHttpServer With Netty recommended method. (#4372) 
Co-authored-by: slfan1989 <louj1988@@>
 2022-06-01 06:45:07 +08:00
Masatake Iwasaki
3228142f53 Make upstream aware of 2.10.2 release. 
(cherry picked from commit 3dcb6367edd44c04a5ae0d195d851abac80d10a0)

Conflicts:
	hadoop-project-dist/pom.xml
 2022-06-01 00:53:01 +09:00
slfan1989
5952934ad5
HDFS-16599. Fix typo in hadoop-hdfs-rbf module (#4368). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-31 01:59:17 +05:30
slfan1989
995f7b0956
HDFS-16597. RBF: Improve RouterRpcServer#reload With Lambda (#4362). Contributed by fanshilun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:48:34 +05:30
xuzq
dc9b21feac
HDFS-16587. Allow configuring the number of JournalNodeRPCServer Handlers (#4339). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:43:37 +05:30
Ayush Saxena
96985f4c45
HDFS-15225. RBF: Add snapshot counts to content summary in router. (#4356). Contributed by Ayush Saxena. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2022-05-27 09:06:07 +05:30
Ashutosh Gupta
bee538f785
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-27 05:18:15 +08:00
Stephen O'Donnell
297f0f6d6a
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock (#4332) 
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
 2022-05-27 01:11:10 +08:00
Rishabh Sharma
0be1fde962
HDFS-16561. Handle error returned by strtol 
* strtol is used in hdfs-chmod.cc. The call
  to strtol could error out when an invalid
  input is provided.
* This PR handles the error given out by
   strtol.
 2022-05-26 11:09:49 +05:30
slfan1989
f469b0e143
HADOOP-18249. Fix getUri() in HttpRequest has been deprecated. (#4335) 
* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.
WebHdfsHandler.java req.getUri() replace uri(), req.getMethod() replace method()
HostRestrictingAuthorizationFilterHandler.java req.getUri() replace uri()
TestHostRestrictingAuthorizationFilterHandler.java remove throws Exception, channelResponse.getStatus() replace status().

* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.

* HADOOP-18249. Fix Some CheckStyle.

Co-authored-by: slfan1989 <louj1988@@>
 2022-05-24 23:11:10 +05:30
Michael Stack
5ab747222b
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4338) 
* HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BPServiceActor IllegalThreadStateException 'fatal exception and exit'

Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.

Signed-off-by: Hexiaoqiao <hexiaoqiao@apache.org>
 2022-05-23 08:45:08 -07:00