big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,441 Commits 2 Branches 2 Tags 264 MiB
c88640c4ad
Commit Graph

25441 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Symious
c88640c4ad
HDFS-16320. Datanode retrieve slownode information from NameNode (#3654) 2021-11-22 18:04:32 +08:00
litao
20b78c8f97
HDFS-16335. Fix HDFSCommands.md (#3681) 2021-11-22 17:07:52 +09:00
Hui Fei
8e9ad2d7af
Revert "HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630)" (#3697) 2021-11-22 10:05:21 +08:00
GuoPhilipse
3d1ca6f6dc
HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-22 07:41:57 +09:00
litao
965e3874e9
HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-11-19 11:37:10 +09:00
GuoPhilipse
0c1eb0125b
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 01:02:01 +09:00
Viraj Jasani
38614d05e1
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 00:55:39 +09:00
litao
dc751df63b
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 00:42:01 +09:00
litao
5b05068fdc
HDFS-16310. RBF: Add client port to CallerContext for Router (#3635) 2021-11-18 13:46:35 +09:00
huhaiyang
91af256a5b
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630) 2021-11-17 22:41:06 +08:00
Viraj Jasani
646c470e5d
HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 2021-11-17 22:17:09 +09:00
Abhishek Das
54a1d78e16 HADOOP-17999. No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem. Contributed by Abhishek Das. (#3639) 2021-11-16 17:56:30 -08:00
Steve Loughran
3391b69692
HADOOP-18002. ABFS rename idempotency broken -remove recovery (#3641) 
Cut modtime-based rename recovery as object modification time
is not updated during rename operation.
Applications will have to use etag API of HADOOP-17979
and implement it themselves.

Why not do the HEAD and etag recovery in ABFS client?
Cuts the IO capacity in half so kills job commit performance.
The manifest committer of MAPREDUCE-7341 will do this recovery
and act as the reference implementation of the algorithm.

Contributed by: Steve Loughran
 2021-11-16 16:47:44 +05:30
Neil
5626734a36
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 2021-11-16 18:21:50 +08:00
litao
c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3643) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-16 11:19:14 +09:00
Symious
89fcbd84f9
HDFS-16313. RBF: Add metrics for each sub cluster (#3638) 2021-11-16 09:47:25 +08:00
Szilard Nemeth
b933f5f54b YARN-11001. Add docs on removing node label mapping from a node. Contributed by Manu Zhang 2021-11-15 16:02:18 +01:00
Viraj Jasani
e14a2dcbba
HADOOP-18006. maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms (#3648) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-11-15 22:57:24 +09:00
litao
573b358fce
HADOOP-18003. Add a method appendIfAbsent for CallerContext (#3644) 2021-11-15 22:44:49 +09:00
litao
60acf8434d
HDFS-16319. Add metrics doc for ReadLockLongHoldCount and WriteLockLongHoldCount (#3653). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:10:24 +05:30
GuoPhilipse
6e6f2e4baa
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 19:59:15 +05:30
Szilard Nemeth
7cb887e6c2 YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori 2021-11-12 15:43:45 +01:00
litao
7fb1bb8f35
HADOOP-18005. Correct log format for LdapGroupsMapping (#3647). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-12 20:12:18 +05:30
Szilard Nemeth
e220e88eca YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori 2021-11-12 15:33:39 +01:00
huhaiyang
db89a9411e
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 2021-11-12 10:07:34 +08:00
jianghuazhu
3d692c0936
HDFS-16305.Record the remote NameNode address when the rolling log is triggered. (#3629) 2021-11-12 09:52:02 +08:00
litao
1c1cf64616
HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-12 10:23:25 +09:00
minni31
8b159d296d
[YARN-10998] Add YARN_ROUTER_HEAPSIZE to yarn-env for routers (#3603) 2021-11-11 09:49:29 -08:00
litao
2a1a11c039
HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-10 19:22:32 +09:00
litao
ea65fc26d8
HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-10 19:16:45 +09:00
jianghuazhu
b307f18a58
HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 2021-11-10 11:32:52 +09:00
litao
cbc7fb6bca
HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-09 17:06:28 +09:00
GuoPhilipse
9b2fa7e47a
HDFS-16307. Improve HdfsBlockPlacementPolicies docs readability 2021-11-08 10:21:39 -08:00
Gautham B A
0a952e4d1f
HDFS-16304. Locate OpenSSL libs for libhdfspp (#3625) 2021-11-08 08:58:07 -08:00
Symious
d2b8d6b534
HDFS-16302. RBF: RouterRpcFairnessPolicyController record requests accepted by each nameservice (#3621) 2021-11-08 09:52:16 +08:00
Xiangyi Zhu
e5cee76785
HDFS-16273. RBF: RouterRpcFairnessPolicyController add availableHandleOnPerNs metrics (#3553) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-08 09:30:53 +09:00
Steve Loughran
45f164a854 Revert "HADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException (#3341)" 
This reverts commit 82658a22d6.
 2021-11-05 14:21:15 +00:00
sumangala-patki
e1ac10ceae
HADOOP-17863. ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler (#3332) 
Closes #3332 

Contributed by Sumangala Patki
 2021-11-05 12:53:51 +00:00
sumangala-patki
19644c0cdc
HADOOP-17862. ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator (#3331) 
closes #3331 

Contributed by Sumangala Patki
 2021-11-05 12:50:37 +00:00
Symious
6f7b965808
HDFS-16296. RouterRpcFairnessPolicyController add rejected permits for each nameservice (#3613) 2021-11-05 11:03:07 +08:00
Gautham B A
1032724aa3
HDFS-16300. Use libcrypto in Windows for libhdfspp (#3617) 2021-11-04 09:19:11 -07:00
sumangala-patki
82658a22d6
HADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException (#3341) 
Addresses transient failures in the following test classes:

* ITestAbfsStreamStatistics: Uses a filesystem level static instance to record read/write statistics, which also tracks these operations in other tests running in parallel. Marked for sequential-only run to avoid transient failure

* ITestAbfsRestOperationException: The use of a static member to track retry count causes transient failures when two tests of this class happen to run together. Switch to non-static variable for assertions on retry count

closes #3341

Contributed by Sumangala Patki
 2021-11-04 14:40:37 +00:00
jianghuazhu
72718a8dc1
HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 2021-11-04 16:43:25 +09:00
Jinhu Wu
a9c51ea57d
HADOOP-17374. support listObjectV2 (#3587) 2021-11-03 21:47:41 -07:00
jianghuazhu
18b2cca74c
HDFS-16291.Make the comment of INode#ReclaimContext more standardized. (#3602) 2021-11-04 09:51:11 +08:00
litao
359b03c83e
HDFS-16266. Add remote port information to HDFS audit log (#3538) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-04 10:16:03 +09:00
daimin
a21895a5b3
HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 2021-11-03 20:19:56 +00:00
Steve Loughran
6c6d1b64d4
HADOOP-17928. Syncable: S3A to warn and downgrade (#3585) 
This switches the default behavior of S3A output streams
to warning that Syncable.hsync() or hflush() have been
called; it's not considered an error unless the defaults
are overridden.

This avoids breaking applications which call the APIs,
at the risk of people trying to use S3 as a safe store
of streamed data (HBase WALs, audit logs etc).

Contributed by Steve Loughran.
 2021-11-02 13:26:16 +00:00
Stephen O'Donnell
2f35cc36cd
HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) (#3598) 2021-11-02 11:14:48 +00:00
jianghuazhu
618fea27d2
HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. (#3544) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-02 00:55:50 +09:00