big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,476 Commits 2 Branches 2 Tags 264 MiB
4cc33e5e37
Commit Graph

26476 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Simbarashe Dzinamarira
4cc33e5e37
HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used. (#5346) 2023-02-22 21:58:44 +00:00
slfan1989
2e997d818d
YARN-11370. [Federation] Refactor MemoryFederationStateStore code. (#5126) 2023-02-22 12:37:35 -08:00
Ayush Saxena
e8a6b2c2c4
HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp. (#5409) 
Followup to the original HADOOP-18582.

Temporary path cleanup is re-enabled for -append jobs
as these will create temporary files when creating or overwriting files.

Contributed by Ayush Saxena
 2023-02-22 19:29:41 +00:00
hchaverr
fb31393b65
HADOOP-18535. Implement token storage solution based on MySQL 
Fixes #1240

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2023-02-22 10:38:50 -08:00
Steve Loughran
11a220c6e7
HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion (#5412) 
Even though DiskChecker.mkdirsWithExistsCheck() will create the directory tree,
it is only called *after* the enumeration of directories with available
space has completed.

Directories which don't exist are reported as having 0 space, therefore
the mkdirs code is never reached.

Adding a simple mkdirs() -without bothering to check the outcome-
ensures that if a dir has been deleted then it will be reconstructed
if possible. If it can't it will still have 0 bytes of space
reported and so be excluded from the allocation.

Contributed by Steve Loughran
 2023-02-22 11:48:12 +00:00
susheel-gupta
49b8ac19f2
YARN-11408. Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method (#5278) 
* added a check of autoQueueCreationV2Disabled

Change-Id: If1e36c5969d270c1b81a4bbd2e883fa819c81f20

* added check of AutoCreateChildQueueDisabled

Change-Id: Ia011b4393ccd8d4d419a2e46b06a5237d050851c

* removed auto-create-child-queue-enabled check and emit

Change-Id: I7a154124519ecbd81379b46a238707c16db1e82a
 2023-02-22 09:46:42 +01:00
Ayush Saxena
fe5bb49ad9
Revert "YARN-11404. Add junit5 dependency to hadoop-mapreduce-client-app to fix few unit test failure. Contributed by Susheel Gupta" 
This reverts commit 8eda456d37.
 2023-02-22 07:28:13 +05:30
slfan1989
4e6e2f318c
YARN-11394. Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. (#5288) 
Contributed by Shilun Fan
 2023-02-21 14:39:32 +00:00
nao
acf82d4d55
HADOOP-18622. Upgrade ant to 1.10.13 (#5360). Contributed by Aleksandr Nikolaev. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-21 16:48:49 +05:30
Viraj Jasani
88914cada0
HDFS-16925. Namenode audit log to only include IP address of client (#5407) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-21 08:26:32 +08:00
zhtttylz
a3b500d046
HDFS-16916. Improve the use of JUnit Test in DFSClient (#5404) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-18 13:17:37 +08:00
slfan1989
7e486038ea
YARN-11439. Fix Typo of hadoop-yarn-ui README.md. (#5405) 
Co-authored-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-18 13:14:01 +08:00
Arnout Engelen
02fd87a4d8
HADOOP-18627. Add stronger wording in 'secure mode' introduction (#5406) 
Make it more clear that when deploying Hadoop 'secure mode' is generally not optional.

Contributed by Arnout Engelen
 2023-02-17 16:30:41 +00:00
Steve Loughran
10e7ca481c
YARN-11441. Revert YARN-10495. 
This reverts commit 7d3c8ef606.
 2023-02-17 15:05:06 +00:00
Bryan Beaudreault
7e19bc31b6
HADOOP-18215. Enhance WritableName to be able to return aliases for classes that use serializers (#4215) 2023-02-16 18:13:25 +00:00
Mehakmeet Singh
7a0903b743
HADOOP-18633. fix test AbstractContractDistCpTest#testDistCpUpdateCheckFileSkip (#5401) 
Contributed by: Mehakmeet Singh
 2023-02-16 10:09:06 +05:30
hfutatzhanghb
723535b788
HDFS-16914. Add some logs for updateBlockForPipeline RPC. (#5381) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-16 09:25:50 +08:00
slfan1989
a5f48eacca
YARN-11425. [Federation] Router Supports SubClusterCleaner. (#5326) 2023-02-15 14:40:34 -08:00
slfan1989
c3706597a3
YARN-11349. [Federation] Router Support DelegationToken With SQL. (#5244) 2023-02-15 14:38:41 -08:00
Ankit Saurabh
f4f2793f3b
HADOOP-18351. Reduce excess logging of errors during S3A prefetching reads (#5274) 
Contributed by Ankit Saurabh
 2023-02-15 18:28:42 +00:00
Zita Dombi
4cbe19f3a2
HDFS-16761. Namenode UI for Datanodes page not loading if any data node is down (#5390) 2023-02-15 16:16:04 +00:00
Steve Loughran
d56977e909
HADOOP-18470. More in the 3.3.5 index.html about security (#5383) 
Expands on the comments in cluster config to tell people
they shouldn't be running a cluster without a private VLAN
in cloud, that Knox is good here, and unsecured clusters
without a VLAN are just computation-as-a-service to crypto miners

Contributed by Steve Loughran
 2023-02-14 17:22:59 +00:00
SimhadriGovindappa
e2ab35084a
HADOOP-18630. Add gh-pages in asf.yaml to deploy the current trunk doc (#5393). Contributed by Simhadri Govindappa. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-14 18:13:29 +05:30
Viraj Jasani
021fcc6c5e
HADOOP-18628. IPC Server Connection should log host name before returning VersionMismatch error (#5385) 
Contributed by Viraj Jasani
 2023-02-14 11:48:48 +00:00
GuoPhilipse
fe0541b58d
HDFS-16913. Fix flaky some unit tests since they offen timeout (#5377) 
Co-authored-by: gf13871 <gf13871@ly.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-14 15:29:38 +08:00
Ayush Saxena
1def35d802
HADOOP-18524. Addendum: Deploy Hadoop trunk version website. (#5389). Contributed by Ayush Saxena. 
Reviewed-by: Vinayakumar B <vinayakumarb@apache.org>
 2023-02-14 11:05:41 +05:30
Viraj Jasani
90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) 2023-02-14 03:24:06 +08:00
Ayush Saxena
30f560554d
HADOOP-18524. Deploy Hadoop trunk version website. (#5386). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2023-02-14 00:03:02 +05:30
Tamas Domok
e4b5314991
MAPREDUCE-7433. Remove unused mapred/LoggingHttpResponseEncoder.java. (#5388) 2023-02-13 16:21:27 +01:00
Steve Vaughan
f42c89dffb
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5372) 
This is a followup to the original patch, 08f58ecf07, which it supercedes
* Switch to org.apache.hadoop.io.IOUtils and closeStream.
* Use cleanupWithLogger to include error logging

Contributed by Steve Vaughan Jr
 2023-02-13 14:31:32 +00:00
hfutatzhanghb
f3c4277576
HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath (#5276) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-11 08:00:43 +08:00
Owen O'Malley
26fba8701c
HDFS-18324. Fix race condition in closing IPC connections. (#5371) 2023-02-10 17:51:03 +00:00
Tamas Domok
151b71d7af
MAPREDUCE-7431. ShuffleHandler refactor and fix after Netty4 upgrade. (#5311) 2023-02-10 17:40:21 +01:00
Viraj Jasani
17c8cdf63c
HDFS-16907. ADDENDUM: Remove unused variables from testDataNodeMXBeanLastHeartbeats. (#5373) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-10 00:33:20 +05:30
Mehakmeet Singh
9e4f50d8a0
HADOOP-18596. Distcp -update to use modification time while checking for file skip. (#5308) 
Adding toggleable support for modification time during distcp -update between two stores with incompatible checksum comparison.

Contributed by: Mehakmeet Singh <mehakmeet.singh.behl@gmail.com>
 2023-02-09 21:31:09 +05:30
huhaiyang
113a9e40cb
HADOOP-18625. Fix method name of RPC.Builder#setnumReaders (#5301) 
Changes method name of RPC.Builder#setnumReaders to setNumReaders()

The original method is still there, just marked deprecated.
It is the one which should be used when working with older branches.

Contributed by Haiyang Hu
 2023-02-09 13:28:34 +00:00
huhaiyang
d5c046518e
HDFS-16910. Fix incorrectly initializing RandomAccessFile caused flush performance decreased for JN (#5359) 2023-02-09 10:47:57 +08:00
Viraj Jasani
4fcceff535
HADOOP-18620 Avoid using grizzly-http-* APIs (#5356) 2023-02-09 10:45:07 +08:00
slfan1989
af20841fb1
YARN-11217. [Federation] Add dumpSchedulerLogs REST APIs for Router. (#5272) 2023-02-08 11:48:38 -08:00
Steve Vaughan
08f58ecf07
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5342) 
Contributed by Steve Vaughan Jr
 2023-02-08 17:15:42 +00:00
He Xiaoqiao
3ba058a894
HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 11:19:07 +08:00
He Xiaoqiao
7e919212c4
Revert "HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB." 
This reverts commit eb04ecd29d.
 2023-02-08 11:15:22 +08:00
hfutatzhanghb
eb04ecd29d
HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 10:33:28 +08:00
hchaverri
d310642626
HDFS-16895. [RBF] NamenodeHeartbeatService should use credentials of logged in user 2023-02-07 18:45:05 +00:00
gardenia
8714403dc7
HADOOP-18621. Resource leak in CryptoOutputStream.close() (#5347) 
When closing we need to wrap the flush() in a try .. finally, otherwise
when flush throws it will stop completion of the remainder of the
close activities and in particular the close of the underlying wrapped
stream object resulting in a resource leak.

Contributed by Colm Dougan
 2023-02-07 12:01:57 +00:00
Viraj Jasani
f02c452cf1
HDFS-16907. Add LastHeartbeatResponseTime for BP service actor (#5349) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-07 09:54:40 +08:00
slfan1989
a6a9fe17e0
YARN-3657. Federation maintenance mechanisms (simple CLI and command propagation). (#5348) 2023-02-06 11:47:07 -08:00
Steve Vaughan
5f5157ac53
HADOOP-18612. Avoid mixing canonical and non-canonical when performing comparisons (#5339) 
Contributed by Steve Vaughan Jr
 2023-02-06 18:28:29 +00:00
Steve Vaughan
aed6fcee5b
HADOOP-18576. Java 11 JavaDoc fails due to missing package comments (#5344) 
Add JavaDoc comments to package-info.java to avoid errors resulting from the use of Hadoop annotations.

Contributed by Steve Vaughan Jr
 2023-02-06 18:17:57 +00:00
hfutatzhanghb
be564f5c20
[HDFS-16903]. Fix javadoc of LightWeightResizableGSet class (#5338) 2023-02-06 13:21:28 +09:00