big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,178 Commits 2 Branches 2 Tags 264 MiB
157af0cb22
Commit Graph

25178 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Varun Saxena
157af0cb22 YARN-11383. Workflow priority mappings is case sensitive (#5171) 
Contributed by Aparajita Choudhary

(cherry picked from commit 2a0dc2ab2f)
 2023-03-05 23:02:17 +05:30
Tom
703158c9c6
HDFS-16896 clear ignoredNodes list when we clear deadnode list on ref… (#5322) (#5444) 
Cherry picked from: 162288bc0a

Co-authored-by: Tom McCormick <tmccormi@linkedin.com>
 2023-03-02 00:44:30 +00:00
ZanderXu
a7c1fad0c9 HDFS-16923. [SBN read] getlisting RPC to observer will throw NPE if path does not exist (#5400) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit 6bd2444815)
 2023-03-01 16:20:50 -08:00
Viraj Jasani
91ce13ea01
HDFS-16935. Fix TestFsDatasetImpl#testReportBadBlocks (#5432) 
Contributed by Viraj Jasani
 2023-03-01 18:53:52 +00:00
Steve Loughran
bca38f84af
HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429) 
POM and LICENSE fixup of transient dependencies
* Exclude hadoop-cloud-storage imports which come in with hadoop-common
* Add explicit import of hadoop's org.codehaus.jettison declaration
  to hadoop-aliyun
* Tune aliyun jars imports
* Cut duplicate and inconsistent hbase-server declarations from
  hadoop-project
* Update LICENSE-binary for the current set of libraries in the
  hadoop 3.3.5 release.

Contributed by Steve Loughran
 2023-02-28 14:05:13 +00:00
rdingankar
94b3c6dd90 HDFS-16917 Add transfer rate quantile metrics for DataNode reads (#5397) 
Co-authored-by: Ravindra Dingankar <rdingankar@linkedin.com>
 2023-02-27 15:49:26 -08:00
Owen O'Malley
c25ac781ca HDFS-16756: Fixup for 3.3 backport. 2023-02-27 15:47:10 -08:00
Simbarashe Dzinamarira
5fe19a0f01 HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used. (#5346) 2023-02-24 13:32:23 -08:00
ZanderXu
9a89deca1d HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry (#4853). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-24 13:31:53 -08:00
Steve Loughran
35e04ff52a
HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file 
+ add a link to mukund's apachecon talk

Change-Id: I3d04b385ff1312aabf2a81d034f54f124d544a54
 2023-02-23 13:26:54 +00:00
Steve Loughran
92d6826d10
Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" 
Causes HADOOP-18641. cyclonedx maven plugin breaks on recent maven releases

This reverts branch-3.3 commit b6fd322307.
 2023-02-23 11:25:50 +00:00
hchaverr
eab7215354
HADOOP-18535. Implement token storage solution based on MySQL 
Fixes #1240

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2023-02-22 14:02:13 -08:00
Ayush Saxena
84e999b35c
HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp. (#5409) 
Followup to the original HADOOP-18582.

Temporary path cleanup is re-enabled for -append jobs
as these will create temporary files when creating or overwriting files.

Contributed by Ayush Saxena
 2023-02-22 19:32:05 +00:00
Steve Loughran
ee71318d72
HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion (#5412) 
Even though DiskChecker.mkdirsWithExistsCheck() will create the directory tree,
it is only called *after* the enumeration of directories with available
space has completed.

Directories which don't exist are reported as having 0 space, therefore
the mkdirs code is never reached.

Adding a simple mkdirs() -without bothering to check the outcome-
ensures that if a dir has been deleted then it will be reconstructed
if possible. If it can't it will still have 0 bytes of space
reported and so be excluded from the allocation.

Contributed by Steve Loughran
 2023-02-22 11:50:17 +00:00
Mehakmeet Singh
a3b0135ea6
HADOOP-18633. fix test AbstractContractDistCpTest#testDistCpUpdateCheckFileSkip (#5422) 
Contributed by: Mehakmeet Singh
 2023-02-22 14:31:46 +05:30
Viraj Jasani
e47ff5f3c0
HDFS-16925. Namenode audit log to only include IP address of client (#5413) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-21 08:33:54 +08:00
Arnout Engelen
477f17be97
HADOOP-18627. Add stronger wording in 'secure mode' introduction (#5406) 
Make it more clear that when deploying Hadoop 'secure mode' is generally not optional.

Contributed by Arnout Engelen
 2023-02-17 16:31:21 +00:00
Steve Loughran
b9b999f7b1
YARN-11441. Revert YARN-10495. 
This reverts commit 1022ed91ac.
 2023-02-17 15:07:41 +00:00
Bryan Beaudreault
aa6c51364a HADOOP-18215. Enhance WritableName to be able to return aliases for classes that use serializers (#4215) 2023-02-16 11:38:20 -08:00
Viraj Jasani
8c9c68c19e
HADOOP-18628. IPC Server Connection should log host name before returning VersionMismatch error (#5385) 
Contributed by Viraj Jasani
 2023-02-15 18:23:44 +00:00
Steve Loughran
cd2401d2cc
HADOOP-18470. More in the 3.3.5 index.html about security (#5383) 
Expands on the comments in cluster config to tell people
they shouldn't be running a cluster without a private VLAN
in cloud, that Knox is good here, and unsecured clusters
without a VLAN are just computation-as-a-service to crypto miners

Contributed by Steve Loughran
 2023-02-14 17:25:20 +00:00
Mehakmeet Singh
a2ceb09323
HADOOP-18596. Distcp -update to use modification time while checking for file skip. (#5387) 
Adding toggleable support for modification time during distcp -update between two stores with incompatible checksum comparison.

Contributed by: Mehakmeet Singh <mehakmeet.singh.behl@gmail.com>
 2023-02-14 15:17:27 +05:30
Steve Vaughan
d437571fe2
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5372) 
This is a followup to the original patch, 08f58ecf07, which it supercedes
* Switch to org.apache.hadoop.io.IOUtils and closeStream.
* Use cleanupWithLogger to include error logging

Contributed by Steve Vaughan Jr
 2023-02-13 14:47:00 +00:00
Viraj Jasani
5099077f4d
HDFS-16907. Add LastHeartbeatResponseTime for BP service actor (#5349) (#5358) 
* ADDENDUM Remove unused variables from testDataNodeMXBeanLastHeartbeats (#5373)

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-12 11:53:59 +05:30
Ayush Saxena
b6d732b96b
HDFS-15368. TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally. Contributed by Xiaoqiao He. 2023-02-12 01:57:34 +05:30
Masatake Iwasaki
d85c12ee69
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. (#2467) 2023-02-12 01:50:39 +05:30
Ahmed Hussein
c17734b747
HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently (#2419) 2023-02-12 01:50:26 +05:30
Owen O'Malley
9e7a9fd46d HDFS-18324. Fix race condition in closing IPC connections. (#5371) 2023-02-10 13:56:52 -08:00
huhaiyang
de08baded6
HADOOP-18625. Fix method name of RPC.Builder#setnumReaders (#5301) 
Changes method name of RPC.Builder#setnumReaders to setNumReaders()

The original method is still there, just marked deprecated.
It is the one which should be used when working with older branches.

Contributed by Haiyang Hu
 2023-02-09 13:29:47 +00:00
Steve Vaughan
b6680907f9
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5342) 
Contributed by Steve Vaughan Jr
 2023-02-09 13:29:00 +00:00
gardenia
752f6d8213
HADOOP-18621. Resource leak in CryptoOutputStream.close() (#5347) 
When closing we need to wrap the flush() in a try .. finally, otherwise
when flush throws it will stop completion of the remainder of the
close activities and in particular the close of the underlying wrapped
stream object resulting in a resource leak.

Contributed by Colm Dougan
 2023-02-07 12:04:00 +00:00
Steve Vaughan
221221d6fb
HADOOP-18612. Avoid mixing canonical and non-canonical when performing comparisons (#5339) 
Contributed by Steve Vaughan Jr
 2023-02-06 18:30:45 +00:00
Tao Li
10b42e9306 HDFS-16902. Add Namenode status to BPServiceActor metrics and improve logging in offerservice (#5334) 
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-03 10:02:45 +08:00
Steve Vaughan
7b6a69faaa
HADOOP-18279. Cancel fileMonitoringTimer even if trustManager isn't defined (#4789) 
Co-authored-by: Steve Vaughan Jr <s_vaughan@apple.com>
 2023-02-01 13:33:34 -08:00
Viraj Jasani
f3fa4af5dc
HADOOP-18592. Sasl connection failure should log remote address. (#5294) 
Contributed by Viraj Jasani <vjasani@apache.org>

Signed-off-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2023-02-01 10:16:42 -08:00
Wei-Chiu Chuang
4836f1ec37
HADOOP-18584. [NFS GW] Fix regression after netty4 migration. (#5252) 
Reviewed-by: Tsz-Wo Nicholas Sze <szetszwo@apache.org>
(cherry picked from commit 9d47108b50)
 2023-02-01 05:33:01 -08:00
Masatake Iwasaki
004121f9cc
HADOOP-18598. maven site generation doesn't include javadocs. (#5319) 
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-31 19:49:21 +09:00
huhaiyang
80b42625cf HDFS-16888. BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#timeout should be volatile (#5296) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 88c8ac750d)
 2023-01-31 17:52:29 +09:00
Ayush Saxena
73f3196db5
HADOOP-18604. Add compile platform in the hadoop version output. (#5327). Contributed by Ayush Saxena. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-28 14:20:27 +05:30
Steve Loughran
0956994492 HADOOP-17717. Update wildfly openssl to 1.1.3.Final. (#5310) 
Contributed by Wei-Chiu Chuang
 2023-01-27 11:59:22 +00:00
kevin wan
5cd006455d HADOOP-18582. skip unnecessary cleanup logic in distcp (#5251) 
Co-authored-by: 万康 <mingge@xiaohongshu.com>
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 3b7b79b37a)
 2023-01-24 23:50:11 +00:00
Viraj Jasani
63443be5f4 HDFS-16891 Avoid the overhead of copy-on-write exception list while loading inodes sub sections in parallel (#5300) 
Reviewed-by: Stephen O'Donnell <sodonnell@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 04f3573f6a)
 2023-01-18 21:14:17 +00:00
PJ Fanning
ada06aa22e
HADOOP-18575: followup: try to avoid repeatedly hitting exceptions when transformer factories do not support attributes (#5253) 
Part of HADOOP-18469 and the hardening of XML/XSL parsers.
Followup to the main HADOOP-18575 patch, to improve performance when
working with xml/xsl engines which don't support the relevant attributes.

Include this change when backporting.

Contributed by PJ Fanning.
 2023-01-16 15:48:15 +00:00
Viraj Jasani
c5cf845d78 HDFS-16887 Log start and end of phase/step in startup progress (#5292) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 1263e024b9)
 2023-01-12 22:27:35 +00:00
skysiders
eef2fdcc29 MAPREDUCE-7375 JobSubmissionFiles don't set right permission after mkdirs (#4237) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 36bf54aba0)
 2023-01-12 21:49:15 +00:00
huangxiaoping
f5e9901e6d HADOOP-18591. Fix a typo in Trash (#5291) 
Signed-off-by: Tao Li <tomscut@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit a90e424d9f)
 2023-01-12 21:22:25 +00:00
Simbarashe Dzinamarira
dde22d522c HDFS-16886: Fixes error in documentation for StateStoreRecordsOperations. (#5290) 2023-01-11 11:47:57 -08:00
Chengbing Liu
af96e0f5b3 HDFS-16872. Fix log throttling by declaring LogThrottlingHelper as static members (#5246) 
Co-authored-by: Chengbing Liu <liuchengbing@qiyi.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit 4cf304de45)
 2023-01-10 10:04:05 -08:00
ZanderXu
a49378cb93 HDFS-16764. [SBN Read] ObserverNamenode should throw ObserverRetryOnActiveException instead of FileNotFoundException during processing of addBlock rpc (#4872) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>

(cherry picked from commit 8d221255f2)
 2023-01-10 10:03:45 -08:00
Dongjoon Hyun
b6fd322307 HADOOP-18590. Publish SBOM artifacts (#5281) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 6f99558c2e)
 2023-01-10 00:43:32 +00:00