big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,680 Commits 2 Branches 2 Tags 264 MiB
1dbaba8e70
Commit Graph

5765 Commits

Author SHA1 Message Date
Viraj Jasani
1dbaba8e70
HADOOP-18740. S3A prefetch cache blocks should be accessed by RW locks (#5675) 
Contributed by Viraj Jasani
 2023-06-07 14:05:52 +01:00
Ayush Saxena
1d0c9ab433
Revert "HADOOP-18207. Introduce hadoop-logging module (#5503)" 
This reverts commit 03a499821c.
 2023-06-05 09:34:40 +05:30
Szilard Nemeth
e0a339223a HADOOP-18709. Add curator based ZooKeeper communication support over SSL/TLS into the common library. Contributed by Ferenc Erdelyi 2023-06-04 14:40:41 -04:00
Viraj Jasani
03a499821c
HADOOP-18207. Introduce hadoop-logging module (#5503) 
Reviewed-by: Duo Zhang <zhangduo@apache.org>
 2023-06-02 18:07:34 -07:00
Steve Loughran
160b9fc3c9
HADOOP-18755. openFile builder new optLong() methods break hbase-filesystem (#5704) 
This is a followup to 
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem

Contributed by Steve Loughran
 2023-06-01 14:31:08 +01:00
Patrick GRANDJEAN
4627242c44
HADOOP-18652. Path.suffix raises NullPointerException (#5653). Contributed by Patrick Grandjean. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 05:16:55 +05:30
LiuGuH
f6770dee47
HDFS-16979. RBF: Add proxyuser port in hdfsauditlog (#5552). Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 05:02:16 +05:30
Steve Loughran
a90c722143
HADOOP-18724. [FOLLOW-UP] cherrypick changes from branch-3.3 backport (#5662) 
* move FileContext.copy() onto optLong()
* move FileUtil onto optLong()

This brings trunk into sync with the branch-3.3 changes
 2023-05-16 18:16:24 +01:00
Viraj Jasani
bef40e9427
HADOOP-18688. S3A audit header to include count of items in delete ops (#5621) 
The auditor-generated http referrer URL now includes the count of keys
to delete in the "ks" query parameter

Contributed by Viraj Jasani
 2023-05-16 10:40:16 +01:00
Steve Loughran
ad1e3a0f5b
HADOOP-18724. (followup) remove deprecation on optLong/optDouble methods (#5650) 
Somehow @Deprecated crept in to the declaration of the
new FSBuilder optLong/optDouble methods.
 2023-05-12 15:22:37 +01:00
WangYuanben
905bfa84a8
HDFS-16965. Add switch to decide whether to enable native codec. (#5520). Contributed by WangYuanben. 
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-12 04:12:02 +05:30
Steve Loughran
e76c09ac3b
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem (#5611) 
This:

1. Adds optLong, optDouble, mustLong and mustDouble
   methods to the FSBuilder interface to let callers explicitly
   passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
   now only set long values instead, so as to avoid problems
   related to overloaded methods resulting in a ".0" being appended
   to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
   the new methods
4. And the s3a code is resilient to parse errors in is numeric options
   -it will downgrade to the default.

This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.

For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.

Contributed by Steve Loughran
 2023-05-11 17:57:25 +01:00
slfan1989
a2dda0ce03
HADOOP-18359. Update commons-cli from 1.2 to 1.5. (#5095). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-10 01:42:12 +05:30
Gautham B A
c974710d8e
HADOOP-18729. Fix mvnsite on Windows 10 (#5618) 2023-05-05 13:08:58 -07:00
Tak Lon (Stephen) Wu
0e46388474
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553) 
The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them.  Applications which use the leasing APIs should migrate to the new
interface where possible.

Contributed by Stephen Wu
 2023-05-03 11:05:55 +01:00
zhangshuyan
fddc9769a5
HADOOP-18726. Set the locale to avoid printing useless logs. (#5612). Contributed by Shuyan Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-03 00:09:36 +08:00
Viraj Jasani
bfcf5dd03b
HADOOP-18697. S3A prefetch: failure of ITestS3APrefetchingInputStream#testRandomReadLargeFile (#5580) 
Contributed by Viraj Jasani
 2023-05-02 15:21:46 +01:00
Szilard Nemeth
73ca64a3ba
YARN-11450. Improvements for TestYarnConfigurationFields and TestConfigurationFieldsBase (#5455) 2023-05-02 15:52:57 +02:00
Pralabh Kumar
d75c6d9d57
HADOOP-18715. Add debug log for getting details of tokenKindMap (#5608). Contributed by Pralabh Kumar. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-29 17:28:49 +05:30
Sebastian Baunsgaard
6aac6cb212
HADOOP-18660. Filesystem Spelling Mistake (#5475). Contributed by Sebastian Baunsgaard. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-25 21:44:04 +05:30
cxzl25
2f66f0b83a
HADOOP-18694. Client.Connection#updateAddress needs to ensure that address is resolved before updating (#5542). Contributed by dzcxzl. 
Reviewed-by: Steve Vaughan <email@stevevaughan.me>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
 2023-04-25 03:52:49 +05:30
Doroszlai, Attila
5b23224970
HADOOP-18714. Wrong StringUtils.join() called in AbstractContractRootDirectoryTest (#5578) 2023-04-24 09:17:12 +02:00
LiuGuH
742e07d9c3
HADOOP-18710. Add RPC metrics for response time (#5545). Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-22 01:06:08 +05:30
Christos Bisias
9e24ed2196
HADOOP-18691. Add a CallerContext getter on the Schedulable interface (#5540) 2023-04-20 10:11:25 -07:00
Nikita Eshkeev
d07356e60e
HADOOP-18597. Simplify single node instructions for creating directories for Map Reduce. (#5305) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-20 16:12:44 +05:30
rdingankar
5119d0c72f
HDFS-16982 Use the right Quantiles Array for Inverse Quantiles snapshot (#5556) 2023-04-18 10:47:37 -07:00
Viraj Jasani
0e3aafe6c0
HADOOP-18399. S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator (#5054) 
Contributed by Viraj Jasani
 2023-04-18 16:37:48 +01:00
Steve Loughran
405ed1dde6
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558) 
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
 2023-04-18 10:12:07 +01:00
Melissa You
2b60d0c1f4
[HDFS-16971] Add read metrics for remote reads in FileSystem Statistics #5534 (#5536) 2023-04-13 09:07:42 -07:00
rdingankar
3e2ae1da00
HDFS-16949 Introduce inverse quantiles for metrics where higher numer… (#5495) 2023-04-10 08:56:00 -07:00
Viraj Jasani
b4bcbb9515
HDFS-16959. RBF: State store cache loading metrics (#5497) 2023-03-29 10:43:13 -07:00
Andras Katona
ee01c64c6c
HADOOP-18676. jettison dependency override in hadoop-common lib (#5513) 2023-03-27 09:59:02 +02:00
Ayush Saxena
b82bcbd8ad
Revert "HADOOP-18676. Fixing jettison vulnerability of hadoop-common lib (#5507)" 
This reverts commit 72b0122706.
 2023-03-25 12:04:28 +05:30
Andras Katona
72b0122706
HADOOP-18676. Fixing jettison vulnerability of hadoop-common lib (#5507) 
* HADOOP-18587. Fixing jettison vulnerability of hadoop-common lib

* no need for excluding, let it come

Change-Id: Ia6e4ad351158dd4b0510dec34bbde531a60e7654
 2023-03-24 16:31:45 +01:00
Ayush Saxena
e3cb9573e1
HADOOP-18662. ListFiles with recursive fails with FNF. (#5477). Contributed by Ayush Saxena. 
Reviewed-by: Steve Loughran <stevel@apache.org
 2023-03-23 08:30:08 +05:30
Yubi Lee
67e02a92e0
HADOOP-18666. A whitelist of endpoints to skip Kerberos authentication doesn't work for ResourceManager and Job History Server (#5480) 2023-03-22 10:54:41 +09:00
Viraj Jasani
9a8287c36f
HADOOP-18669. Remove Log4Json Layout (#5493) 2023-03-21 10:07:06 +08:00
Viraj Jasani
405bfa2800
HADOOP-18654. Remove unused custom appender TaskLogAppender (#5457) 2023-03-16 00:45:37 +08:00
Viraj Jasani
aff840c59c
HADOOP-18653. LogLevel servlet to determine log impl before using setLevel (#5456) 
The log level can only be set on Log4J log implementations;
probes are used to downgrade to a warning when other
logging back ends are used

Contributed by Viraj Jasani
 2023-03-13 12:30:12 +00:00
Steve Loughran
09469bf47d
HADOOP-18661. Fix bin/hadoop usage script terminology. (#5473) 
Followup to HADOOP-13209: s/slaves/r/workers in
the usage message you get when you type "bin/hadoop"

Contributed by Steve Loughran
 2023-03-13 12:24:36 +00:00
Viraj Jasani
28d2753d2f
HADOOP-18645. Provide keytab file key name with ServiceStateException (#5433) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-03-01 09:34:12 +08:00
rdingankar
0ca5686034
HDFS-16917 Add transfer rate quantile metrics for DataNode reads (#5397) 
Co-authored-by: Ravindra Dingankar <rdingankar@linkedin.com>
 2023-02-27 18:26:32 +00:00
Simbarashe Dzinamarira
4cc33e5e37
HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used. (#5346) 2023-02-22 21:58:44 +00:00
hchaverr
fb31393b65
HADOOP-18535. Implement token storage solution based on MySQL 
Fixes #1240

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2023-02-22 10:38:50 -08:00
Steve Loughran
11a220c6e7
HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion (#5412) 
Even though DiskChecker.mkdirsWithExistsCheck() will create the directory tree,
it is only called *after* the enumeration of directories with available
space has completed.

Directories which don't exist are reported as having 0 space, therefore
the mkdirs code is never reached.

Adding a simple mkdirs() -without bothering to check the outcome-
ensures that if a dir has been deleted then it will be reconstructed
if possible. If it can't it will still have 0 bytes of space
reported and so be excluded from the allocation.

Contributed by Steve Loughran
 2023-02-22 11:48:12 +00:00
Arnout Engelen
02fd87a4d8
HADOOP-18627. Add stronger wording in 'secure mode' introduction (#5406) 
Make it more clear that when deploying Hadoop 'secure mode' is generally not optional.

Contributed by Arnout Engelen
 2023-02-17 16:30:41 +00:00
Bryan Beaudreault
7e19bc31b6
HADOOP-18215. Enhance WritableName to be able to return aliases for classes that use serializers (#4215) 2023-02-16 18:13:25 +00:00
slfan1989
c3706597a3
YARN-11349. [Federation] Router Support DelegationToken With SQL. (#5244) 2023-02-15 14:38:41 -08:00
Ankit Saurabh
f4f2793f3b
HADOOP-18351. Reduce excess logging of errors during S3A prefetching reads (#5274) 
Contributed by Ankit Saurabh
 2023-02-15 18:28:42 +00:00
Steve Loughran
d56977e909
HADOOP-18470. More in the 3.3.5 index.html about security (#5383) 
Expands on the comments in cluster config to tell people
they shouldn't be running a cluster without a private VLAN
in cloud, that Knox is good here, and unsecured clusters
without a VLAN are just computation-as-a-service to crypto miners

Contributed by Steve Loughran
 2023-02-14 17:22:59 +00:00