Commit Graph

25294 Commits

Author SHA1 Message Date
Viraj Jasani
9d44f503db
HADOOP-17910. [JDK 17] TestNetUtils fails (#3481)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-27 10:12:47 +09:00
Chao Sun
1397cf2496 Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same"
This reverts commit 59f01a548e.
2021-09-25 21:32:43 -07:00
Chao Sun
2ee294b1b1 Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko."
This reverts commit 7bc305db5d.
2021-09-25 09:29:33 -07:00
Renukaprasad C
4c516536be
HDFS-16236. Example command for daemonlog is not correct (#3476) 2021-09-25 18:32:52 +08:00
AngersZhuuuu
5f9321a5d4
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 2021-09-25 18:17:44 +08:00
Wei-Chiu Chuang
87632bbacf
HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 2021-09-24 08:35:23 -07:00
litao
3113a119af
HDFS-16232. Fix java doc for BlockReaderRemote#newBlockReader (#3456) 2021-09-24 13:14:25 +08:00
Ayush Saxena
98e2781dde
HDFS-16063. Add toString to EditLogFileInputStream. Contributed by Dionisii Iuzhakov. 2021-09-23 10:45:02 +05:30
Viraj Jasani
329b95b620
HADOOP-17892. Add Hadoop code formatter in dev-support (#3387)
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Gautham B A <gautham.bangalore@gmail.com>
2021-09-23 12:34:19 +08:00
Gautham B A
35a8d48872
HDFS-16205. Make hdfs_allowSnapshot tool cross platform (#3388) 2021-09-22 10:29:38 -07:00
Liang-Chi Hsieh
138add2cb2
HADOOP-17868. Add more tests for BuiltInGzipCompressor (#3336) 2021-09-22 07:59:28 -07:00
Viraj Jasani
c7e7b2f907
HDFS-16219. RBF: Set default map tasks and bandwidth in RouterFederationRename (#3452) 2021-09-22 15:34:51 +09:00
Rintaro Ikeda
962068d2d8
HADOOP-17926. Maven-eclipse-plugin is no longer needed since Eclipse can import Maven projects by itself. (#3465) 2021-09-22 15:26:38 +09:00
LeonGao
b1431813d0
HDFS-10648. Expose Balancer metrics through Metrics2 (#3427) 2021-09-21 12:26:17 -07:00
Tamas Domok
8f4456d4a1
YARN-10961. TestCapacityScheduler: reuse appHelper where feasible. Contributed by Tamas Domok
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-21 16:13:04 +02:00
9uapaw
9f6430c9ed
YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori 2021-09-21 16:08:24 +02:00
Mehakmeet Singh
acffe203b8
HADOOP-17195. ABFS: OutOfMemory error while uploading huge files (#3446)
Addresses the problem of processes running out of memory when
there are many ABFS output streams queuing data to upload,
especially when the network upload bandwidth is less than the rate
data is generated.

ABFS Output streams now buffer their blocks of data to
"disk", "bytebuffer" or "array", as set in
"fs.azure.data.blocks.buffer"

When buffering via disk, the location for temporary storage
is set in "fs.azure.buffer.dir"

For safe scaling: use "disk" (default); for performance, when
confident that upload bandwidth will never be a bottleneck,
experiment with the memory options.

The number of blocks a single stream can have queued for uploading
is set in "fs.azure.block.upload.active.blocks".
The default value is 20.

Contributed by Mehakmeet Singh.
2021-09-21 12:48:06 +01:00
Neil
ae2c5ccfcf
HADOOP-17893. Improve PrometheusSink for Namenode TopMetrics (#3426)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-21 10:43:50 +09:00
Viraj Jasani
feee41aa00
HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink (#3386)
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-21 10:11:20 +09:00
Szilard Nemeth
4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth 2021-09-20 16:47:46 +02:00
Tamas Domok
f93e8fbf2d
HDFS-16129. Fixing the signature secret file misusage in HttpFS. Contributed by Tamas Domok
* HDFS-16129. Fixing the signature secret file misusage in HttpFS.

The signature secret file was not used in HttpFs.
 - if the configuration did not contain the deprecated
httpfs.authentication.signature.secret.file option then it
used the random secret provider
 - if both option (httpfs. and hadoop.http.) was set then
the HttpFSAuthenticationFilter could not read the file
because the file path was not substituted properly

!NOTE! behavioral change: the deprecated httpfs. configuration
values are overwritten with the hadoop.http. values.

The commit also contains a follow up change to the YARN-10814,
empty secret files will result in a random secret provider.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-20 14:29:50 +02:00
Thomas Leplus
f92c6750ec
HDFS-16230. Remove irrelevant trim() call in TestStorageRestore. (#3455). Contributed by Thomas Leplus. 2021-09-20 00:09:05 +05:30
Adam Antal
a9b2469a53
YARN-10950. Code cleanup in QueueCapacities (#3454) 2021-09-19 14:42:02 +02:00
TiborKovacsCloudera
5dcb23c1f0
YARN-10937. Fix log message arguments in LogAggregationFileController (#3450) 2021-09-19 14:35:34 +02:00
Szilard Nemeth
aa74a303ed
YARN-10913. AbstractCSQueue: Group preemption methods and fields into a separate class (#3420) 2021-09-19 13:11:56 +02:00
Gautham B A
d6eb077247
HADOOP-17902. Fix Hadoop build on Debian 10 (#3408) 2021-09-18 11:51:33 -07:00
Viraj Jasani
c9763a99c1
HDFS-16227. De-flake TestMover#testMoverWithStripedFile (#3429) 2021-09-18 19:02:45 +08:00
jianghuazhu
cd5c6395e8
HDFS-16229.Remove the use of obsolete BLOCK_DELETION_INCREMENT. (#3451). Contributed by JiangHua Zhu.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-09-18 11:06:51 +05:30
Rintaro Ikeda
607c20c612
HADOOP-17919. Fix command line example in Hadoop Cluster Setup documentation. (#3453) 2021-09-17 22:24:44 +09:00
Steve Loughran
5ebcd4bb92
HADOOP-17126. implement non-guava Precondition checkNotNull
This adds a new class org.apache.hadoop.util.Preconditions which is

* @Private/@Unstable
* Intended to allow us to move off Google Guava
* Is designed to be trivially backportable
  (i.e contains no references to guava classes internally)

Please use this instead of the guava equivalents, where possible.

Contributed by: Ahmed Hussein

Change-Id: Ic392451bcfe7d446184b7c995734bcca8c07286e
2021-09-17 11:06:13 +01:00
Viraj Jasani
7c25a77911
HDFS-16217. RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri by adding appropriate config resources (#3434)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-17 16:39:29 +09:00
LeonGao
deb52cdcbb
HDFS-16224. testBalancerWithObserverWithFailedNode times out (#3425) 2021-09-16 23:59:36 -07:00
litao
71a601241c
HADOOP-17914. Print RPC response length in the exception message (#3436) 2021-09-17 14:45:14 +08:00
adamantal
f5c76c8e31 YARN-10936. Log typo corrected. (#3449)
Contributed by: Tibor Kovacs
2021-09-17 08:27:51 +02:00
Liang-Chi Hsieh
a424878e0e
HADOOP-17891. Fix compilation error under skipShade (ADDENDUM) (#3441) 2021-09-16 09:58:28 -07:00
Gautham B A
16ca362564
HADOOP-17913. Filter deps with release labels (#3437) 2021-09-16 09:18:58 -07:00
lfengnan
4d21655d04
HDFS-16218. Use HdfsConfiguration for passing in Router principal (#3444)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-16 14:58:02 +09:00
litao
2d479309cc
HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation (#3366)
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-09-16 10:00:02 +09:00
Mehakmeet Singh
c54bf19978
HADOOP-17871. S3A CSE: minor tuning (#3412)
This migrates the fs.s3a-server-side encryption configuration options
to a name which covers client-side encryption too.

fs.s3a.server-side-encryption-algorithm becomes fs.s3a.encryption.algorithm
fs.s3a.server-side-encryption.key becomes fs.s3a.encryption.key

The existing keys remain valid, simply deprecated and remapped
to the new values. If you want server-side encryption options
to be picked up regardless of hadoop versions, use
the old keys.

(the old key also works for CSE, though as no version of Hadoop
with CSE support has shipped without this remapping, it's less
relevant)


Contributed by: Mehakmeet Singh
2021-09-15 22:29:22 +01:00
Steve Loughran
10f3abeae7
Revert "HADOOP-17195. OutOfMemory error while performing hdfs CopyFromLocal to ABFS (#3406)" (#3443)
This reverts commit 52c024cc3a.
2021-09-15 22:27:49 +01:00
Mehakmeet Singh
52c024cc3a
HADOOP-17195. OutOfMemory error while performing hdfs CopyFromLocal to ABFS (#3406)
This migrates the fs.s3a-server-side encryption configuration options
to a name which covers client-side encryption too.

fs.s3a.server-side-encryption-algorithm becomes fs.s3a.encryption.algorithm
fs.s3a.server-side-encryption.key becomes fs.s3a.encryption.key

The existing keys remain valid, simply deprecated and remapped
to the new values. If you want server-side encryption options
to be picked up regardless of hadoop versions, use
the old keys.

(the old key also works for CSE, though as no version of Hadoop
with CSE support has shipped without this remapping, it's less
relevant)


Contributed by: Mehakmeet Singh
2021-09-15 22:27:28 +01:00
Eric Badger
43f0a34dd4 YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne. 2021-09-15 20:03:45 +00:00
Weihao Zheng
3aa76f7e48
HADOOP-17907. FileUtil#fullyDelete deletes contents of sym-linked directory when symlink cannot be deleted because of local fs fault (#3431). Contributed by Weihao Zheng.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-09-15 23:07:21 +05:30
wangzhaohui
3ecaa39668
HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317)
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
2021-09-15 09:56:51 -07:00
Liang-Chi Hsieh
b8f7c7527a
HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries (#3385) 2021-09-14 11:17:18 -07:00
Benjamin Teke
5dc2f7b137
YARN-10915. AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources (#3418)
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-14 18:05:40 +02:00
Tamas Domok
783d94f5cd
YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy (#3403)
* YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:54:25 +02:00
Tamas Domok
63c892278f
YARN-10912. AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic (#3390)
- capacityConfigType update is extracted to a separate method
 - validation logic is extracted to a helper function
 - min resource must not be greater than max resource is now checked
   after the max resource is updated

Change-Id: I731c2639281721afed32c30854bafcf048d6ee28

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:30:44 +02:00
litao
4f563ff1ba
HDFS-16225. Fix typo for FederationTestUtils (#3428). Contributed by tomscut. 2021-09-14 14:28:19 +05:30
jianghuazhu
1032104b9a
HDFS-15920.Solve the problem that the value of SafeModeMonitor#RECHECK_INTERVAL can be configured. (#2831). Contributed by JiangHua Zhu.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-09-14 14:24:36 +05:30