Commit Graph

5504 Commits

Author SHA1 Message Date
Steve Loughran
6c6d1b64d4
HADOOP-17928. Syncable: S3A to warn and downgrade (#3585)
This switches the default behavior of S3A output streams
to warning that Syncable.hsync() or hflush() have been
called; it's not considered an error unless the defaults
are overridden.

This avoids breaking applications which call the APIs,
at the risk of people trying to use S3 as a safe store
of streamed data (HBase WALs, audit logs etc).

Contributed by Steve Loughran.
2021-11-02 13:26:16 +00:00
Viraj Jasani
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555) 2021-10-21 16:54:25 +09:00
Szilard Nemeth
414d40155c
YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule (#3560)
* YARN-10958. Initial commit

* Fix javadoc + behaviour

* Fix review comments

* fix checkstyle + blanks

* fix checkstyle + blanks

* Fix checkstyle + blanks
2021-10-20 10:48:42 +02:00
Masatake Iwasaki
46a7117995
HADOOP-17908. Add missing RELEASENOTES and CHANGELOG to upstream. (#3433) 2021-10-20 13:54:46 +09:00
Steve Loughran
2194b9714e
HADOOP-17945. JsonSerialization raises EOFException reading JSON data stored on google GCS (#3501)
Contributed By: Steve Loughran
2021-10-19 15:33:37 +05:30
Mehakmeet Singh
cb8c98fbb0
HADOOP-17953. S3A: Tests to lookup global or per-bucket configuration for encryption algorithm (#3525)
Followup to S3-CSE work of HADOOP-13887

Contributed by Mehakmeet Singh
2021-10-19 10:58:27 +01:00
Ahmed Hussein
cb2b7970ee
HADOOP-17123. remove guava Preconditions from Hadoop-common-project modules (#3543) 2021-10-14 19:04:00 +09:00
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
Xing Lin
97c0f96879 HADOOP-16532. Fix TestViewFsTrash to use the correct homeDir. Contributed by Xing Lin. (#3514) 2021-10-13 13:44:00 -07:00
Viraj Jasani
e103c83765
HADOOP-17952. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-common-project modules (#3503)
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
2021-10-07 11:23:35 +09:00
Viraj Jasani
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-07 10:58:29 +09:00
Ahmed Hussein
c36f9402dc
HADOOP-17930. implement non-guava Precondition checkState (#3522)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-10-07 10:55:00 +09:00
huhaiyang
68c2accc20
HDFS-16247. RBF: Fix the ProcessingAvgTime and ProxyAvgTime code comments and document metrics describe ms unit (#3511) 2021-10-04 23:52:26 +08:00
Ayush Saxena
5f0452602f
HDFS-16222. Fix ViewDFS with mount points for HDFS only API. (#3422). Contributed by Ayush Saxena.
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
2021-10-03 10:02:35 +05:30
Neil
bf9106c812
HDFS-16242. JournalMetrics should add JournalId MetricTag. (#3494)
Contributed by Max Xie
2021-10-01 09:35:14 +02:00
Ahmed Hussein
0c498f21de
HADOOP-17929. implement non-guava Precondition checkArgument (#3473)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-10-01 15:17:10 +08:00
litao
211db3fe08
HADOOP-17938. Print lockWarningThreshold in InstrumentedLock#logWarni… (#3485)
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-10-01 10:21:14 +08:00
pbacsko
7097e5b793
HADOOP-17905. Modify Text.ensureCapacity() to efficiently max out the… (#3423) 2021-09-29 17:25:29 -07:00
Petre Bogdan Stolojan
b7c2864613
HADOOP-17198. Support S3 Access Points (#3260)
Add support for S3 Access Points. This provides extra security as it
ensures applications are not working with buckets belong to third parties.

To bind a bucket to an access point, set the access point (ap) ARN,
which must be done for each specific bucket, using the pattern

fs.s3a.bucket.$BUCKET.accesspoint.arn = ARN

* The global/bucket option `fs.s3a.accesspoint.required` to
mandate that buckets must declare their access point.
* This is not compatible with S3Guard.

Consult the documentation for further details.

Contributed by Bogdan Stolojan
2021-09-29 10:54:17 +01:00
Chao Sun
7ed949600d
HADOOP-17936. Fix test failure after reverting HADOOP-16878 (#3482) 2021-09-27 13:56:22 -07:00
Gautham B A
ae6cd4cea5
HADOOP-17931. Fix typos in usage message in winutils.exe (#3490) 2021-09-27 13:41:55 -07:00
Inigo Goiri
1e8c25d528
Revert "Fix winutils typos (#3484)" (#3489)
Reverting to include JIRA number HADOOP-17931.
2021-09-27 09:40:47 -07:00
Gautham B A
0f3734a99e
Fix winutils typos (#3484) 2021-09-27 09:37:45 -07:00
Viraj Jasani
9d44f503db
HADOOP-17910. [JDK 17] TestNetUtils fails (#3481)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-27 10:12:47 +09:00
Chao Sun
1397cf2496 Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same"
This reverts commit 59f01a548e.
2021-09-25 21:32:43 -07:00
Renukaprasad C
4c516536be
HDFS-16236. Example command for daemonlog is not correct (#3476) 2021-09-25 18:32:52 +08:00
Liang-Chi Hsieh
138add2cb2
HADOOP-17868. Add more tests for BuiltInGzipCompressor (#3336) 2021-09-22 07:59:28 -07:00
Mehakmeet Singh
acffe203b8
HADOOP-17195. ABFS: OutOfMemory error while uploading huge files (#3446)
Addresses the problem of processes running out of memory when
there are many ABFS output streams queuing data to upload,
especially when the network upload bandwidth is less than the rate
data is generated.

ABFS Output streams now buffer their blocks of data to
"disk", "bytebuffer" or "array", as set in
"fs.azure.data.blocks.buffer"

When buffering via disk, the location for temporary storage
is set in "fs.azure.buffer.dir"

For safe scaling: use "disk" (default); for performance, when
confident that upload bandwidth will never be a bottleneck,
experiment with the memory options.

The number of blocks a single stream can have queued for uploading
is set in "fs.azure.block.upload.active.blocks".
The default value is 20.

Contributed by Mehakmeet Singh.
2021-09-21 12:48:06 +01:00
Neil
ae2c5ccfcf
HADOOP-17893. Improve PrometheusSink for Namenode TopMetrics (#3426)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-21 10:43:50 +09:00
Szilard Nemeth
4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth 2021-09-20 16:47:46 +02:00
Tamas Domok
f93e8fbf2d
HDFS-16129. Fixing the signature secret file misusage in HttpFS. Contributed by Tamas Domok
* HDFS-16129. Fixing the signature secret file misusage in HttpFS.

The signature secret file was not used in HttpFs.
 - if the configuration did not contain the deprecated
httpfs.authentication.signature.secret.file option then it
used the random secret provider
 - if both option (httpfs. and hadoop.http.) was set then
the HttpFSAuthenticationFilter could not read the file
because the file path was not substituted properly

!NOTE! behavioral change: the deprecated httpfs. configuration
values are overwritten with the hadoop.http. values.

The commit also contains a follow up change to the YARN-10814,
empty secret files will result in a random secret provider.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-20 14:29:50 +02:00
Rintaro Ikeda
607c20c612
HADOOP-17919. Fix command line example in Hadoop Cluster Setup documentation. (#3453) 2021-09-17 22:24:44 +09:00
Steve Loughran
5ebcd4bb92
HADOOP-17126. implement non-guava Precondition checkNotNull
This adds a new class org.apache.hadoop.util.Preconditions which is

* @Private/@Unstable
* Intended to allow us to move off Google Guava
* Is designed to be trivially backportable
  (i.e contains no references to guava classes internally)

Please use this instead of the guava equivalents, where possible.

Contributed by: Ahmed Hussein

Change-Id: Ic392451bcfe7d446184b7c995734bcca8c07286e
2021-09-17 11:06:13 +01:00
litao
71a601241c
HADOOP-17914. Print RPC response length in the exception message (#3436) 2021-09-17 14:45:14 +08:00
Mehakmeet Singh
c54bf19978
HADOOP-17871. S3A CSE: minor tuning (#3412)
This migrates the fs.s3a-server-side encryption configuration options
to a name which covers client-side encryption too.

fs.s3a.server-side-encryption-algorithm becomes fs.s3a.encryption.algorithm
fs.s3a.server-side-encryption.key becomes fs.s3a.encryption.key

The existing keys remain valid, simply deprecated and remapped
to the new values. If you want server-side encryption options
to be picked up regardless of hadoop versions, use
the old keys.

(the old key also works for CSE, though as no version of Hadoop
with CSE support has shipped without this remapping, it's less
relevant)


Contributed by: Mehakmeet Singh
2021-09-15 22:29:22 +01:00
Steve Loughran
10f3abeae7
Revert "HADOOP-17195. OutOfMemory error while performing hdfs CopyFromLocal to ABFS (#3406)" (#3443)
This reverts commit 52c024cc3a.
2021-09-15 22:27:49 +01:00
Mehakmeet Singh
52c024cc3a
HADOOP-17195. OutOfMemory error while performing hdfs CopyFromLocal to ABFS (#3406)
This migrates the fs.s3a-server-side encryption configuration options
to a name which covers client-side encryption too.

fs.s3a.server-side-encryption-algorithm becomes fs.s3a.encryption.algorithm
fs.s3a.server-side-encryption.key becomes fs.s3a.encryption.key

The existing keys remain valid, simply deprecated and remapped
to the new values. If you want server-side encryption options
to be picked up regardless of hadoop versions, use
the old keys.

(the old key also works for CSE, though as no version of Hadoop
with CSE support has shipped without this remapping, it's less
relevant)


Contributed by: Mehakmeet Singh
2021-09-15 22:27:28 +01:00
Weihao Zheng
3aa76f7e48
HADOOP-17907. FileUtil#fullyDelete deletes contents of sym-linked directory when symlink cannot be deleted because of local fs fault (#3431). Contributed by Weihao Zheng.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-09-15 23:07:21 +05:30
Ayush Saxena
d9eb5ad6d3
HADOOP-17900. Move ClusterStorageCapacityExceededException to Public from LimitedPrivate. (#3404). Contributed by Ayush Saxena. 2021-09-13 22:50:39 +05:30
LeonGao
90bc688c78
HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS (#3346) Contributed by Leon Gao
* Router to support resolving monitored namenodes with DNS

* Style

* fix style and test failure

* Add test for NNHAServiceTarget const

* Resolve comments

* Fix test

* Comments and style

* Create a simple function to extract port

* Use LambdaTestUtils.intercept

* fix javadoc

* Trigger Build
2021-09-10 16:40:08 -07:00
pbacsko
827e19271a
HADOOP-17901. Performance degradation in Text.append() after HADOOP-1… (#3411) 2021-09-10 16:01:37 -07:00
9uapaw
811fd23f23
YARN-10852. Optimise CSConfiguration getAllUserWeightsForQueue (#3392) 2021-09-10 16:59:46 +02:00
Adam Binford
4ced012f33
HADOOP-17804. Expose prometheus metrics only after a flush and dedupe with tag values (#3369)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-09 16:49:40 +09:00
Liang-Chi Hsieh
e708836641
HADOOP-17887. Remove the wrapper class GzipOutputStream (#3377) 2021-09-08 21:23:25 -07:00
Szilard Nemeth
5428d36b56 HADOOP-17857. Check real user ACLs in addition to proxied user ACLs. Contributed by Eric Payne 2021-09-08 17:27:41 +02:00
Masatake Iwasaki
ce7a5bfbd3
HADOOP-17899. Avoid using implicit dependency on junit-jupiter-api. (#3399) 2021-09-08 18:10:50 +09:00
Steve Loughran
6e3aeb1544
HADOOP-17894. CredentialProviderFactory.getProviders() recursion loading JCEKS file from S3A (#3393)
* CredentialProviderFactory to detect and report on recursion.
* S3AFS to remove incompatible providers.
* Integration Test for this.

Contributed by Steve Loughran.
2021-09-07 15:29:37 +01:00
Chris Nauroth
1d808f59d7 HADOOP-15129. Datanode caches namenode DNS lookup failure and cannot startup (#3348)
Co-authored-by:  Karthik Palaniappan

Change-Id: Id079a5319e5e83939d5dcce5fb9ebe3715ee864f
2021-09-03 18:43:48 +00:00
Viraj Jasani
99a157fa4a
HADOOP-17874. ExceptionsHandler to add terse/suppressed Exceptions in thread-safe manner (#3343)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-03 10:25:33 +09:00
Yellow Flash
4ea60b5733
HADOOP-17870. Http Filesystem to qualify relative paths. (#3338)
Contributed by Yellowflash
2021-08-31 13:55:52 +01:00