big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,988 Commits 2 Branches 2 Tags 264 MiB
4e842a7ff3
Commit Graph

24988 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Neil
c182bf65e0
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36)
 2021-11-16 22:59:25 +08:00
Renukaprasad C
3bb4a09295
HADOOP-17946. Upgrade commons-lang to 3.12.0 (#3575) 
(cherry picked from commit b923fa7a1c)
 2021-11-16 22:59:25 +08:00
Ahmed Hussein
25bcc7e3ee HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 2021-11-16 11:51:35 +09:00
litao
340dee4469
HDFS-16319. Add metrics doc for ReadLockLongHoldCount and WriteLockLongHoldCount (#3653). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:12:13 +05:30
litao
421013825f
HADOOP-18005. Correct log format for LdapGroupsMapping (#3647). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:01:09 +05:30
GuoPhilipse
b11e017777
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:00:28 +05:30
Wei-Chiu Chuang
5333e872e2
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) (#3640) 
(cherry picked from commit 356ebbbe80)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java

Change-Id: If76fd0d77fafc90fe2f2c19ab1d0c43a58510f6b

Co-authored-by: bshashikant <shashikant@apache.org>
 2021-11-12 14:32:02 +08:00
litao
c4e8826194 HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616)
 2021-11-12 10:25:40 +09:00
litao
808da62cfa HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039)
 2021-11-10 19:23:58 +09:00
litao
0db04fab20 HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit ea65fc26d8)
 2021-11-10 19:18:18 +09:00
jianghuazhu
46e69715b5 HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a58)
 2021-11-10 11:40:08 +09:00
Gautham B A
e392ce7a73
HADOOP-17988. Disable JIRA plugin for YETUS on Hadoop (#3610) 2021-11-09 22:05:10 +05:30
litao
26b7bc75ed HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit cbc7fb6bca)
 2021-11-09 17:09:05 +09:00
Steve Loughran
7b632dd22b Revert "HADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException (#3341)" 
This reverts commit 0379aebafe.
 2021-11-05 14:22:07 +00:00
sumangala-patki
689dd7bf17
HADOOP-17863. ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler (#3332) 
Closes #3332

Contributed by Sumangala Patki

Change-Id: I2abd33bd62bb734a431cccfc50a52bdeb2bf7db6
 2021-11-05 12:55:45 +00:00
Jinhu Wu
0557da6820 HADOOP-17374. support listObjectV2 (#3587) 
(cherry picked from commit a9c51ea57d)
 2021-11-04 21:45:04 -07:00
sumangala-patki
0379aebafe
HADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException (#3341) 
Addresses transient failures in the following test classes:

* ITestAbfsStreamStatistics: Uses a filesystem level static instance to record read/write statistics, which also tracks these operations in other tests running in parallel. Marked for sequential-only run to avoid transient failure

* ITestAbfsRestOperationException: The use of a static member to track retry count causes transient failures when two tests of this class happen to run together. Switch to non-static variable for assertions on retry count

closes #3341

Contributed by Sumangala Patki

Change-Id: Ied4dec35c81e94efe5f999acae4bb8fde278202e
 2021-11-04 15:57:42 +00:00
Steve Loughran
a68671eaf7
HADOOP-17928. Syncable: S3A to warn and downgrade (#3585) 
This switches the default behavior of S3A output streams
to warning that Syncable.hsync() or hflush() have been
called; it's not considered an error unless the defaults
are overridden.

This avoids breaking applications which call the APIs,
at the risk of people trying to use S3 as a safe store
of streamed data (HBase WALs, audit logs etc).

Contributed by Steve Loughran.

Change-Id: I0a02ec1e622343619f147f94158c18928a73a885
 2021-11-04 14:41:42 +00:00
jianghuazhu
4d04efb900 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1)
 2021-11-04 16:50:31 +09:00
daimin
2844b98558 HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 
(cherry picked from commit a21895a5b3)
 2021-11-03 20:41:55 +00:00
Stephen O'Donnell
9cf841b1a6 HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) (#3598) 
(cherry picked from commit 2f35cc36cd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
 2021-11-02 11:17:59 +00:00
jianghuazhu
5fbd9d333e HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. (#3544) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 618fea27d2)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java
 2021-11-02 01:04:33 +09:00
litao
7e57fc053a HDFS-16279. Print detail datanode info when process first storage report (#3564) 
(cherry picked from commit 6aa664c624)
 2021-10-29 10:28:40 +09:00
daimin
847cff7d91
HDFS-16282. Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:16:06 +05:30
Anoop Sam John
913d06ad4d
HADOOP-17770 WASB : Support disabling buffered reads in positional reads (#3233) 2021-10-22 11:45:42 +05:30
GuoPhilipse
c4f1db1966
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy (#3559). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 08:02:21 +05:30
Michael Kuchenbecker
78723e045a HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915a)
 2021-10-21 13:48:38 -07:00
Tamas Domok
e7785bb7e5
HADOOP-17974. Import statements in hadoop-aws trigger clover failures. #3572 
Contributed by Tamas Domok

Change-Id: I47da62596ce23d71709c65eb493bf656967d4415
 2021-10-21 18:43:54 +01:00
Ahmed Hussein
742d88b1c6 YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne) 2021-10-21 17:04:29 +00:00
Mehakmeet Singh
bd077c3814
HADOOP-17953. S3A: Tests to lookup global or per-bucket configuration for encryption algorithm (#3525) 
Followup to S3-CSE work of HADOOP-13887

Contributed by Mehakmeet Singh
 2021-10-21 12:03:50 +01:00
Nicholas Marion
0e79681ccf
HADOOP-17971. Exclude IBM Java security classes from being shaded/relocated. (#3567) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3e1135469e)
 2021-10-20 14:29:58 +09:00
Szilard Nemeth
6f45666d0b HADOOP-17857. Check real user ACLs in addition to proxied user ACLs. Contributed by Eric Payne 
(cherry picked from commit 5428d36b56)
 2021-10-19 20:40:30 +00:00
Steve Loughran
b8f3e54ff7 HADOOP-17945. JsonSerialization raises EOFException reading JSON data stored on google GCS (#3501) 
Contributed By: Steve Loughran
 2021-10-19 15:36:10 +05:30
daimin
6019239825 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc5)
 2021-10-18 12:02:26 +01:00
adol001
21bd015df2
HADOOP-17932. Distcp file length comparison have no effect (#3519) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 280ae1c0a9)
 2021-10-18 19:09:09 +09:00
lujiefsi
4024afec28
YARN-10976. Fix resource leak due to Files.walk (#3552) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ae95caa60e)
 2021-10-18 15:27:17 +09:00
Ayush Saxena
5b1cf17f88
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang. 2021-10-16 14:35:53 +05:30
Xing Lin
af920f138b HADOOP-16532. Fix TestViewFsTrash to use the correct homeDir. Contributed by Xing Lin. (#3514) 
(cherry picked from commit 97c0f96879)
 2021-10-13 14:58:08 -07:00
Benjamin Teke
700045896c
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues (#3504) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-10-12 18:05:50 +02:00
Masatake Iwasaki
9e2936f8d1
HADOOP-17424. Replace HTrace with No-Op tracer (#3520) 
(cherry picked from commit 1a205cc3ad)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTracing.java

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
 2021-10-12 00:07:09 +09:00
Masatake Iwasaki
99a1b185b6 HADOOP-17925. BUILDING.txt should not encourage to activate docs profile on building binary artifacts. (#3509) 
(cherry picked from commit 65d9523aa2)
 2021-10-11 08:57:07 +00:00
langlaile1221
a35a6ec06b
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
(cherry picked from commit 88d8c3d0c8)
 2021-10-11 16:22:46 +08:00
Takanobu Asanuma
038fa67db4 HADOOP-17955. Bump netty to the latest 4.1.68. (#3528) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit cf4d2d826c)
 2021-10-08 10:37:14 +09:00
Viraj Jasani
77ee5a4266
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8071dbb9c6)
 2021-10-07 11:00:19 +09:00
Ahmed Hussein
2cdc6a245d HADOOP-17930. implement non-guava Precondition checkState (#3522) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit c36f9402dc)
 2021-10-07 10:57:20 +09:00
S O'Donnell
f437b3bdaf HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell 
(cherry picked from commit a30231ff8e)
 2021-10-06 15:39:34 +01:00
Viraj Jasani
fd3069d70c HADOOP-17947. Additional element types for VisibleForTesting (ADDENDUM) (#3521) 
(cherry picked from commit 783e4805e7)
 2021-10-06 02:18:54 +09:00
Steve Loughran
6f7b45641a
HADOOP-17922. move to fs.s3a.encryption.algorithm - JCEKS integration (#3466) 
The ordering of the resolution of new and deprecated s3a encryption options
& secrets is the same when JCEKS and other hadoop credentials stores are used
to store them as when they are in XML files: per-bucket settings always take
priority over global values, even when the bucket-level options use the
old option names.

Contributed by Mehakmeet Singh and Steve Loughran

Change-Id: I871672071efa2eb6b600cb2658fceeef57f658a3
 2021-10-05 11:39:43 +01:00
Mehakmeet Singh
769059c2f5
HADOOP-17871. S3A CSE: minor tuning (#3412) 
This migrates the fs.s3a-server-side encryption configuration options
to a name which covers client-side encryption too.

fs.s3a.server-side-encryption-algorithm becomes fs.s3a.encryption.algorithm
fs.s3a.server-side-encryption.key becomes fs.s3a.encryption.key

The existing keys remain valid, simply deprecated and remapped
to the new values. If you want server-side encryption options
to be picked up regardless of hadoop versions, use
the old keys.

(the old key also works for CSE, though as no version of Hadoop
with CSE support has shipped without this remapping, it's less
relevant)

Contributed by: Mehakmeet Singh

Change-Id: I51804b21b287dbce18864f0a6ad17126aba2b281
 2021-10-05 11:39:25 +01:00
Mehakmeet Singh
abb367aec6
HADOOP-17817.HADOOP-17823. S3A to raise IOE if both S3-CSE and S3Guard enabled (#3239) 
S3A S3Guard tests to skip if S3-CSE are enabled (#3263)

    Follow on to
    * HADOOP-13887. Encrypt S3A data client-side with AWS SDK (S3-CSE)

    If the S3A bucket is set up to use S3-CSE encryption, all tests which turn
    on S3Guard are skipped, so they don't raise any exceptions about
    incompatible configurations.

Contributed by Mehakmeet Singh

Change-Id: I9f4188109b56a1f4e5a31fae265d980c5795db1e
 2021-10-05 11:38:57 +01:00