big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,456 Commits 2 Branches 2 Tags 264 MiB
53a1c7653f
Commit Graph

5326 Commits

Author SHA1 Message Date
Wei-Chiu Chuang
fa4915fdbb
Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
Wei-Chiu Chuang
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959) 
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.

Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb

* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661). Contributed by Vinayakumar B.

(cherry picked from commit c4c0683dff)

Co-authored-by: Xiaoyu Yao <xyao@apache.org>
Co-authored-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-05-17 18:57:46 -07:00
Xiaoyu Yao
3f9c9ccf46
HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider.… (#3010) 
* HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider. (#2334)

(cherry picked from commit 4c5ad57818)
 2021-05-13 16:57:58 -07:00
dependabot[bot]
b2897fdd66
HADOOP-17683. Update commons-io to 2.8.0 (#2974) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 29105ffb63)
 2021-05-12 10:58:39 +09:00
Borislav Iordanov
f5ef78d46f HADOOP-17665 Ignore missing keystore configuration in reloading mechanism 2021-05-10 15:41:56 -05:00
Mike
0f12f3e125
HADOOP-17036. TestFTPFileSystem failing as ftp server dir already exists. 
Contributed by Mikhail Pryakhin.

(cherry picked from commit 017d24e970)
 2021-05-07 14:20:29 +09:00
hchaverr
cedebf1c27 HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 
(cherry picked from commit f40e3eb059)
 2021-05-06 17:50:32 -07:00
Viraj Jasani
be34c1222a HADOOP-11616. Remove workaround for Curator's ChildReaper requiring Guava 15+ (#2973) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b93e448f9a)
 2021-05-06 04:53:06 +09:00
Takanobu Asanuma
65bf544118 HADOOP-16954. Add -S option in "Count" command to show only Snapshot Counts. Contributed by hemanthboyina. 
(cherry picked from commit b89d875f7b)
 2021-05-04 17:44:34 +01:00
Istvan Fajth
00ba886a22 HADOOP-17675. LdapGroupsMapping$LdapSslSocketFactory ClassNotFoundException (#2965) 
(cherry picked from commit 0d78d73973)
 2021-05-04 11:36:15 +01:00
kishendas
98aa4fc32c
HADOOP-17657: implement StreamCapabilities in SequenceFile.Writer and fall back to flush, if hflush is not supported (#2949) 
Co-authored-by: Kishen Das <kishen@cloudera.com>
Reviewed-by: Steve Loughran <stevel@apache.org>
(cherry picked from commit e571025f5b)
 2021-05-04 16:35:00 +08:00
lfengnan
43fac739bb
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e525ab81c)
 2021-05-02 19:19:55 +09:00
litao
f45365f201
HDFS-15975. Use LongAdder instead of AtomicLong for branch-3.3 (#2940) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-27 22:39:25 +09:00
Steve Loughran
fb71e6c91e
HADOOP-17597. Optionally downgrade on S3A Syncable calls (#2801) 
Followup to HADOOP-13327, which changed S3A output stream hsync/hflush calls
to raise an exception.

Adds a new option fs.s3a.downgrade.syncable.exceptions

When true, calls to Syncable hsync/hflush on S3A output streams will
log once at warn (for entire process life, not just the stream), then
increment IOStats with the relevant operation counter

With the downgrade option false (default)
* IOStats are incremented
* The UnsupportedOperationException current raised includes a link to the
  JIRA.

Contributed by Steve Loughran.

Change-Id: I967e077eda1d1a1a3795b4d22e003fe7997b6679
 2021-04-24 18:32:39 +01:00
Mehakmeet Singh
389d3034c6
HADOOP-17471. ABFS to collect IOStatistics (#2731) (#2950) 
The ABFS Filesystem and its input and output streams now implement
the IOStatisticSource interface and provide IOStatistics on
their interactions with Azure Storage.

This includes the min/max/mean durations of all REST API calls.

Contributed by Mehakmeet Singh <mehakmeet.singh@cloudera.com>
 2021-04-24 17:59:26 +01:00
Eric Badger
930f384e30 MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
(cherry picked from commit da93cd962c)
 2021-04-23 18:42:50 +00:00
Eric Badger
f7e382c62e YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 
(cherry picked from commit 31477a224f)
 2021-04-16 20:21:01 +00:00
Masatake Iwasaki
a3405cb56a HADOOP-17569. Building native code fails on Fedora 33. (#2886) 
(cherry picked from commit 2717203f85)
 2021-04-15 12:39:07 +00:00
He Xiaoqiao
a3cff8776a
HADOOP-17613. Log not flushed fully when daemon shutdown. Contributed by Renukaprasad C. 2021-04-14 12:28:34 +08:00
Akira Ajisaka
72355c7b6e
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be)
 2021-04-13 17:10:00 +09:00
touchida
dca2bf9dd5 HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 
(cherry picked from commit 95e6892675)
 2021-04-08 17:20:08 +08:00
Viraj Jasani
8b4b3d6fe6 HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 3f2682b92b)
 2021-04-06 14:18:31 +09:00
Ayush Saxena
a549b4a82e HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq. 
(cherry picked from commit 936bf09c37)
 2021-04-02 17:11:16 +08:00
Brahma Reddy Battula
90bbaca88b HADOOP-17587. Kinit with keytab should not display the keytab file's full path in any logs. Contributed by Ravuri Sushma sree. 
(cherry picked from commit bc7689abf5)
 2021-04-02 10:04:19 +05:30
Brahma Reddy Battula
c60e81b5a8 HADOOP-17610. DelegationTokenAuthenticator prints token information. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 478402cc74)
 2021-04-02 09:56:47 +05:30
litao
62937d15bb HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d6)
 2021-04-02 10:57:05 +09:00
Borislav Iordanov
c365149e16 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures.

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:50:28 -07:00
He Xiaoqiao
ae2d856515
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:01 +08:00
Stephen O'Donnell
56ef16468a
HADOOP-17222. Create socket address leveraging URI cache (#2817) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-30 11:59:44 +01:00
Ayush Saxena
9c9b16c957
HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2808). Contributed by Ayush Saxena. 
* HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).

* HADOOP-17531.Addendum: DistCp: Reduce memory usage on copying huge directories. (#2820)

Signed-off-by: Steve Loughran <stevel@apache.org>
 2021-03-27 09:25:25 +05:30
Xiaoyu Yao
67d52af225 HADOOP-16828. Zookeeper Delegation Token Manager fetch sequence number by batch. Contributed by Fengnan Li. 
(cherry picked from commit 6288e15118)
 2021-03-25 14:44:02 +00:00
Takanobu Asanuma
9e1e89b1b9 HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel. 
(cherry picked from commit b4d97a8dc7)
 2021-03-24 17:53:19 +09:00
kwangsun
3aae563421
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. (#2780) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c8d327a4f1)
 2021-03-22 11:45:42 +09:00
Ayush Saxena
27944772d3 HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2021-03-19 00:13:31 +05:30
Jim Brennan
ad74038e02 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 
(cherry picked from commit 299b8062f1)
 2021-03-15 20:17:02 +00:00
He Xiaoqiao
7fb49a48d1 HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 
(cherry picked from commit b1dc6c40a0)
 2021-03-14 14:56:16 +00:00
Steve Loughran
469fcdaf8f HADOOP-16721. Improve S3A rename resilience (#2742) 
The S3A connector's rename() operation now raises FileNotFoundException if
the source doesn't exist; a FileAlreadyExistsException if the destination
exists and is unsuitable for the source file/directory.

When renaming to a path which does not exist, the connector no longer checks
for the destination parent directory existing -instead it simply verifies
that there is no file immediately above the destination path.
This is needed to avoid race conditions with delete() and rename()
calls working on adjacent subdirectories.

Contributed by Steve Loughran.
 2021-03-11 12:54:15 +00:00
Mike
5ffcee8979
HADOOP-17528. SFTP File System: close the connection pool when closing a FileSystem (#2701) 
Contributed by Mike Pryakhin.

Change-Id: I59ef67c38c313f30c5e000b2fe41fcf715cf3a4b
 2021-03-09 19:58:11 +00:00
Akira Ajisaka
4462da0a84
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa)
 2021-03-04 14:56:54 +09:00
S O'Donnell
24a0304159 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee)
 2021-03-01 11:50:23 +00:00
Ahmed Hussein
792329fde9 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit e04bcb3a06)
 2021-02-26 19:56:07 +00:00
Steve Loughran
4423a7e736
HADOOP-16906. Abortable (#2684) 
Adds an Abortable.abort() interface for streams to enable output streams to be terminated; this
is implemented by the S3A connector's output stream. It allows for commit protocols
to be implemented which commit/abort work by writing to the final destination and
using the abort() call to cancel any write which is not intended to be committed.
Consult the specification document for information about the interface and its use.

Contributed by Jungtaek Lim and Steve Loughran.

Change-Id: I7fcc25e9dd8c10ce6c29f383529f3a2642a201ae
 2021-02-17 11:29:19 +00:00
Steve Loughran
98e4d516ea
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.

Change-Id: I892d768de6268f4dd6f175b3fe3b7e5bcaa91194
 2021-02-10 10:31:22 +00:00
He Xiaoqiao
f61e3f5599
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li. 2021-01-31 19:46:13 +08:00
Steve Loughran
2d124f2f5e HADOOP-17483. Magic committer is enabled by default. (#2656) 
* core-default.xml updated so that fs.s3a.committer.magic.enabled = true
* CommitConstants updated to match
* All tests which previously enabled the magic committer now rely on
  default settings. This helps make sure it is enabled.
* Docs cover the switch, mention its enabled and explain why you may
  want to disable it.
Note: this doesn't switch to using the committer -it just enables the path
rewriting magic which it depends on.

Contributed by Steve Loughran.
 2021-01-27 19:05:07 +00:00
Steve Loughran
3e1eb16837
HADOOP-17493. Revert name of DELEGATION_TOKENS_ISSUED constant/statistic (#2649) 
Follow-on to HADOOP-16830/HADOOP-17271.

Contributed by Steve Loughran.

Change-Id: I16db6e788c9fd628d3295671d7c2861c249d5ef1
 2021-01-27 16:40:27 +00:00
Steve Loughran
fb603e81f0
HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark (#2530) 
This needs SPARK-33739 in the matching spark branch in order to work

Contributed by Steve Loughran.

Change-Id: I4fe75b057159e35aacc072da3cb7343467c0c3f1
 2021-01-26 19:42:16 +00:00
Akira Ajisaka
886b245de6
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file (#2628) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d82009599a)
 2021-01-24 22:57:22 +09:00
Anton Kutuzov
dcf6d77279 HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov. 
(cherry picked from commit 91d4ba57c5)
 2021-01-22 18:09:57 -08:00
Mehakmeet Singh
d20b2deac3
HADOOP-17272. ABFS Streams to support IOStatistics API (#2604) 
Contributed by Mehakmeet Singh.

Change-Id: I3445dec84b9b9e43bb1e41f709944ea05416bd74
 2021-01-22 14:21:31 +00:00