big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,513 Commits 2 Branches 2 Tags 264 MiB
22e7567475
Commit Graph

5515 Commits

Author SHA1 Message Date
Viraj Jasani
b8a98e4f82 HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit c488abbc79)
 2021-06-21 10:28:05 +09:00
Takanobu Asanuma
25138c98bf HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml (#3099) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9e7c7ad129)
 2021-06-17 10:00:36 +09:00
Mingliang Liu
91bcfbd72e
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042)" 
This reverts commit 8c0f948054.
 2021-06-11 00:35:41 -07:00
Steve Loughran
4ac9123619
HADOOP-17631. Configuration ${env.VAR:-FALLBACK} to eval FALLBACK when restrictSystemProps=true (#2977) 
Contributed by Steve Loughran.

Change-Id: I9b82109eddeb659c01896152cf603d458e2a04cd
 2021-06-08 22:05:00 +01:00
July
8c0f948054
HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042) 
Contributed by kaifeiYi (yikf).

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2021-06-04 13:37:24 -07:00
Konstantin V Shvachko
c5535caf6e HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f9)
 2021-05-27 18:53:46 -07:00
Steve Loughran
464bbd5b7c
HADOOP-17511. Add audit/telemetry logging to S3A connector (#2807) 
The S3A connector supports
"an auditor", a plugin which is invoked
at the start of every filesystem API call,
and whose issued "audit span" provides a context
for all REST operations against the S3 object store.

The standard auditor sets the HTTP Referrer header
on the requests with information about the API call,
such as process ID, operation name, path,
and even job ID.

If the S3 bucket is configured to log requests, this
information will be preserved there and so can be used
to analyze and troubleshoot storage IO.

Contributed by Steve Loughran.

Change-Id: Ic0a105c194342ed2d529833ecc42608e8ba2f258
 2021-05-25 12:55:38 +01:00
Mehakmeet Singh
a786847b8f
HADOOP-17670. S3AFS and ABFS to log IOStats at DEBUG mode or optionally at INFO level in close() (#2963) 
When the S3A and ABFS filesystems are closed,
their IOStatistics are logged at debug in the log:

org.apache.hadoop.fs.statistics.IOStatisticsLogging

Set `fs.iostatistics.logging.level` to `info` for the statistics
to be logged at info. (also: `warn` or `error` for even higher
log levels).

Contributed by: Mehakmeet Singh

Change-Id: I56d44ad89fc1c0dd4baf701681834e7fd96c544f
 2021-05-24 13:04:20 +01:00
Vinayakumar B
dbf1ef4aff
HDFS-15790. Make ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist (#2767) 
(cherry picked from commit 2bbeae3240)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
 2021-05-24 18:00:38 +08:00
Wei-Chiu Chuang
86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959)" 
This reverts commit 4ffe5eb1dd.
 2021-05-24 17:37:18 +08:00
Viraj Jasani
87e32fbb8d HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548)
 2021-05-22 19:10:28 +09:00
Wei-Chiu Chuang
fa4915fdbb
Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
Wei-Chiu Chuang
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959) 
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.

Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb

* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661). Contributed by Vinayakumar B.

(cherry picked from commit c4c0683dff)

Co-authored-by: Xiaoyu Yao <xyao@apache.org>
Co-authored-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-05-17 18:57:46 -07:00
Xiaoyu Yao
3f9c9ccf46
HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider.… (#3010) 
* HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider. (#2334)

(cherry picked from commit 4c5ad57818)
 2021-05-13 16:57:58 -07:00
dependabot[bot]
b2897fdd66
HADOOP-17683. Update commons-io to 2.8.0 (#2974) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 29105ffb63)
 2021-05-12 10:58:39 +09:00
Borislav Iordanov
f5ef78d46f HADOOP-17665 Ignore missing keystore configuration in reloading mechanism 2021-05-10 15:41:56 -05:00
Ayush Saxena
2a95ba29bb HADOOP-17644. Add back the exceptions removed by HADOOP-17432 for compatibility. Contributed by Quan Li. 2021-05-09 04:00:30 +05:30
Mike
0f12f3e125
HADOOP-17036. TestFTPFileSystem failing as ftp server dir already exists. 
Contributed by Mikhail Pryakhin.

(cherry picked from commit 017d24e970)
 2021-05-07 14:20:29 +09:00
hchaverr
cedebf1c27 HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 
(cherry picked from commit f40e3eb059)
 2021-05-06 17:50:32 -07:00
Viraj Jasani
be34c1222a HADOOP-11616. Remove workaround for Curator's ChildReaper requiring Guava 15+ (#2973) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b93e448f9a)
 2021-05-06 04:53:06 +09:00
Takanobu Asanuma
65bf544118 HADOOP-16954. Add -S option in "Count" command to show only Snapshot Counts. Contributed by hemanthboyina. 
(cherry picked from commit b89d875f7b)
 2021-05-04 17:44:34 +01:00
Istvan Fajth
00ba886a22 HADOOP-17675. LdapGroupsMapping$LdapSslSocketFactory ClassNotFoundException (#2965) 
(cherry picked from commit 0d78d73973)
 2021-05-04 11:36:15 +01:00
kishendas
98aa4fc32c
HADOOP-17657: implement StreamCapabilities in SequenceFile.Writer and fall back to flush, if hflush is not supported (#2949) 
Co-authored-by: Kishen Das <kishen@cloudera.com>
Reviewed-by: Steve Loughran <stevel@apache.org>
(cherry picked from commit e571025f5b)
 2021-05-04 16:35:00 +08:00
lfengnan
43fac739bb
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e525ab81c)
 2021-05-02 19:19:55 +09:00
litao
f45365f201
HDFS-15975. Use LongAdder instead of AtomicLong for branch-3.3 (#2940) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-27 22:39:25 +09:00
Steve Loughran
fb71e6c91e
HADOOP-17597. Optionally downgrade on S3A Syncable calls (#2801) 
Followup to HADOOP-13327, which changed S3A output stream hsync/hflush calls
to raise an exception.

Adds a new option fs.s3a.downgrade.syncable.exceptions

When true, calls to Syncable hsync/hflush on S3A output streams will
log once at warn (for entire process life, not just the stream), then
increment IOStats with the relevant operation counter

With the downgrade option false (default)
* IOStats are incremented
* The UnsupportedOperationException current raised includes a link to the
  JIRA.

Contributed by Steve Loughran.

Change-Id: I967e077eda1d1a1a3795b4d22e003fe7997b6679
 2021-04-24 18:32:39 +01:00
Mehakmeet Singh
389d3034c6
HADOOP-17471. ABFS to collect IOStatistics (#2731) (#2950) 
The ABFS Filesystem and its input and output streams now implement
the IOStatisticSource interface and provide IOStatistics on
their interactions with Azure Storage.

This includes the min/max/mean durations of all REST API calls.

Contributed by Mehakmeet Singh <mehakmeet.singh@cloudera.com>
 2021-04-24 17:59:26 +01:00
Eric Badger
930f384e30 MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
(cherry picked from commit da93cd962c)
 2021-04-23 18:42:50 +00:00
Eric Badger
f7e382c62e YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 
(cherry picked from commit 31477a224f)
 2021-04-16 20:21:01 +00:00
Masatake Iwasaki
a3405cb56a HADOOP-17569. Building native code fails on Fedora 33. (#2886) 
(cherry picked from commit 2717203f85)
 2021-04-15 12:39:07 +00:00
He Xiaoqiao
a3cff8776a
HADOOP-17613. Log not flushed fully when daemon shutdown. Contributed by Renukaprasad C. 2021-04-14 12:28:34 +08:00
Akira Ajisaka
72355c7b6e
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be)
 2021-04-13 17:10:00 +09:00
Akira Ajisaka
77315abe47
HADOOP-17608. Fix TestKMS failure (#2880) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2bd810a507)
 2021-04-12 12:54:59 +09:00
touchida
dca2bf9dd5 HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 
(cherry picked from commit 95e6892675)
 2021-04-08 17:20:08 +08:00
Brahma Reddy Battula
236a9a7713 HADOOP-17617. Incorrect representation of RESPONSE for Get Key Version in KMS index.md.vm file. Contributed by Ravuri Sushma sree 
(cherry picked from commit ae88174c29)
 2021-04-07 23:50:02 +05:30
Wei-Chiu Chuang
1110b03752 HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282)
 2021-04-07 13:17:40 +08:00
Viraj Jasani
8b4b3d6fe6 HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 3f2682b92b)
 2021-04-06 14:18:31 +09:00
Ayush Saxena
a549b4a82e HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq. 
(cherry picked from commit 936bf09c37)
 2021-04-02 17:11:16 +08:00
Brahma Reddy Battula
90bbaca88b HADOOP-17587. Kinit with keytab should not display the keytab file's full path in any logs. Contributed by Ravuri Sushma sree. 
(cherry picked from commit bc7689abf5)
 2021-04-02 10:04:19 +05:30
Brahma Reddy Battula
c60e81b5a8 HADOOP-17610. DelegationTokenAuthenticator prints token information. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 478402cc74)
 2021-04-02 09:56:47 +05:30
litao
62937d15bb HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d6)
 2021-04-02 10:57:05 +09:00
Borislav Iordanov
c365149e16 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures.

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:50:28 -07:00
He Xiaoqiao
ae2d856515
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:01 +08:00
Stephen O'Donnell
56ef16468a
HADOOP-17222. Create socket address leveraging URI cache (#2817) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-30 11:59:44 +01:00
Ayush Saxena
9c9b16c957
HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2808). Contributed by Ayush Saxena. 
* HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).

* HADOOP-17531.Addendum: DistCp: Reduce memory usage on copying huge directories. (#2820)

Signed-off-by: Steve Loughran <stevel@apache.org>
 2021-03-27 09:25:25 +05:30
Xiaoyu Yao
67d52af225 HADOOP-16828. Zookeeper Delegation Token Manager fetch sequence number by batch. Contributed by Fengnan Li. 
(cherry picked from commit 6288e15118)
 2021-03-25 14:44:02 +00:00
Takanobu Asanuma
9e1e89b1b9 HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel. 
(cherry picked from commit b4d97a8dc7)
 2021-03-24 17:53:19 +09:00
kwangsun
3aae563421
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. (#2780) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c8d327a4f1)
 2021-03-22 11:45:42 +09:00
Ayush Saxena
27944772d3 HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2021-03-19 00:13:31 +05:30
Jim Brennan
ad74038e02 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 
(cherry picked from commit 299b8062f1)
 2021-03-15 20:17:02 +00:00