big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,006 Commits 2 Branches 2 Tags 264 MiB
2cf952baf4
Commit Graph

5599 Commits

Author SHA1 Message Date
Mingliang Liu
6e5692e7e2
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042)" 
This reverts commit 4a26a61ecd.
 2021-06-11 00:34:24 -07:00
Viraj Jasani
4ef27a596f
HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
Steve Loughran
762a83e044
HADOOP-17631. Configuration ${env.VAR:-FALLBACK} to eval FALLBACK when restrictSystemProps=true (#2977) 
Contributed by Steve Loughran.
 2021-06-08 21:56:40 +01:00
Viraj Jasani
f4b24c68e7
HADOOP-17743. Replace Guava Lists usage by Hadoop's own Lists in hadoop-common, hadoop-tools and cloud-storage projects (#3072) 2021-06-07 13:24:09 +09:00
July
4a26a61ecd
HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042) 
Contributed by kaifeiYi (yikf).

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2021-06-04 13:36:09 -07:00
Viraj Jasani
59fc4061cb
HADOOP-17152. Provide Hadoop's own Lists utility to reduce dependency on Guava (#3061) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-06-03 18:56:00 +09:00
Konstantin V Shvachko
8ce30f51f9 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 2021-05-27 18:47:20 -07:00
Hui Fei
98a74e2351
HDFS-16041. TestErasureCodingCLI fails (#3052) 2021-05-26 20:12:17 +09:00
Viraj Jasani
986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-26 17:14:31 +09:00
Steve Loughran
832a3c6a89
HADOOP-17511. Add audit/telemetry logging to S3A connector (#2807) 
The S3A connector supports
"an auditor", a plugin which is invoked
at the start of every filesystem API call,
and whose issued "audit span" provides a context
for all REST operations against the S3 object store.

The standard auditor sets the HTTP Referrer header
on the requests with information about the API call,
such as process ID, operation name, path,
and even job ID.

If the S3 bucket is configured to log requests, this
information will be preserved there and so can be used
to analyze and troubleshoot storage IO.

Contributed by Steve Loughran.
 2021-05-25 10:25:41 +01:00
Mehakmeet Singh
c665ab02ed
HADOOP-17670. S3AFS and ABFS to log IOStats at DEBUG mode or optionally at INFO level in close() (#2963) 
When the S3A and ABFS filesystems are closed,
their IOStatistics are logged at debug in the log:

org.apache.hadoop.fs.statistics.IOStatisticsLogging

Set `fs.iostatistics.logging.level` to `info` for the statistics 
to be logged at info. (also: `warn` or `error` for even higher
log levels).


Contributed by: Mehakmeet Singh
 2021-05-24 13:02:11 +01:00
Vinayakumar B
2bbeae3240
HDFS-15790. Make ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist (#2767) 2021-05-24 02:45:39 -07:00
Viraj Jasani
c70ee2d548
HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-05-22 19:06:58 +09:00
Viraj Jasani
e4062ad027
HADOOP-17115. Replace Guava Sets usage by Hadoop's own Sets in hadoop-common and hadoop-tools (#2985) 
Signed-off-by: Sean Busbey <busbey@apache.org>
 2021-05-20 10:47:04 -05:00
Hongbing Wang
f7247922b7
HDFS-16018. Optimize the display of hdfs "count -e" or "count -t" com… (#2994) 2021-05-20 11:23:54 +08:00
Xiaoyu Yao
86729e130f
HADOOP-17699. Remove hardcoded SunX509 usage from SSLFactory. (#3016) 2021-05-18 10:11:36 -07:00
Akira Ajisaka
35ca1dcb9d
HADOOP-17685. Fix junit deprecation warnings in hadoop-common module. (#2983) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-13 14:22:25 +09:00
Viraj Jasani
fdd20a3cf4
HADOOP-17689. Avoid Potential NPE in org.apache.hadoop.fs (#3008) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-13 00:05:58 +09:00
dependabot[bot]
29105ffb63
HADOOP-17683. Update commons-io to 2.8.0 (#2974) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-05-12 10:57:45 +09:00
Viraj Jasani
c80f07422f
HADOOP-17686. Avoid potential NPE by using Path#getParentPath API in hadoop-huaweicloud (#2990) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-12 10:34:01 +09:00
Borislav Iordanov
91430889a5
HADOOP-17665 Ignore missing keystore configuration in reloading mechanism 2021-05-10 13:31:48 -07:00
hexiaoqiao
6d6766bc22
HADOOP-17690. Improve the log for The DecayRpcScheduler. Contributed by Bhavik Patel. 2021-05-10 14:11:36 +08:00
Ayush Saxena
7f93349ee7 HADOOP-17644. Add back the exceptions removed by HADOOP-17432 for compatibility. Contributed by Quan Li. 2021-05-09 03:40:08 +05:30
hchaverr
f40e3eb059 HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 2021-05-06 16:42:27 -07:00
Viraj Jasani
b93e448f9a
HADOOP-11616. Remove workaround for Curator's ChildReaper requiring Guava 15+ (#2973) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-06 04:52:02 +09:00
Istvan Fajth
0d78d73973
HADOOP-17675. LdapGroupsMapping$LdapSslSocketFactory ClassNotFoundException (#2965) 2021-05-04 11:33:59 +01:00
kishendas
e571025f5b
HADOOP-17657: implement StreamCapabilities in SequenceFile.Writer and fall back to flush, if hflush is not supported (#2949) 
Co-authored-by: Kishen Das <kishen@cloudera.com>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2021-05-04 01:20:56 -07:00
lfengnan
6e525ab81c
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 19:18:47 +09:00
Wei-Chiu Chuang
b2e54762a4
HDFS-15624. fix the function of setting quota by storage type (#2377) (#2955) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>

Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44
 2021-04-28 23:54:39 -07:00
Wei-Chiu Chuang
f41a368c14
HADOOP-11245. Update NFS gateway to use Netty4 (#2832) 
Reviewed-by: Tsz-Wo Nicholas Sze <szetszwo@apache.org>
 2021-04-28 23:43:11 -07:00
Wei-Chiu Chuang
90c6caf650 Revert "HDFS-15624. fix the function of setting quota by storage type (#2377)" 
This reverts commit 394b9f7a5c.

Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
 2021-04-26 11:27:15 +08:00
Steve Loughran
027c8fb257
HADOOP-17597. Optionally downgrade on S3A Syncable calls (#2801) 
Followup to HADOOP-13327, which changed S3A output stream hsync/hflush calls
to raise an exception.

Adds a new option fs.s3a.downgrade.syncable.exceptions

When true, calls to Syncable hsync/hflush on S3A output streams will
log once at warn (for entire process life, not just the stream), then
increment IOStats with the relevant operation counter

With the downgrade option false (default)
* IOStats are incremented
* The UnsupportedOperationException current raised includes a link to the
  JIRA.

Contributed by Steve Loughran.
 2021-04-23 18:44:41 +01:00
Mehakmeet Singh
6085f09db5
HADOOP-17471. ABFS to collect IOStatistics (#2731) 
The ABFS Filesystem and its input and output streams now implement
the IOStatisticSource interface and provide IOStatistics on
their interactions with Azure Storage.

This includes the min/max/mean durations of all REST API calls.

Contributed by Mehakmeet Singh <mehakmeet.singh@cloudera.com>
 2021-04-23 10:28:31 +01:00
Vinayakumar B
c4c0683dff
HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661). Contributed by Vinayakumar B. 2021-04-22 01:02:03 +05:30
Viraj Jasani
9a261b0a16
HADOOP-17642. Remove appender EventCounter to avoid instantiation (#2922) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-17 20:47:44 +09:00
litao
af0448d37b
HDFS-15975. Use LongAdder instead of AtomicLong (#2907) 2021-04-17 16:49:09 +09:00
Masatake Iwasaki
2717203f85
HADOOP-17569. Building native code fails on Fedora 33. (#2886) 2021-04-15 21:33:18 +09:00
Viraj Jasani
9179638017
HADOOP-17524. Remove EventCounter and Log counters from JVM Metrics (#2909) 
Reviewed-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-15 18:04:46 +09:00
Akira Ajisaka
156ecc89be
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-13 17:08:49 +09:00
Akira Ajisaka
2bd810a507
HADOOP-17608. Fix TestKMS failure (#2880) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-04-12 12:53:58 +09:00
Brahma Reddy Battula
ae88174c29 HADOOP-17617. Incorrect representation of RESPONSE for Get Key Version in KMS index.md.vm file. Contributed by Ravuri Sushma sree 2021-04-07 23:49:17 +05:30
Wei-Chiu Chuang
dac60b8282
HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-06 21:55:01 -07:00
He Xiaoqiao
3a89471c36
HADOOP-17613. Log not flushed fully when daemon shutdown. Contributed by Renukaprasad C. 2021-04-07 12:18:23 +08:00
Boyina, Hemanth Kumar
e86050fae5 HADOOP-17588. CryptoInputStream#close() should be syncronized. Contributed by RenukaPrasad C 2021-04-06 18:06:10 +05:30
Viraj Jasani
3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 13:39:10 +09:00
zhuqi
5eaa1b7230
HADOOP-17619: Fix DelegationTokenRenewer#updateRenewalTime java doc error. (#2846). Contributed by Qi Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-04-04 14:52:03 +05:30
Brahma Reddy Battula
bc7689abf5 HADOOP-17587. Kinit with keytab should not display the keytab file's full path in any logs. Contributed by Ravuri Sushma sree. 2021-04-02 10:03:50 +05:30
Brahma Reddy Battula
478402cc74 HADOOP-17610. DelegationTokenAuthenticator prints token information. Contributed by Ravuri Sushma sree. 2021-04-02 09:56:00 +05:30
litao
4bd04126d6
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-02 10:33:40 +09:00
Akira Ajisaka
a94a23ab9e Revert "HADOOP-17608. Fix NPE in TestKMS (#2828)" 
This reverts commit d69088a097.
 2021-04-01 04:24:26 +09:00