big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,998 Commits 2 Branches 2 Tags 264 MiB
1921e94292
Commit Graph

22998 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Steve Loughran
1921e94292
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3 
Contributed by Steve Loughran.

Includes
-S3A glob scans don't bother trying to resolve symlinks
-stack traces don't get lost in getFileStatuses() when exceptions are wrapped
-debug level logging of what is up in Globber
-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.
-ITestRestrictedReadAccess tests incomplete read access to files.

This adds a builder API for constructing globbers which other stores can use
so that they too can skip symlink resolution when not needed.

Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7
 2019-10-01 18:11:05 +01:00
Márton Elek
918b470deb
HDDS-2166. Some RPC metrics are missing from SCM prometheus endpoint 
Closes #1505
 2019-10-01 18:01:51 +02:00
Wei-Chiu Chuang
6ef6594c7e HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
 2019-10-01 21:16:46 +05:30
Sunil G
99cd7572f1 YARN-9801. SchedConfCli does not work wiwith https mode. Contributed by Prabhu Joseph 2019-10-01 20:06:21 +05:30
bibinchundatt
425a6c89ae YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:02:26 +05:30
Mukul Kumar Singh
8efd25b33a
HDDS-1615. ManagedChannel references are being leaked in ReplicationSupervisor.java. Contributed by Mukul Kumar Singh. (#1547) 2019-10-01 14:31:15 +05:30
Sunil G
137546a78a YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph 2019-10-01 09:08:41 +05:30
Konstantin V Shvachko
b3275ab1f2 HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 16:48:10 -07:00
Doroszlai, Attila
e5bba592a8 HDDS-2205. checkstyle.sh reports wrong failure count 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-30 16:32:38 -07:00
Abhishek Modi
4d3c580b03 YARN-9859. Refactoring of OpportunisticContainerAllocator. Contributed by Abhishek Modi. 2019-09-30 23:40:15 +05:30
bshashikant
98ca07ebed HDDS-2207. Update Ratis to latest snapshot. Contributed by Shashikant Bannerjee. (#1550) 2019-09-30 20:36:27 +05:30
dchitlangia
b46d82339f
HDDS-2202. Remove unused import in OmUtils 
Closes #1543
 2019-09-30 16:02:30 +02:00
Shashikant Banerjee
a530ac3f50
HDDS-2153. Add a config to tune max pending requests in Ratis leader 
Closes #1474
 2019-09-30 15:42:04 +02:00
cxorm
d6b0a8da77
HDDS-2183. Container and pipline subcommands of scmcli should be grouped 
Closes #1532
 2019-09-30 14:38:01 +02:00
Konstantin V Shvachko
760b523e58 Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao." 
This reverts commit 0feba4396f.
 2019-09-29 13:11:31 -07:00
Sneha Vijayarajan
c0edc848a8
HADOOP-16548 : Disable Flush() over config 2019-09-28 20:39:42 -07:00
Inigo Goiri
d8313b2274 HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun. 2019-09-28 17:20:44 -07:00
Ayush Saxena
2200871607 HDFS-14876. Remove unused imports from TestBlockMissingException.java and TestClose.java. Contributed by Lisheng Sun. 2019-09-28 22:03:53 +05:30
Ayush Saxena
c4c8d5fd0e HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 2019-09-28 21:44:44 +05:30
Ayush Saxena
0d5d0b914a Revert "HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao." 
This reverts commit ce58c05f1d.
 2019-09-28 21:32:15 +05:30
Bharat Viswanadham
14b4fbc019
HDDS-1146. Adding container related metrics in SCM. (#1541) 2019-09-27 15:24:30 -07:00
Sahil Takiar
13b427fc05 HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable (#963) Contributed by Sahil Takiar. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
 2019-09-27 12:25:03 -07:00
Ayush Saxena
ce58c05f1d HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 2019-09-27 18:58:19 +05:30
Doroszlai, Attila
a93a139b5d
HDDS-2185. createmrenv failure not reflected in acceptance test result 
Closes #1533
 2019-09-27 12:47:25 +02:00
Sammi Chen
8a9ede5cff HADOOP-15616. Incorporate Tencent Cloud COS File System Implementation. Contributed by Yang Yu. 2019-09-27 16:53:13 +08:00
Zhankun Tang
2870668cfa Make upstream aware of 3.1.3 release. 2019-09-27 16:44:07 +08:00
Doroszlai, Attila
9bf7a6e5b2 HDDS-2149. Replace findbugs with spotbugs 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-26 22:56:05 -07:00
Doroszlai, Attila
0371e953ac HDDS-2179. ConfigFileGenerator fails with Java 10 or newer 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-26 21:59:36 -07:00
Bharat Viswanadham
b6ef8cc120 HDD-2193. Adding container related metrics in SCM. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-26 21:42:20 -07:00
dchitlangia
c55ac6a1c7 HDDS-2174. Delete GDPR Encryption Key from metadata when a Key is deleted 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-26 21:10:34 -07:00
Chen Liang
2adcc3c932 HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang. 2019-09-26 15:48:19 -07:00
Inigo Goiri
b1e55cfb55 HDFS-14461. RBF: Fix intermittently failing kerberos related unit test. Contributed by Xiaoqiao He. 2019-09-26 15:26:23 -07:00
Anu Engineer
06998a1126
HDDS-2180. Add Object ID and update ID on VolumeList Object. (#1526) 2019-09-26 09:41:03 -07:00
Eric Badger
18a8c2404e YARN-9857. TestDelegationTokenRenewer throws NPE but tests pass. Contributed by Ahmed Hussein 2019-09-26 16:15:08 +00:00
Ayush Saxena
1a2a352ecd HDFS-11934. Add assertion to TestDefaultNameNodePort#testGetAddressFromConf. Contributed by Nikhil Navadiya. 2019-09-26 20:45:37 +05:30
Ayush Saxena
a3f68933d7 HDFS-14873. Fix dfsadmin doc for triggerBlockReport. Contributed by Fei Hui. 2019-09-26 19:54:07 +05:30
Vivek Ratnavel Subramanian
7b6219a8d1
HDDS-2182. Fix checkstyle violations introduced by HDDS-1738 
Closes #1529
 2019-09-26 12:09:28 +02:00
Gabor Bota
587a8eeec8 HDFS-14874. Fix TestHDFSCLI and TestDFSShell test break because of logging change in mkdir (#1522). Contributed by Gabor Bota. 2019-09-26 10:15:13 +05:30
Jonathan Hung
606e341c1a Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-25 17:54:53 -07:00
Márton Elek
f647185905 HDDS-2067. Create generic service facade with tracing/metrics/logging support 
Signed-off-by: Anu Engineer <aengineer@apache.org>
Co-Authored-By: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
 2019-09-25 17:19:22 -07:00
Takanobu Asanuma
bdaaa3bbf2 HDFS-14832. RBF: Add Icon for ReadOnly False. Contributed by hemanthboyina 2019-09-25 23:48:49 +09:00
Xieming Li
c89d22d13a
HADOOP-16602. mvn package fails in hadoop-aws. 
Contributed by Xieming Li.

Follow-up to HADOOP-16445

Change-Id: I72c62d55b734a0f67556844f398ef4a50d9ea585
 2019-09-25 14:15:35 +01:00
Doroszlai, Attila
16f626f7f0
HDDS-2165. Freon fails if bucket does not exists 
Closes #1503
 2019-09-25 13:31:35 +02:00
Steve Loughran
e346e3638c HADOOP-15691 Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.

Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5
 2019-09-25 12:16:41 +01:00
Yi Sheng
e6fb6ee94d
HDDS-1738. Add nullable annotation for OMResponse classes 
Closes #1499
 2019-09-25 12:14:07 +02:00
Doroszlai, Attila
8baebb54e1
HDDS-2171. Dangling links in test report due to incompatible realpath 
Closes #1515
 2019-09-25 12:02:42 +02:00
Szilard Nemeth
c72457787d YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko 2019-09-25 10:36:22 +02:00
Szilard Nemeth
bec0864394 YARN-9808. Zero length files in container log output haven't got a header. Contributed by Adam Antal 2019-09-25 10:28:34 +02:00
Prabhu Joseph
3f89084ac7
HDFS-14845. Ignore AuthenticationFilterInitializer for HttpFSServerWebServer and honor hadoop.http.authentication configs. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-09-25 10:19:30 +09:00
Vivek Ratnavel Subramanian
a3463812ef HDDS-2168. TestOzoneManagerDoubleBufferWithOMResponse sometimes fails with out of memory error (#1509) 2019-09-24 16:28:25 -07:00