big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,027 Commits 2 Branches 2 Tags 264 MiB
b7cb8fe07c
Commit Graph

23027 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sahil Takiar
13b427fc05 HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable (#963) Contributed by Sahil Takiar. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
 2019-09-27 12:25:03 -07:00
Ayush Saxena
ce58c05f1d HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 2019-09-27 18:58:19 +05:30
Doroszlai, Attila
a93a139b5d
HDDS-2185. createmrenv failure not reflected in acceptance test result 
Closes #1533
 2019-09-27 12:47:25 +02:00
Sammi Chen
8a9ede5cff HADOOP-15616. Incorporate Tencent Cloud COS File System Implementation. Contributed by Yang Yu. 2019-09-27 16:53:13 +08:00
Zhankun Tang
2870668cfa Make upstream aware of 3.1.3 release. 2019-09-27 16:44:07 +08:00
Doroszlai, Attila
9bf7a6e5b2 HDDS-2149. Replace findbugs with spotbugs 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-26 22:56:05 -07:00
Doroszlai, Attila
0371e953ac HDDS-2179. ConfigFileGenerator fails with Java 10 or newer 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-26 21:59:36 -07:00
Bharat Viswanadham
b6ef8cc120 HDD-2193. Adding container related metrics in SCM. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-26 21:42:20 -07:00
dchitlangia
c55ac6a1c7 HDDS-2174. Delete GDPR Encryption Key from metadata when a Key is deleted 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-26 21:10:34 -07:00
Chen Liang
2adcc3c932 HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang. 2019-09-26 15:48:19 -07:00
Inigo Goiri
b1e55cfb55 HDFS-14461. RBF: Fix intermittently failing kerberos related unit test. Contributed by Xiaoqiao He. 2019-09-26 15:26:23 -07:00
Anu Engineer
06998a1126
HDDS-2180. Add Object ID and update ID on VolumeList Object. (#1526) 2019-09-26 09:41:03 -07:00
Eric Badger
18a8c2404e YARN-9857. TestDelegationTokenRenewer throws NPE but tests pass. Contributed by Ahmed Hussein 2019-09-26 16:15:08 +00:00
Ayush Saxena
1a2a352ecd HDFS-11934. Add assertion to TestDefaultNameNodePort#testGetAddressFromConf. Contributed by Nikhil Navadiya. 2019-09-26 20:45:37 +05:30
Ayush Saxena
a3f68933d7 HDFS-14873. Fix dfsadmin doc for triggerBlockReport. Contributed by Fei Hui. 2019-09-26 19:54:07 +05:30
Vivek Ratnavel Subramanian
7b6219a8d1
HDDS-2182. Fix checkstyle violations introduced by HDDS-1738 
Closes #1529
 2019-09-26 12:09:28 +02:00
Gabor Bota
587a8eeec8 HDFS-14874. Fix TestHDFSCLI and TestDFSShell test break because of logging change in mkdir (#1522). Contributed by Gabor Bota. 2019-09-26 10:15:13 +05:30
Jonathan Hung
606e341c1a Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-25 17:54:53 -07:00
Márton Elek
f647185905 HDDS-2067. Create generic service facade with tracing/metrics/logging support 
Signed-off-by: Anu Engineer <aengineer@apache.org>
Co-Authored-By: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
 2019-09-25 17:19:22 -07:00
Takanobu Asanuma
bdaaa3bbf2 HDFS-14832. RBF: Add Icon for ReadOnly False. Contributed by hemanthboyina 2019-09-25 23:48:49 +09:00
Xieming Li
c89d22d13a
HADOOP-16602. mvn package fails in hadoop-aws. 
Contributed by Xieming Li.

Follow-up to HADOOP-16445

Change-Id: I72c62d55b734a0f67556844f398ef4a50d9ea585
 2019-09-25 14:15:35 +01:00
Doroszlai, Attila
16f626f7f0
HDDS-2165. Freon fails if bucket does not exists 
Closes #1503
 2019-09-25 13:31:35 +02:00
Steve Loughran
e346e3638c HADOOP-15691 Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.

Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5
 2019-09-25 12:16:41 +01:00
Yi Sheng
e6fb6ee94d
HDDS-1738. Add nullable annotation for OMResponse classes 
Closes #1499
 2019-09-25 12:14:07 +02:00
Doroszlai, Attila
8baebb54e1
HDDS-2171. Dangling links in test report due to incompatible realpath 
Closes #1515
 2019-09-25 12:02:42 +02:00
Szilard Nemeth
c72457787d YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko 2019-09-25 10:36:22 +02:00
Szilard Nemeth
bec0864394 YARN-9808. Zero length files in container log output haven't got a header. Contributed by Adam Antal 2019-09-25 10:28:34 +02:00
Prabhu Joseph
3f89084ac7
HDFS-14845. Ignore AuthenticationFilterInitializer for HttpFSServerWebServer and honor hadoop.http.authentication configs. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-09-25 10:19:30 +09:00
Vivek Ratnavel Subramanian
a3463812ef HDDS-2168. TestOzoneManagerDoubleBufferWithOMResponse sometimes fails with out of memory error (#1509) 2019-09-24 16:28:25 -07:00
Bharat Viswanadham
6917754ba7
HDDS-2172.Ozone shell should remove description about REST protocol support. Contributed by Siddharth Wagle. 2019-09-24 16:06:10 -07:00
Jonathan Hung
c2731d4b63 YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-24 13:51:32 -07:00
Ayush Saxena
66400c1cbb HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 2019-09-25 01:31:15 +05:30
Ayush Saxena
eb96a3093e HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-09-25 01:16:30 +05:30
Anu Engineer
f16cf877e5
HDDS-2170. Add Object IDs and Update ID to Volume Object (#1510) 2019-09-24 11:01:05 -07:00
Inigo Goiri
afa1006a53 HDFS-14843. Double Synchronization in BlockReportLeaseManager. Contributed by David Mollitor. 2019-09-24 09:58:42 -07:00
Inigo Goiri
816d3cb908 HDFS-14837. Review of Block.java. Contributed by David Mollitor. 2019-09-24 09:54:09 -07:00
Wei-Chiu Chuang
43203b466d HDFS-14868. RBF: Fix typo in TestRouterQuota. Contributed by Jinglun. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-24 09:38:58 -07:00
Márton Elek
91f50b98ca
HDDS-2167. Hadoop31-mr acceptance test is failing due to the shading 
Closes #1507
 2019-09-24 17:52:36 +02:00
Mukund Thakur
51c64b357d
HDFS-13660. DistCp job fails when new data is appended in the file while the DistCp copy job is running 
This uses the length of the file known at the start of the copy to determine the amount of data to copy.

* If a file is appended to during the copy, the original bytes are copied.
* If a file is truncated during a copy, or the attempt to read the data fails with a truncated stream,
  distcp will now fail. Until now these failures were not detected.

Contributed by Mukund Thakur.

Change-Id: I576a49d951fa48d37a45a7e4c82c47488aa8e884
 2019-09-24 11:23:24 +01:00
avijayanhwx
8f1a135118 HDDS-2081. Fix TestRatisPipelineProvider#testCreatePipelinesDnExclude. Contributed by Aravindan Vijayan. (#1506) 2019-09-24 12:59:51 +05:30
Duo Zhang
e8e7d7b4cd
HADOOP-16561. [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes (#1500) 
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2019-09-24 08:52:12 +08:00
Duo Zhang
bfe1dac29d
HADOOP-16560. [YARN] use protobuf-maven-plugin to generate protobuf classes (#1496) 
Signed-off-by:  Vinayakumar B <vinayakumarb@apache.org>
 2019-09-24 08:50:36 +08:00
Hanisha Koneru
0a716bd3a5 HDDS-2159. Fix Race condition in ProfileServlet#pid. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-23 17:21:19 -07:00
Xiaoyu Yao
6cbe5d3809 HDDS-2160. Add acceptance test for ozonesecure-mr compose. Contributed by Xiaoyu Yao. (#1490) 2019-09-23 13:28:38 -07:00
dchitlangia
3fd3d746fc HDDS-2161. Create RepeatedKeyInfo structure to be saved in deletedTable 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-23 13:15:47 -07:00
Jonathan Hung
3d78b1223d YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 2019-09-23 13:12:16 -07:00
Ayush Saxena
c30e495557 HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar. 2019-09-23 21:22:50 +05:30
Siddharth Seth
2b5fc95851
HADOOP-16591 Fix S3A ITest*MRjob failures. 
Contributed by Siddharth Seth.

Change-Id: I7f08201c9f7c0551514049389b5b398a84855191
 2019-09-23 14:58:03 +01:00
Gabor Bota
aa664d7259
HADOOP-16138. hadoop fs mkdir / of nonexistent abfs container raises NPE (#1302). Contributed by Gabor Bota. 
Change-Id: I2f637865c871e400b95fe7ddaa24bf99fa192023
 2019-09-23 13:29:01 +02:00
Vinayakumar B
07c81e9bfc
HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes (#1494). Contributed by Vinayakumar B. 2019-09-23 12:37:47 +05:30