big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
16,840 Commits 2 Branches 2 Tags 264 MiB
415e5a157a
Commit Graph

6426 Commits

Author SHA1 Message Date
Andrew Wang
40c2f31f8d HDFS-12218. Rename split EC / replicated block metrics in BlockManager. 2017-09-07 16:56:35 -07:00
Xiao Chen
52b894db33 HDFS-12369. Edit log corruption due to hard lease recovery of not-closed file which has snapshots. 2017-09-07 16:30:12 -07:00
Yongjun Zhang
d77ed238a9 HDFS-12357. Let NameNode to bypass external attribute provider for configured users. Contributed by Yongjun Zhang, Arun Suresh. 2017-09-07 09:50:36 -07:00
Arpit Agarwal
5ff74167dd HDFS-12376. Enable JournalNode Sync by default. Contributed by Hanisha Koneru. 2017-09-07 08:18:28 -07:00
Kai Zheng
2adf8bed71 HDFS-12402. Refactor ErasureCodingPolicyManager and related codes. Contributed by Sammi Chen 2017-09-07 20:38:23 +08:00
Chris Douglas
1f3bc63e67 HADOOP-12077. Provide a multi-URI replication Inode for ViewFs. Contributed by Gera Shegalov 2017-09-05 23:51:51 -07:00
Kai Zheng
d7f27043ce HDFS-12392. Writing striped file failed due to different cell size. Contributed by Sammi Chen 2017-09-06 13:29:52 +08:00
Andrew Wang
d4035d42f0 HDFS-12377. Refactor TestReadStripedFileWithDecoding to avoid test timeouts. 2017-09-05 16:33:29 -07:00
Andrew Wang
ccd2ac60ec HDFS-11882. Precisely calculate acked length of striped block groups in updatePipeline. 2017-09-05 14:16:03 -07:00
Xiao Chen
0ba8ff4b77 HDFS-12359. Re-encryption should operate with minimum KMS ACL requirements. 2017-09-05 10:08:27 -07:00
Xiao Chen
633c1ea455 HDFS-12383. Re-encryption updater should handle canceled tasks better. 2017-09-03 19:50:41 -07:00
Andrew Wang
0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Xiao Chen
1b3b9938cf HDFS-12300. Audit-log delegation token related operations. 2017-08-31 23:20:01 -07:00
Mingliang Liu
1fbb662c70 HDFS-12363. Possible NPE in BlockManager$StorageInfoDefragmenter#scanAndCompactStorages. Contributed by Xiao Chen 2017-08-31 22:36:56 -07:00
Arpit Agarwal
fd66a243bf HDFS-12356. Unit test for JournalNode sync during Rolling Upgrade. Contributed by Hanisha Koneru. 2017-08-30 10:29:42 -07:00
Rakesh Radhakrishnan
200b11368d HDFS-12258. ec -listPolicies should list all policies in system, no matter it's enabled or disabled. Contributed by Wei Zhou. 2017-08-30 12:58:56 +05:30
Xiao Chen
f3661fd08e HDFS-12336. Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory. Contributed by Wellington Chevreuil. 2017-08-29 21:30:02 -07:00
Yongjun Zhang
cf93d60d3f HDFS-12191. Provide option to not capture the accessTime change of a file to snapshot if no other modification has been done to this file. Contributed by Yongjun Zhang. 2017-08-29 16:57:03 -07:00
Manoj Govindassamy
63fc1b0b6d HDFS-11912. Snapshot functionality test along with randomized file IO operations. (George Huang via Manoj Govindassamy) 2017-08-29 11:27:47 -07:00
Arpit Agarwal
a1e3f84afe HDFS-12293. DataNode should log file name on disk error. Contributed by Ajay Kumar. 2017-08-28 15:49:59 -07:00
Arpit Agarwal
077a5eed9f HDFS-12358. Handle IOException when transferring edit log to Journal current dir through JN sync. Contributed by Hanisha Koneru. 2017-08-26 22:47:55 -07:00
Brahma Reddy Battula
bb6a3c8330 HDFS-12248. SNN will not upload fsimage on IOE and Interrupted exceptions. (Brahma Reddy Battula) 2017-08-26 13:47:12 +08:00
Manoj Govindassamy
b89ffcff36 HDFS-11986. Dfsadmin should report erasure coding related information separately. 2017-08-25 17:21:56 -07:00
Lei Xu
36bada3032 HDFS-12215. DataNode.transferBlock does not create its daemon in the xceiver thread group. (Lei Xu) 2017-08-25 16:01:14 -07:00
Andrew Wang
f29a0fc288 HDFS-12303. Change default EC cell size to 1MB for better performance. Contributed by Wei Zhou. 2017-08-25 14:14:23 -07:00
Kihwal Lee
8455d70756 HDFS-12299. Race Between update pipeline and DN Re-Registration 2017-08-25 14:49:29 -05:00
Arpit Agarwal
4b2c442d4e HDFS-12319. DirectoryScanner will throw IllegalStateException when Multiple BP's are present. Contributed by Brahma Reddy Battula. 2017-08-25 10:41:52 -07:00
John Zhuge
3123895db0 HDFS-12318. Fix IOException condition for openInfo in DFSInputStream. Contributed by legend. 2017-08-24 00:26:37 -07:00
Xiao Chen
1000a2af04 HDFS-10899. Add functionality to re-encrypt EDEKs. 2017-08-23 17:06:16 -07:00
John Zhuge
b6bfb2fcb2 HDFS-11738. Hedged pread takes more time when block moved from initial locations. Contributed by Vinayakumar B. 2017-08-21 13:45:30 -07:00
Manoj Govindassamy
913760cb4f HDFS-11988. Verify HDFS Snapshots with open files captured are consistent across truncates and appends to current version file. 2017-08-21 11:08:38 -07:00
Manoj Govindassamy
4230872dd6 HDFS-12316. Verify HDFS snapshot deletion doesn't crash the ongoing file writes. 2017-08-17 16:23:48 -07:00
Andrew Wang
b298948897 HDFS-12072. Provide fairness between EC and non-EC recovery tasks. Contributed by Eddy Xu. 2017-08-17 15:26:11 -07:00
Andrew Wang
dd7916d3cd HDFS-12250. Reduce usage of FsPermissionExtension in unit tests. Contributed by Chris Douglas. 2017-08-17 09:35:36 -07:00
Andrew Wang
96b3a6b972 HDFS-11082. Provide replicated EC policy to replicate files. Contributed by SammiChen. 2017-08-16 22:17:06 -07:00
Akira Ajisaka
08aaa4b36f
HDFS-12269. Better to return a Map rather than HashMap in getErasureCodingCodecs. Contributed by Huafeng Wang. 2017-08-17 13:20:27 +09:00
Ravi Prakash
f34646d652 HDFS-12301. NN File Browser UI: Navigate to a path when enter is pressed 2017-08-15 15:44:59 -07:00
Wei-Chiu Chuang
e3ae3e2644 HDFS-12066. When Namenode is in safemode,may not allowed to remove an user's erasure coding policy. Contributed by lufei. 2017-08-15 07:41:43 -07:00
Wei-Chiu Chuang
1040bae6fc HDFS-12054. FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode. Contributed by lufei. 2017-08-15 07:38:43 -07:00
Yiqun Lin
2e43c28e01 HDFS-11696. Fix warnings from Spotbugs in hadoop-hdfs. Contributed by Yiqun Lin. 2017-08-15 16:48:49 +08:00
Chris Douglas
645a8f2a4d HADOOP-14726. Mark FileStatus::isDir as final 2017-08-14 21:57:20 -07:00
Anu Engineer
d72124a442 HDFS-12162. Update listStatus document to describe the behavior when the argument is a file. Contributed by Ajay Kumar. 2017-08-14 11:32:49 -07:00
Lei Xu
ce797a1706 HDFS-12221. Replace xcerces in XmlEditsVisitor. (Ajay Kumar via lei) 2017-08-14 10:27:47 -07:00
John Zhuge
8b242f09a6 HDFS-11303. Hedged read might hang infinitely if read data from all DN failed . Contributed by Chen Zhang, Wei-chiu Chuang, and John Zhuge. 2017-08-11 19:42:07 -07:00
Yiqun Lin
f13ca94954 HDFS-12287. Remove a no-longer applicable TODO comment in DatanodeManager. Contributed by Chen Liang. 2017-08-11 14:13:45 +08:00
John Zhuge
312e57b954 HDFS-11957. Enable POSIX ACL inheritance by default. Contributed by John Zhuge. 2017-08-10 10:30:47 -07:00
Kihwal Lee
b5c02f95b5 HDFS-12278. LeaseManager operations are inefficient in 2.8. Contributed by Rushabh S Shah. 2017-08-09 16:46:05 -05:00
Kihwal Lee
69afa26f19 HDFS-12157. Do fsyncDirectory(..) outside of FSDataset lock. Contributed by inayakumar B. 2017-08-09 09:03:51 -05:00
Wei-Chiu Chuang
9a3c2379ef HDFS-12182. BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks. Contributed by Wellington Chevreuil. 2017-08-08 23:44:18 -07:00
Kai Zheng
a53b8b6fdc HDFS-11975. Provide a system-default EC policy. Contributed by Huichun Lu 2017-08-09 10:12:58 +08:00
Haohui Mai
1db4788b7d HADOOP-14598. Blacklist Http/HttpsFileSystem in FsUrlStreamHandlerFactory. Contributed by Steve Loughran. 2017-08-08 16:33:18 -07:00
Haohui Mai
71b8dda4f6 HDFS-10326. Disable setting tcp socket send/receive buffers for write pipelines. Contributed by Daryn Sharp. 2017-08-08 14:58:16 -07:00
Kai Zheng
0b67436068 HDFS-12306. Add audit log for some erasure coding operations. Contributed by Huafeng Wang 2017-08-07 19:30:10 +08:00
Arpit Agarwal
bbc6d254c8 HDFS-12224. Add tests to TestJournalNodeSync for sync after JN downtime. Contributed by Hanisha Koneru. 2017-08-04 12:51:33 -07:00
Lei Xu
fe3341786a HDFS-12251. Add document for StreamCapabilities. (Lei (Eddy) Xu) 2017-08-04 11:21:58 -07:00
Andrew Wang
f4c6b00a9f HDFS-12131. Add some of the FSNamesystem JMX values as metrics. Contributed by Erik Krogen. 2017-08-03 15:44:51 -07:00
Manoj Govindassamy
79df1e750e HDFS-9388. Decommission related code to support Maintenance State for datanodes. 2017-08-02 14:22:46 -07:00
Chris Douglas
12e44e7bda HDFS-6984. Serialize FileStatus via protobuf. 2017-08-02 12:12:48 -07:00
Sean Mackrory
1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Haohui Mai
6814324c33 HDFS-12107. FsDatasetImpl#removeVolumes floods the logs when removing the volume. Contributed by Kelvin Chu. 2017-08-01 18:34:07 -07:00
Manoj Govindassamy
52d7bafcf4 HDFS-12217. HDFS snapshots doesn't capture all open files when one of the open files is deleted. 2017-08-01 16:28:20 -07:00
Sean Mackrory
7774759830 HDFS-12151. Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes. 2017-08-01 14:34:25 -06:00
Akira Ajisaka
b38a1eea8e
HADOOP-14245. Use Mockito.when instead of Mockito.stub. Contributed by Andras Bokor. 2017-08-01 15:15:43 +09:00
Weiwei Yang
ea568123fa HDFS-12154. Incorrect javadoc description in StorageLocationChecker#check. Contributed by Nandakumar. 2017-07-31 17:02:44 -07:00
Arpit Agarwal
3e23415a92 HDFS-12082. BlockInvalidateLimit value is incorrectly set after namenode heartbeat interval reconfigured. Contributed by Weiwei Yang. 2017-07-31 11:33:55 -07:00
Andrew Wang
480c8db40c HDFS-12206. Rename the split EC / replicated block metrics. 2017-07-28 11:24:44 -07:00
Lei Xu
77791e4c36 HDFS-12044. Mismatch between BlockManager.maxReplicationStreams and ErasureCodingWorker.stripedReconstructionPool pool size causes slow and bursty recovery. (Contributed by Lei (Eddy) Xu) 2017-07-28 10:50:49 -07:00
Wei-Chiu Chuang
369f731264 HDFS-12062. removeErasureCodingPolicy needs super user permission. Contributed by Wei-Chiu Chuang. 2017-07-28 00:51:03 -07:00
Yongjun Zhang
c6330f22a5 HDFS-12190. Enable 'hdfs dfs -stat' to display access time. Contributed by Yongjun Zhang. 2017-07-27 16:48:24 -07:00
Jitendra Pandey
e3c7300225 HDFS-2319. Add test cases for FSshell -stat. Contributed by XieXianshan and Bharat Viswanadham. 2017-07-27 13:23:15 -07:00
Brahma Reddy Battula
c4a85c694f HDFS-11896. Non-dfsUsed will be doubled on dead node re-registration. Contributed by Brahma Reddy Battula. 2017-07-27 12:02:57 -07:00
Kihwal Lee
a68b5b31cf HDFS-12171. Reduce IIP object allocations for inode lookup. Contributed by Daryn Sharp. 2017-07-25 11:03:49 -05:00
Brahma Reddy Battula
6d983cca52 HADOOP-14455. ViewFileSystem#rename should support be supported within same nameservice with different mountpoints. Contributed by Brahma Reddy Battula. 2017-07-25 23:51:53 +08:00
Kihwal Lee
1a79dcfc45 HDFS-12143. Improve performance of getting and removing inode features. Contributed by Daryn Sharp. 2017-07-25 10:28:57 -05:00
Akira Ajisaka
770cc46228
HDFS-12176. dfsadmin shows DFS Used%: NaN% if the cluster has zero block. Contributed by Weiwei Yang. 2017-07-24 16:24:23 +09:00
Kihwal Lee
8e3a992ecc HDFS-11742. Improve balancer usability after HDFS-8818. Contributed by Kihwal Lee 2017-07-21 09:14:19 -05:00
Konstantin V Shvachko
8c2c812832 HDFS-11472. Fix inconsistent replica size after a data pipeline failure. Contributed by Erik Krogen and Wei-Chiu Chuang. 2017-07-20 14:47:25 -07:00
Anu Engineer
413b23eb04 HDFS-12158. Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection. Contributed by Mukul Kumar Singh. 2017-07-19 10:29:06 -07:00
Brahma Reddy Battula
04ff412dab HDFS-12133. Correct ContentSummaryComputationContext Logger class name.. Contributed by Surendra Singh Lilhore. 2017-07-19 23:43:10 +08:00
Brahma Reddy Battula
f8cd55fe33 HDFS-12067. Correct dfsadmin commands usage message to reflects IPC port. Contributed by steven-wugang. 2017-07-19 23:21:43 +08:00
Tsz-Wo Nicholas Sze
ed27f2b2cc HDFS-12138. Remove redundant 'public' modifiers from BlockCollection. Contributed by Chen Liang 2017-07-17 13:54:16 -07:00
Brahma Reddy Battula
b778887af5 HDFS-12112. TestBlockManager#testBlockManagerMachinesArray sometimes fails with NPE. Contributed by Wei-Chiu Chuang. 2017-07-15 10:38:31 +08:00
Xiao Chen
0a6d5c0cf1 HADOOP-14521. KMS client needs retry logic. Contributed by Rushabh S Shah. 2017-07-14 22:22:53 -07:00
Tsz-Wo Nicholas Sze
f413ee33df HDFS-12130. Optimizing permission check for getContentSummary. Contributed by Chen Liang 2017-07-14 14:35:51 -07:00
Tsz-Wo Nicholas Sze
a1f12bb543 Revert "HDFS-12130. Optimizing permission check for getContentSummary." to fix commit message. 
This reverts commit a29fe100b3.
 2017-07-14 14:34:01 -07:00
Kihwal Lee
e7d187a1b6 HDFS-12140. Remove BPOfferService lock contention to get block pool id. Contributed by Daryn Sharp. 2017-07-14 16:07:17 -05:00
Kihwal Lee
8d86a93915 HDFS-12137. DN dataset lock should be fair. Contributed by Daryn Sharp. 2017-07-14 15:41:43 -05:00
Tsz-Wo Nicholas Sze
a29fe100b3 HDFS-12130. Optimizing permission check for getContentSummary. 2017-07-14 13:36:27 -07:00
Xiaoyu Yao
e15e2713e1 HDFS-11502. Datanode UI should display hostname based on JMX bean instead of window.location.hostname. Contributed by Jeffrey E Rodriguez and Yuanbo Liu. 2017-07-12 16:26:19 -07:00
Akira Ajisaka
9144fd9e9b
HADOOP-14629. Improve exception checking in FileContext related JUnit tests. Contributed by Andras Bokor. 2017-07-12 11:35:50 +09:00
Yiqun Lin
3a7f02b815 HDFS-12085. Reconfigure namenode heartbeat interval fails if the interval was set with time unit. Contributed by Weiwei Yang. 2017-07-11 20:40:11 +08:00
Andrew Wang
f10864a820 Add release notes, changes, jdiff for 3.0.0-alpha4 2017-07-07 11:01:59 -07:00
Akira Ajisaka
8fc5dcc2a1
Revert "HADOOP-14587. Use GenericTestUtils.setLogLevel when available in hadoop-common. Contributed by Wenxin He." 
This reverts commit 82cb2a6497.
 2017-07-08 02:53:18 +09:00
Akira Ajisaka
82cb2a6497
HADOOP-14587. Use GenericTestUtils.setLogLevel when available in hadoop-common. Contributed by Wenxin He. 2017-07-07 15:00:47 +09:00
Akira Ajisaka
186650d21d
HDFS-12078. Add time unit to the description of property dfs.namenode.stale.datanode.interval in hdfs-default.xml. Contributed by Weiwei Yang. 2017-07-04 14:51:52 +09:00
Akira Ajisaka
b0560e0624
HDFS-12079. Description of dfs.block.invalidate.limit is incorrect in hdfs-default.xml. Contributed by Weiwei Yang. 2017-07-04 14:02:14 +09:00
Wei-Chiu Chuang
bcba844d11 HDFS-12042. Lazy initialize AbstractINodeDiffList#diffs for snapshots to reduce memory consumption. Contributed by Misha Dmitriev. 2017-06-30 10:28:01 -07:00
Arpit Agarwal
6a9dc5f44b HDFS-12043. Add counters for block re-replication. Contributed by Chen Liang. 2017-06-30 10:20:12 -07:00
Arpit Agarwal
a2f0cbd92f Revert "HDFS-12043. Add counters for block re-replication. Contributed by Chen Liang." 
Accidentally committed the wrong patch version, reverting to fix that.

This reverts commit 900221f95e.
 2017-06-30 10:19:27 -07:00
Andrew Wang
af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00