big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
15,746 Commits 2 Branches 2 Tags 264 MiB
62e4573efb
Commit Graph

3577 Commits

Author SHA1 Message Date
Zhe Zhang
e806db7190 HDFS-10584. Allow long-running Mover tool to login with keytab. Contributed by Rakesh R. 2016-08-26 16:43:45 -07:00
Zhe Zhang
407b519fb1 HDFS-10798. Make the threshold of reporting FSNamesystem lock contention configurable. Contributed by Erik Krogen. 2016-08-26 14:19:55 -07:00
Kihwal Lee
8b7adf4ddf HDFS-10768. Optimize mkdir ops. Contributed by Daryn Sharp. 2016-08-26 15:39:56 -05:00
Kihwal Lee
13fb1b50e6 HDFS-8915. TestFSNamesystem.testFSLockGetWaiterCount fails intermittently. Contributed by Masatake Iwasaki. 2016-08-26 09:53:10 -05:00
Xiaoyu Yao
4da5000dd3 HDFS-10748. TestFileTruncate#testTruncateWithDataNodesRestart runs sometimes timeout. Contributed by Yiqun Lin. 2016-08-25 09:00:44 -07:00
Kai Zheng
793447f799 HDFS-8905. Refactor DFSInputStream#ReaderStrategy. Contributed by Kai Zheng and Sammi Chen 2016-08-24 21:57:23 +08:00
Daryn Sharp
ec252ce0fc HDFS-10762. Pass IIP for file status related methods 2016-08-24 08:46:47 -05:00
Jason Lowe
126d165efd HDFS-9745. TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts. Contributed by Xiao Chen 2016-08-23 14:46:58 +00:00
Wei-Chiu Chuang
f0efea490e HDFS-8986. Add option to -du to calculate directory space usage excluding snapshots. Contributed by Xiao Chen. 2016-08-23 04:14:24 -07:00
Akira Ajisaka
e90f3359de HDFS-10783. The option '-maxSize' and '-step' fail in OfflineImageViewer. Contributed by Yiqun Lin. 2016-08-23 19:57:23 +09:00
Eric Yang
c49333becf HDFS-8312. Added permission check for moving file to Trash. (Weiwei Yang via Eric Yang) 2016-08-22 18:29:56 -07:00
Kihwal Lee
3ca4d6ddfd Revert "HDFS-10762. Pass IIP for file status related methods. Contributed by Daryn Sharp." 
This reverts commit 22fc46d765.
 2016-08-22 16:57:45 -05:00
Kihwal Lee
22fc46d765 HDFS-10762. Pass IIP for file status related methods. Contributed by Daryn Sharp. 2016-08-22 15:37:02 -05:00
Akira Ajisaka
8179f9a493 HDFS-10645. Make block report size as a metric and add this metric to datanode web ui. Contributed by Yuanbo Liu. 2016-08-19 16:15:00 +09:00
Xiao Chen
2aa5e2c403 HDFS-10549. Correctly revoke file leases when closing files. Contributed by Yiqun Lin. 2016-08-17 15:52:38 -07:00
Kihwal Lee
869393643d HDFS-10745. Directly resolve paths into INodesInPath. Contributed by Daryn Sharp. 2016-08-17 15:53:03 -05:00
Mingliang Liu
4d4d95fdd5 HDFS-10641. TestBlockManager#testBlockReportQueueing fails intermittently. (Contributed by Daryn Sharp) 2016-08-15 20:28:40 -07:00
Mingliang Liu
382d615260 HDFS-10678. Documenting NNThroughputBenchmark tool. (Contributed by Mingliang Liu) 2016-08-15 20:22:14 -07:00
Anu Engineer
5628b36c08 HDFS-10559. DiskBalancer: Use SHA1 for Plan ID. Contributed by Xiaobing Zhou. 2016-08-15 20:10:21 -07:00
Kihwal Lee
864f878d59 HDFS-10763. Open files can leak permanently due to inconsistent lease update. Contributed by Kihwal Lee. 2016-08-15 17:28:09 -05:00
Kihwal Lee
03dea65e0b HDFS-10744. Internally optimize path component resolution. Contributed by Daryn Sharp. 2016-08-15 16:45:44 -05:00
Kihwal Lee
83e57e083f HDFS-9696. Garbage snapshot records linger forever. Contributed by Kihwal Lee 2016-08-15 13:01:23 -05:00
Anu Engineer
9f29f423e4 HDFS-10737. disk balancer add volume path to report command. Contributed by Yuanbo Liu. 2016-08-15 09:47:30 -07:00
Kai Zheng
b5af9be72c HDFS-8668. Erasure Coding: revisit buffer used for encoding and decoding. Contributed by Sammi Chen 2016-08-13 13:52:37 +08:00
Wei-Chiu Chuang
a428d4f50e HDFS-8897. Balancer should handle fs.defaultFS trailing slash in HA. Contributed by John Zhuge. 2016-08-10 22:17:35 -07:00
Kai Zheng
e7e8aed208 HDFS-10720. Fix intermittent test failure of TestDataNodeErasureCodingMetrics. Contributed by Rakesh R 2016-08-12 11:15:05 +08:00
Xiaoyu Yao
ec289bbece HDFS-10643. Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 2016-08-10 15:49:36 -07:00
Xiaoyu Yao
fdafd0cb4f Revert "Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao." 
This reverts commit 9f816d7ca7.
 2016-08-10 15:47:55 -07:00
Xiaoyu Yao
9f816d7ca7 Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 2016-08-10 15:14:38 -07:00
Wei-Chiu Chuang
d00d3add9e HDFS-8224. Schedule a block for scanning if its metadata file is corrupt. Contributed by Rushabh S Shah. 2016-08-09 18:11:34 -07:00
Lei Xu
9c6a4383ca HDFS-10681. DiskBalancer: query command should report Plan file path apart from PlanID. (Manoj Govindassamy via lei) 2016-08-09 15:59:22 -07:00
Lei Xu
cc48251bfd HDFS-10457. DataNode should not auto-format block pool directory if VERSION is missing. (Wei-Chiu Chuang via lei) 2016-08-09 15:51:19 -07:00
Kihwal Lee
0f701f433d HDFS-10738. Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test failure. Contributed by Rakesh R. 2016-08-09 14:54:11 -05:00
Kihwal Lee
b10c936020 HDFS-10342. BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt. Contributed by Kuhu Shukla. 2016-08-09 11:56:21 -05:00
Arpit Agarwal
8c0638471f HADOOP-10682. Replace FsDatasetImpl object lock with a separate lock object. (Chen Liang) 2016-08-08 12:02:53 -07:00
Konstantin V Shvachko
4d3af47f27 HDFS-10693. metaSave should print blocks, not LightWeightHashSet. Contributed by Yuanbo Liu. 2016-08-07 14:29:26 -07:00
Akira Ajisaka
ef432579a7 HDFS-10715. NPE when applying AvailableSpaceBlockPlacementPolicy. Contributed by Guangbin Zhu. 2016-08-05 17:31:11 +09:00
Kihwal Lee
438a9f047e HDFS-10673. Optimize FSPermissionChecker's internal path usage. Contributed by Daryn Sharp. 2016-08-04 16:14:13 -05:00
Kihwal Lee
57369a678c HDFS-10343. BlockManager#createLocatedBlocks may return blocks on failed storages. Contributed by Kuhu Shukla. 2016-08-04 14:55:21 -05:00
Kihwal Lee
4a8e6dc02f HDFS-10722. Fix race condition in TestEditLog#testBatchedSyncWithClosedLogs. Contributed by Daryn Sharp. 2016-08-04 11:53:44 -05:00
Kihwal Lee
6f63566694 HDFS-10569. A bug causes OutOfIndex error in BlockListAsLongs. Contributed by Weiwei Yang. 2016-08-03 15:20:20 -05:00
Kihwal Lee
22ef5286bc HDFS-10674. Optimize creating a full path from an inode. Contributed by Daryn Sharp. 2016-08-03 13:12:43 -05:00
Kihwal Lee
5e5b8793fb HDFS-5805. TestCheckpoint.testCheckpoint fails intermittently on branch-2. Contributed by Eric Badger. 2016-08-02 15:37:36 -05:00
Akira Ajisaka
a5fb298e56 HDFS-10707. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon. 2016-08-02 17:07:59 +09:00
Konstantin V Shvachko
c4463f2ef2 Revert "HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi." 
This reverts commit 85a20508bd.
 2016-08-01 22:34:40 -07:00
Daryn Sharp
9f473cf903 HDFS-10655. Fix path related byte array conversion bugs. (daryn) 2016-08-01 10:14:28 -05:00
Akira Ajisaka
204a2055b1 HDFS-10691. FileDistribution fails in hdfs oiv command due to ArrayIndexOutOfBoundsException. Contributed by Yiqun Lin. 2016-07-29 15:39:48 +09:00
Xiao Chen
d9aae22fdf HDFS-9276. Failed to Update HDFS Delegation Token for long running application in HA mode. Contributed by Liangliang Gu and John Zhuge 2016-07-28 16:29:40 -07:00
Xiaoyu Yao
ce3d68e9c3 HDFS-10676. Add namenode metric to measure time spent in generating EDEKs. Contributed by Hanisha Koneru. 2016-07-28 16:07:45 -07:00
Lei Xu
a3d0cba810 HDFS-10689. Hdfs dfs chmod should reset sticky bit permission when the bit is omitted in the octal mode. (Manoj Govindassamy via Lei Xu) 2016-07-28 13:52:14 -07:00
Lei Xu
67406460f0 HDFS-4176. EditLogTailer should call rollEdits with a timeout. (Lei Xu) 2016-07-28 13:32:00 -07:00
Xiao Chen
328c855a57 HDFS-10650. DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission. Contributed by John Zhuge. 2016-07-28 13:15:02 -07:00
Akira Ajisaka
414fbfab41 HDFS-10696. TestHDFSCLI fails. Contributed by Kai Sasaki. 2016-07-28 16:58:23 +09:00
Andrew Wang
098ec2b11f HDFS-10519. Add a configuration option to enable in-progress edit log tailing. Contributed by Jiayi Zhou. 2016-07-27 17:55:41 -07:00
Akira Ajisaka
54fe17a607 HADOOP-9427. Use JUnit assumptions to skip platform-specific tests. Contributed by Gergely Novák. 2016-07-27 19:41:09 +09:00
Brahma Reddy Battula
55d5993a8e HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 2016-07-27 10:26:42 +05:30
Wei-Chiu Chuang
d84ab8a578 HDFS-10598. DiskBalancer does not execute multi-steps plan. Contributed by Lei (Eddy) Xu. 2016-07-26 16:01:50 -07:00
Arpit Agarwal
d2cf8b54c5 HDFS-10642. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. (Contributed by Mingliang Liu) 2016-07-26 12:27:46 -07:00
Steve Loughran
2aa18e1925 Revert "HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu." 
This reverts commit 7cac7655fd.
 2016-07-26 14:30:02 +01:00
Brahma Reddy Battula
7cac7655fd HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 2016-07-26 10:54:24 +05:30
Vinitha Reddy Gankidi
85a20508bd HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi. 2016-07-25 18:50:59 -07:00
Kai Zheng
b79ba4fe96 HDFS-10651. Clean up some configuration related codes about legacy block reader. Contributed by Youwei Wang 2016-07-23 17:45:33 +08:00
Arpit Agarwal
c6e3a0020f HDFS-10660. Expose storage policy apis via HDFSAdmin interface. (Contributed by Rakesh R) 2016-07-22 12:33:50 -07:00
Xiao Chen
132deb4cac HDFS-10225. DataNode hot swap drives should disallow storage type changes. Contributed by Lei (Eddy) Xu. 2016-07-21 16:41:02 -07:00
Akira Ajisaka
fcde6940e0 HDFS-10287. MiniDFSCluster should implement AutoCloseable. Contributed by Andras Bokor. 2016-07-21 11:26:08 -07:00
Jing Zhao
bd3dcf46e2 HDFS-10653. Optimize conversion from path string to components. Contributed by Daryn Sharp. 2016-07-21 11:14:39 -07:00
Akira Ajisaka
38128baff4 HDFS-10425. Clean up NNStorage and TestSaveNamespace. Contributed by Andras Bokor. 2016-07-20 10:44:11 -07:00
Akira Ajisaka
fc570b55b9 HDFS-10603. Fix flaky tests in org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot. Contributed by Yiqun Lin. 2016-07-19 13:49:24 -07:00
Akira Ajisaka
5537c6b234 HADOOP-13351. TestDFSClientSocketSize buffer size tests are flaky. Contributed by Aaron Fabbri and Mingliang Liu. 2016-07-14 14:40:58 -07:00
Uma Maheswara Rao G
438b7c5935 HDFS-10590: Fix TestReconstructStripedBlocks.testCountLiveReplicas test failures. Contributed by Rakesh R 2016-07-13 00:07:11 -07:00
Zhe Zhang
087290e6b1 HDFS-10544. Balancer doesn't work with IPFailoverProxyProvider. 2016-07-12 23:18:37 -07:00
Jitendra Pandey
c447efebdb HDFS-10579. HDFS web interfaces lack configs for X-FRAME-OPTIONS protection. Contributed by Anu Engineer. 2016-07-11 14:55:33 -07:00
Yongjun Zhang
da6f1b88dd HDFS-10512. VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks. Contributed by Wei-Chiu Chuang and Yiqun Lin. 2016-07-08 19:40:44 -07:00
Akira Ajisaka
932aed64d7 HDFS-10336. TestBalancer failing intermittently because of not reseting UserGroupInformation completely. Contributed by Yiqun Lin. 2016-07-08 14:39:11 -07:00
Jitendra Pandey
6e597600f7 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 14:58:19 -07:00
Uma Maheswara Rao G
ab092c56c2 HDFS-10592:Fix intermittent test failure of TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning. Contributed by Rakesh R 2016-07-07 13:52:35 -07:00
Kai Zheng
8b281bce85 HDFS-10548. Remove the long deprecated BlockReaderRemote. Contributed by Kai Zheng 2016-07-03 11:56:23 +08:00
Colin Patrick Mccabe
e4a2545620 HDFS-9805. Add server-side configuration for enabling TCP_NODELAY for DataTransferProtocol and default it to true (Gary Helmling via cmccabe) 2016-06-29 12:41:42 -07:00
Andrew Wang
c0829f4493 HDFS-6434. Default permission for creating file should be 644 for WebHdfs/HttpFS. Contributed by Wellington Chevreuil. 2016-06-28 13:55:26 -07:00
Vinayakumar B
2a0082c51d HDFS-10440. Improve DataNode web UI (Contributed by Weiwei Yang) 2016-06-28 16:49:39 +05:30
Vinayakumar B
73615a789d HDFS-10536. Standby NN can not trigger log roll after EditLogTailer thread failed 3 times in EditLogTailer.triggerActiveLogRoll method (Contributed by XingFeng Shen 2016-06-26 15:45:32 +05:30
Zhe Zhang
22be604fa3 Revert "HDFS-10534. NameNode WebUI should display DataNode usage rate with a certain percentile. Contributed by Kai Sasaki." 
This reverts commit 0424056a77.
 2016-06-24 09:36:03 -07:00
Zhe Zhang
0424056a77 HDFS-10534. NameNode WebUI should display DataNode usage rate with a certain percentile. Contributed by Kai Sasaki. 2016-06-24 08:58:29 -07:00
Anu Engineer
b2584bee45 HDFS-10571. TestDiskBalancerCommand#testPlanNode failed with IllegalArgumentException. Contributed by Xiaobing Zhou. 2016-06-23 18:27:02 -07:00
Anu Engineer
8a6e354122 HDFS-10552. DiskBalancer "-query" results in NPE if no plan for the node. Contributed by Anu Engineer. 2016-06-23 18:27:02 -07:00
Anu Engineer
e8de28181a HDFS-10551. o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected. Contributed by Anu Engineer. 2016-06-23 18:27:02 -07:00
Arpit Agarwal
66fa34c839 HDFS-10557. Fix handling of the -fs Generic option. (Arpit Agarwal) 2016-06-23 18:27:01 -07:00
Anu Engineer
7b23ad1ef7 HDFS-10550. DiskBalancer: fix issue of order dependency in iteration in ReportCommand test. Contributed by Xiaobing Zhou. 2016-06-23 18:27:01 -07:00
Anu Engineer
3225c24e0e HDFS-10520. DiskBalancer: Fix Checkstyle issues in test code. Contributed by Anu Engineer. 2016-06-23 18:27:00 -07:00
Anu Engineer
7e2be5c4a0 HDFS-10518. DiskBalancer: Pretty-print json in Query command. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
b502102bb1 HDFS-9461. DiskBalancer: Add Report Command. Contributed by Xiaobing Zhou. 2016-06-23 18:21:08 -07:00
Anu Engineer
64ccb232cc HDFS-10478. DiskBalancer: resolve volume path names. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
5df2d2b8fd HDFS-10399. DiskBalancer: Add JMX for DiskBalancer. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
75882ec0b0 HDFS-9545: DiskBalancer: Add Plan Command. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
1594b472bb HDFS-9543. DiskBalancer: Add Data mover. Contributed by Anu Engineer. 2016-06-23 18:20:24 -07:00
Anu Engineer
7820737cfa HDFS-9735. DiskBalancer : Refactor moveBlockAcrossStorage to be used by disk balancer. Contributed by Anu Engineer. 2016-06-23 18:20:24 -07:00
Anu Engineer
050677077b HDFS-9720. DiskBalancer : Add configuration parameters. Contributed by Anu Engineer. 2016-06-23 18:18:48 -07:00
Anu Engineer
6c606bf5c8 HDFS-9709. DiskBalancer : Add tests for disk balancer using a Mock Mover class. Contributed by Anu Engineer. 2016-06-23 18:18:48 -07:00
Arpit Agarwal
75a711a2d5 HDFS-9703. DiskBalancer: getBandwidth implementation. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
918722bdd2 HDFS-9702. DiskBalancer: getVolumeMap implementation. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
9847640603 HDFS-9683. DiskBalancer: Add cancelPlan implementation. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
e646c2eb50 HDFS-9681. DiskBalancer: Add QueryPlan implementation. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
2b1b2faf76 HDFS-9671. DiskBalancer: SubmitPlan implementation. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
66f0bb646d HDFS-9647. DiskBalancer: Add getRuntimeSettings. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
96fe685b7a HDFS-9645. DiskBalancer: Add Query RPC. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
0501d430e2 HDFS-9595. DiskBalancer: Add cancelPlan RPC. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
7100c0da35 HDFS-9588. DiskBalancer: Add submitDiskbalancer RPC. (Anu Engineer via Arpit Agarwal) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
5724a10316 HDFS-9469. DiskBalancer: Add Planner. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Tsz-Wo Nicholas Sze
30c6ebd699 HDFS-9449. DiskBalancer: Add connectors. Contributed by Anu Engineer 2016-06-23 18:18:48 -07:00
Tsz-Wo Nicholas Sze
91a5c48143 HDFS-9420. Add DataModels for DiskBalancer. Contributed by Anu Engineer 2016-06-23 18:18:48 -07:00
Xiaoyu Yao
dca298d79e HDFS-10469. Add number of active xceivers to datanode metrics. Contributed by Hanisha Koneru. 2016-06-23 14:13:39 -07:00
Kai Zheng
e6cb07520f HDFS-10460. Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block. Contributed by Rakesh R 2016-06-24 17:39:32 +08:00
Uma Maheswara Rao G
17eae9ebb3 HDFS-10473: Allow only suitable storage policies to be set on striped files. Contributed by Uma Maheswara Rao G 2016-06-22 11:17:43 -07:00
Akira Ajisaka
d433b16ce6 HDFS-7597. DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping. Contributed by Daryn Sharp, Bob Hansen, and Xiao Chen. 2016-06-22 11:00:16 +09:00
Brahma Reddy Battula
f2ac132d6a HDFS-9530. ReservedSpace is not cleared for abandoned Blocks (Contributed by Brahma Reddy Battula) 2016-06-21 15:42:28 +05:30
Jitendra Pandey
7adc4d1769 HDFS-10538. Remove AsyncDistributedFileSystem. Contributed by Xiaobing Zhou and Jitendra Pandey. 2016-06-20 17:22:55 -07:00
Colin Patrick Mccabe
5f6b4157a4 HDFS-10328. Add per-cache-pool default replication num configuration (xupeng via cmccabe) 2016-06-20 10:42:50 -07:00
Arpit Agarwal
0319d73c3b HDFS-10437. ReconfigurationProtocol not covered by HDFSPolicyProvider. (Arpit Agarwal) 2016-06-18 16:16:54 -07:00
Akira Ajisaka
09e82acaf9 HADOOP-12943. Add -w -r options in dfs -test command. Contributed by Weiwei Yang. 2016-06-17 16:20:38 +09:00
Vinayakumar B
cc6c265171 HDFS-10256. Use GenericTestUtils.getTestDir method in tests for temporary directories (Contributed by Vinayakumar B) 2016-06-16 16:47:06 +05:30
Colin Patrick Mccabe
2449db507d HDFS-10505. OIV's ReverseXML processor should support ACLs (Surendra Singh Lilhore via cmccabe) 2016-06-15 22:35:19 -07:00
Ming Ma
b48f27e794 HDFS-9922. Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes. (Chris Trezzo via mingma) 2016-06-15 22:00:52 -07:00
Ming Ma
7d521a29ee HDFS-9016. Display upgrade domain information in fsck. (mingma) 2016-06-14 20:05:50 -07:00
Andrew Wang
709a814fe0 HDFS-10516. Fix bug when warming up EDEK cache of more than one encryption zone. Contributed by Xiao Chen. 2016-06-13 11:50:19 -07:00
Ravi Prakash
ae047655f4 HDFS-10220. A large number of expired leases can make namenode unresponsive and cause failover (Nicolas Fraison via raviprak) 2016-06-08 13:44:22 -07:00
Andrew Wang
1ee9ea0026 Remove redundant TestMiniDFSCluster.testDualClusters. Contributed by Jiayi Zhou. 2016-06-08 12:58:56 -07:00
Jing Zhao
be34e85e68 HDFS-10468. HDFS read ends up ignoring an interrupt. Contributed by Jing Zhao 2016-06-07 10:48:21 -07:00
Tsz-Wo Nicholas Sze
35f255b03b Revert "Revert "HADOOP-13226 Support async call retry and failover."" 
This reverts commit 5360da8bd9.
 2016-06-06 16:31:43 +08:00
Tsz-Wo Nicholas Sze
7e7b1ae037 Revert "Revert "HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou."" 
This reverts commit 8cf47d8589.
 2016-06-06 16:31:38 +08:00
Tsz-Wo Nicholas Sze
db41e6d285 Revert "Revert "HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou"" 
This reverts commit 5ee5912ebd.
 2016-06-06 16:31:34 +08:00
Tsz-Wo Nicholas Sze
b3d81f38da Revert "Revert "HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit b82c74b910.
 2016-06-06 16:31:30 +08:00
Tsz-Wo Nicholas Sze
cba9a01889 Revert "Revert "HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit f23d5dfc60.
 2016-06-06 16:29:38 +08:00
Tsz-Wo Nicholas Sze
aa20fa150d Revert "Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou"" 
This reverts commit 4d36b221a2.
 2016-06-06 16:28:47 +08:00
Tsz-Wo Nicholas Sze
eded3d109e Revert "Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit 106234d873.
 2016-06-06 16:28:21 +08:00
Andrew Wang
106234d873 Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit fc94810d3f.
 2016-06-03 18:09:55 -07:00
Andrew Wang
4d36b221a2 Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou" 
This reverts commit 1b9f18623a.
 2016-06-03 18:09:18 -07:00
Andrew Wang
f23d5dfc60 Revert "HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit 7251bb922b.
 2016-06-03 18:09:17 -07:00
Andrew Wang
b82c74b910 Revert "HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit 02d4e478a3.
 2016-06-03 18:09:14 -07:00
Andrew Wang
5ee5912ebd Revert "HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou" 
This reverts commit f4b9bcd87c.
 2016-06-03 18:09:13 -07:00
Andrew Wang
8cf47d8589 Revert "HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou." 
This reverts commit 21890c4239.
 2016-06-03 18:09:12 -07:00
Andrew Wang
5360da8bd9 Revert "HADOOP-13226 Support async call retry and failover." 
This reverts commit 83f2f78c11.
 2016-06-03 18:09:09 -07:00
Arpit Agarwal
97e2449477 HDFS-10341. Add a metric to expose the timeout number of pending replication blocks. (Contributed by Akira Ajisaka) 2016-06-02 13:14:45 -07:00
Akira Ajisaka
1df6f5735c HDFS-10471. DFSAdmin#SetQuotaCommand's help msg is not correct. Contributed by Yiqun Lin. 2016-06-03 04:10:32 +09:00
Masatake Iwasaki
aadb77e412 HDFS-10367. TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception. Contributed by Brahma Reddy Battula. 2016-06-02 20:55:04 +09:00
Akira Ajisaka
69555fca06 HDFS-9476. TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail. Contributed by Masatake Iwasaki. 2016-06-02 18:52:47 +09:00
Kai Zheng
d749cf65e1 HDFS-9833. Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data. Contributed by Rakesh R. 2016-06-02 12:56:21 +08:00
Colin Patrick Mccabe
8ceb06e239 HADOOP-13137. TraceAdmin should support Kerberized cluster (Wei-Chiu Chuang via cmccabe) 2016-05-31 17:54:34 -07:00
Colin Patrick Mccabe
c7921c9bdd HDFS-9466. TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky (Wei-Chiu Chuang via cmccabe) 2016-05-31 17:37:52 -07:00
Colin Patrick Mccabe
29d6cadc52 HDFS-10415. TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called (Mingliang Liu via cmccabe) 2016-05-31 16:54:52 -07:00
Tsz-Wo Nicholas Sze
83f2f78c11 HADOOP-13226 Support async call retry and failover. 2016-05-31 16:30:11 -07:00
Kai Zheng
93d8a7f2a2 Revert "HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng" 
This reverts commit a6c79f92d5.
 2016-06-01 08:41:15 +08:00
Chris Nauroth
21890c4239 HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou. 2016-05-27 14:20:18 -07:00
Yongjun Zhang
5ea6fd85c7 HDFS-10276. HDFS should not expose path info that user has no permission to see. (Yuanbo Liu via Yongjun Zhang) 2016-05-27 10:02:02 -07:00
Kai Zheng
916140604f HADOOP-12911. Upgrade Hadoop MiniKDC with Kerby. Contributed by Jiajia Li 2016-05-28 14:23:39 +08:00
Karthik Kambatla
55c3e2de3d HDFS-10463. TestRollingFileSystemSinkWithHdfs needs some cleanup. (Daniel Templeton via kasha) 2016-05-26 14:24:31 -07:00
Tsz-Wo Nicholas Sze
f4b9bcd87c HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou 2016-05-26 12:15:17 -07:00
Kai Zheng
77202fa103 HADOOP-13010. Refactor raw erasure coders. Contributed by Kai Zheng 2016-05-27 13:23:34 +08:00
Kai Zheng
f69f5ab3b6 HDFS-10434. Fix intermittent test failure of TestDataNodeErasureCodingMetrics. Contributed by Rakesh R 2016-05-27 04:42:53 +08:00
Kai Zheng
a6c79f92d5 HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng 2016-05-27 04:29:42 +08:00
Tsz-Wo Nicholas Sze
02d4e478a3 HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-24 14:51:27 -07:00
Tsz-Wo Nicholas Sze
15ed080e36 HDFS-9365. Balaner does not work with the HDFS-6376 HA setup. 2016-05-24 12:49:48 -07:00
Karthik Kambatla
57c31a3fef HDFS-9782. RollingFileSystemSink should have configurable roll interval. (Daniel Templeton via kasha) 2016-05-24 10:37:00 -07:00
Allen Wittenauer
4b0f55b6ea HDFS-7766. Add a flag to WebHDFS op=CREATE to not respond with a 307 redirect (Ravi Prakash via aw) 2016-05-23 15:52:01 -07:00
Chris Nauroth
500e946729 HDFS-10438. When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server. Contributed by Chris Nauroth. 2016-05-20 13:01:43 -07:00
Chris Nauroth
bcde1562d2 HDFS-10424. DatanodeLifelineProtocol not able to use under security cluster. Contributed by Chris Nauroth. 2016-05-20 12:47:41 -07:00
Kihwal Lee
204de430bf HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 2016-05-19 15:52:32 -05:00
Andrew Wang
010e6ac328 HDFS-2173. saveNamespace should not throw IOE when only one storage directory fails to write VERSION file. Contributed by Andras Bokor. 2016-05-18 14:13:41 -07:00
Lei Xu
cf552aa87b HDFS-10360. DataNode may format directory and lose blocks if current/VERSION is missing. (Wei-Chiu Chuang via lei) 2016-05-18 09:38:37 -07:00
Arpit Agarwal
f4d8fde822 HDFS-9926. MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils. (Contributed by Josh Elser) 2016-05-18 09:20:57 -07:00
Xiaoyu Yao
dd99f5fc79 HDFS-10383. Safely close resources in DFSTestUtil. Contributed by Mingliang Liu. 2016-05-17 16:12:00 -07:00
Yongjun Zhang
e24fe2641b HDFS-9732, Improve DelegationTokenIdentifier.toString() for better logging. Contributed by Yongjun Zhang. 2016-05-17 14:03:08 -07:00
Kihwal Lee
4a5819dae2 HDFS-10303. DataStreamer#ResponseProcessor calculates packet ack latency incorrectly. Contributed by Surendra Singh Lilhore. 2016-05-17 08:53:31 -05:00
Akira Ajisaka
9fe5828f05 HDFS-10242. Cannot create space quota of zero. Contributed by Takashi Ohnishi. 2016-05-17 15:53:09 +09:00
Sangjin Lee
61f46be071 HDFS-10208. Addendum for HDFS-9579: to handle the case when client machine can't resolve network path (Ming Ma via sjlee) 2016-05-16 18:49:47 -07:00
Allen Wittenauer
730bc746f9 HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw) 
This commit contains the following JIRA issues:

    HADOOP-12931. bin/hadoop work for dynamic subcommands
    HADOOP-12932. bin/yarn work for dynamic subcommands
    HADOOP-12933. bin/hdfs work for dynamic subcommands
    HADOOP-12934. bin/mapred work for dynamic subcommands
    HADOOP-12935. API documentation for dynamic subcommands
    HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
    HADOOP-13086. enable daemonization of dynamic commands
    HADOOP-13087. env var doc update for dynamic commands
    HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
    HADOOP-13089. hadoop distcp adds client opts twice when dynamic
    HADOOP-13094. hadoop-common unit tests for dynamic commands
    HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
    HADOOP-13107. clean up how rumen is executed
    HADOOP-13108. dynamic subcommands need a way to manipulate arguments
    HADOOP-13110. add a streaming subcommand to mapred
    HADOOP-13111. convert hadoop gridmix to be dynamic
    HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
    HADOOP-13117. clarify daemonization and security vars for dynamic commands
    HADOOP-13120. add a --debug message when dynamic commands have been used
    HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
    HADOOP-13129. fix typo in dynamic subcommand docs
    HADOOP-13151. Underscores should be escaped in dynamic subcommands document
    HADOOP-13153. fix typo in debug statement for dynamic subcommands
 2016-05-16 17:54:45 -07:00
Kai Zheng
ad9441122f HDFS-8449. Add tasks count metrics to datanode for ECWorker. Contributed by Bo Li. 2016-05-15 23:39:09 -07:00
Andrew Wang
45788204ae HDFS-10333. Intermittent org.apache.hadoop.hdfs.TestFileAppend failure in trunk. Contributed by Yiqun Lin. 2016-05-15 22:15:05 -07:00
Lei Xu
a2774debf7 HDFS-9389. Add maintenance states to AdminStates. (Ming Ma via lei) 2016-05-12 15:42:54 -07:00
Tsz-Wo Nicholas Sze
7251bb922b HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-11 17:53:40 -07:00
Colin Patrick Mccabe
687233f20d HADOOP-13065. Add a new interface for retrieving FS and FC Statistics (Mingliang Liu via cmccabe) 2016-05-11 13:45:39 -07:00
Zhe Zhang
307db61aac HDFS-10172. hdfs erasurecode command should remove the redundant -usage option. Contributed by Yuanbo Liu. 2016-05-10 15:20:17 -07:00
Kihwal Lee
b9e5a32fa1 HDFS-10372. Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume. Contributed by Rushabh Shah. 2016-05-09 15:50:09 -05:00
Akira Ajisaka
411fb4b2b7 HDFS-10337. OfflineEditsViewer stats option should print 0 instead of null for the count of operations. Contributed by Yiqun Lin. 2016-05-08 15:49:13 -07:00
Masatake Iwasaki
2835f146b5 HDFS-2043. TestHFlush failing intermittently. Contributed by Lin Yiqun. 2016-05-06 15:36:47 +09:00
Xiaoyu Yao
dacd1f50fe HDFS-10324. Trash directory in an encryption zone should be pre-created with correct permissions. Contributed by Wei-Chiu Chuang. 2016-05-05 18:08:47 -07:00
Ravi Prakash
d8faf47f32 HADOOP-13051. Test for special characters in path being respected during globPaths. Contributed by Harsh J. 2016-05-05 14:18:39 -07:00
Ming Ma
1268cf5fbe HDFS-10320. Rack failures may result in NN terminate. (Xiao Chen via mingma) 2016-05-04 17:02:26 -07:00
Arpit Agarwal
6d77d6eab7 HDFS-9902. Support different values of dfs.datanode.du.reserved per storage type. (Contributed by Brahma Reddy Battula) 2016-05-03 16:52:43 -07:00
Xiaoyu Yao
45a753ccf7 HDFS-10344. DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash. Contributed by Xiaoyu Yao. 2016-05-02 19:30:47 -07:00
Tsz-Wo Nicholas Sze
1b9f18623a HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou 2016-05-02 11:16:30 -07:00
Kihwal Lee
af9b000535 HDFS-10260. TestFsDatasetImpl#testCleanShutdownOfVolume often fails. Contributed by Rushabh Shah. 2016-04-29 09:58:44 -05:00
Kihwal Lee
6243eabb48 HDFS-9958. BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages. Contributed by Kuhu Shukla 2016-04-28 12:44:53 -05:00
Lei Xu
0a152103f1 HDFS-3702. Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client. (Contributed by Lei (Eddy) Xu) 2016-04-27 14:22:51 -07:00
Kihwal Lee
919a1d824a HDFS-10330. Add Corrupt Blocks Information in Metasave output. Contributed by Kuhu Shukla. 2016-04-27 08:19:48 -05:00
Tsz-Wo Nicholas Sze
fc94810d3f HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-04-26 17:10:13 -07:00
Zhe Zhang
5865fe2bf0 HDFS-9869. Erasure Coding: Rename replication-based names in BlockManager to more generic [part-2]. Contributed by Rakesh R. 2016-04-25 22:01:54 -07:00
Masatake Iwasaki
8eadd7145e HDFS-10325. TestDFSShell#testMoveWithTargetPortEmpty fails. Contributed by Takanobu Asanuma. 2016-04-26 07:11:13 +09:00
Tsz-Wo Nicholas Sze
f308561f1d HDFS-8057 Move BlockReader implementation to the client implementation package. Contributed by Takanobu Asanuma 2016-04-25 12:01:48 -07:00
Kihwal Lee
10f0f7851a HDFS-10318. TestJMXGet hides the real error in case of test failure. Contributed by Andras Bokor. 2016-04-25 11:38:14 -05:00
Masatake Iwasaki
6fcde2e38d HDFS-9905. WebHdfsFileSystem#runWithRetry should display original stack trace on error. (Wei-Chiu Chuang via iwasakims) 2016-04-23 23:37:56 +09:00
Jing Zhao
63e5412f1a HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou. 2016-04-22 15:14:40 -07:00
Xiaoyu Yao
b4be288c5d HDFS-10207. Support enable Hadoop IPC backoff without namenode restart. Contributed by Xiaobing Zhou. 2016-04-21 10:18:48 -07:00
Chris Nauroth
63ac2db59a HDFS-10312. Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction. Contributed by Chris Nauroth. 2016-04-20 13:39:44 -07:00
Colin Patrick Mccabe
ad36fa6f42 HDFS-10308. TestRetryCacheWithHA#testRetryCacheOnStandbyNN failing (Rakesh R via cmccabe) 2016-04-20 11:33:11 -07:00
Walter Su
90d1af1416 HDFS-9744. TestDirectoryScanner#testThrottling occasionally time out after 300 seconds. Contributed by Lin Yiqun. 2016-04-19 18:54:11 +08:00
Walter Su
4b4df2cd48 HDFS-10284. o.a.h.hdfs.server.blockmanagement.TestBlockManagerSafeMode.testCheckSafeMode fails intermittently. Contributed by Mingliang Liu. 2016-04-19 18:07:46 +08:00
Steve Loughran
438efc3634 HDFS-10291 TestShortCircuitLocalRead failing (stevel) 
(cherry picked from commit dd64769699ea4c85abdf9c590b9db680837e8b27)
 2016-04-19 17:48:20 +08:00
Kai Zheng
b5d4c7dc76 HADOOP-12924. Configure raw erasure coders for supported codecs. Contributed by Rui Li. 2016-04-20 15:47:01 +08:00
Colin Patrick Mccabe
cb3ca460ef HDFS-10265. OEV tool fails to read edit xml file if OP_UPDATE_BLOCKS has no BLOCK tag (Wan Chang via cmccabe) 2016-04-18 13:47:56 -07:00
Walter Su
ab903029a9 HDFS-10275. TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly. Contributed by Lin Yiqun. 2016-04-18 20:29:29 +08:00
Walter Su
67523ffcf4 HDFS-9412. getBlocks occupies FSLock and takes too long to complete. Contributed by He Tianyi. 2016-04-18 09:28:02 +08:00
Jing Zhao
55e19b7f0c HDFS-10293. StripedFileTestUtil#readAll flaky. Contributed by Mingliang Liu. 2016-04-15 10:53:40 -07:00
Jing Zhao
89a838769f HDFS-10283. o.a.h.hdfs.server.namenode.TestFSImageWithSnapshot#testSaveLoadImageWithAppending fails intermittently. Contributed by Mingliang Liu. 2016-04-15 10:49:21 -07:00
Kihwal Lee
b9c9d03591 HDFS-10281. TestPendingCorruptDnMessages fails intermittently. Contributed by Mingliang Liu. 2016-04-14 15:24:39 -05:00
Xiaoyu Yao
809226752d HDFS-10286. Fix TestDFSAdmin#testNameNodeGetReconfigurableProperties. Contributed by Xiaobing Zhou. 2016-04-14 10:57:02 -07:00
Kihwal Lee
0d1c1152f1 HDFS-10282. The VolumeScanner should warn about replica files which are misplaced. Contributed by Colin Patrick McCabe. 2016-04-14 07:58:24 -05:00
Xiaoyu Yao
5566177c9a HDFS-10209. Support enable caller context in HDFS namenode audit log without restart namenode. Contributed by Xiaobing Zhou. 2016-04-13 16:51:13 -07:00
Xiaoyu Yao
4895c73dd4 Revert "HDFS-10209. Support enable caller context in HDFS namenode audit log without restart namenode" 
This reverts commit 192112d5a2.
 2016-04-13 16:45:18 -07:00
Andrew Wang
314aa21a89 HDFS-10279. Improve validation of the configured number of tolerated failed volumes. Contributed by Lin Yiqun. 2016-04-13 16:39:50 -07:00
Xiaobing Zhou
192112d5a2 HDFS-10209. Support enable caller context in HDFS namenode audit log without restart namenode 2016-04-13 15:34:41 -07:00
Kihwal Lee
d2f3bbc290 HDFS-10270. TestJMXGet:testNameNode() fails. Contributed by Gergely Novák 2016-04-13 11:22:36 -05:00
Walter Su
903428bf94 HDFS-9772. TestBlockReplacement#testThrottler doesn't work as expected. Contributed by Lin Yiqun. 2016-04-13 17:49:45 +08:00
Zhe Zhang
6ef42873a0 HDFS-9918. Erasure Coding: Sort located striped blocks based on decommissioned states. Contributed by Rakesh R. 2016-04-12 13:38:58 -07:00
Akira Ajisaka
209303be3a HDFS-8356. Document missing properties in hdfs-default.xml. Contributed by Ray Chiang. 2016-04-12 15:26:50 +09:00
Vinayakumar B
a9a607f8fc HDFS-10271. Extra bytes are getting released from reservedSpace for append (Contributed by Brahma Reddy Battula) 2016-04-12 11:44:34 +05:30
Uma Maheswara Rao G
3c18a53cbd HDFS-9719. Refactoring ErasureCodingWorker into smaller reusable constructs. Contributed by Kai Zheng. 2016-04-06 22:50:24 -07:00
Colin Patrick Mccabe
4bd7cbc29d HDFS-10267. Extra "synchronized" on FsDatasetImpl#recoverAppend and FsDatasetImpl#recoverClose 2016-04-06 21:07:31 -07:00
Eric Payne
aede8c10ec HDFS-9945. Datanode command for evicting writers. Contributed by Kihwal Lee 2016-04-06 20:20:14 +00:00
Colin Patrick Mccabe
188f65287d HDFS-6520. hdfs fsck passes invalid length value when creating BlockReader (Xiao Chen via cmccabe) 2016-04-06 11:28:34 -07:00
Jing Zhao
221b3a8722 HDFS-10192. Namenode safemode not coming out during failover. Contributed by Brahma Reddy Battula. 2016-04-06 10:42:59 -07:00
Kihwal Lee
917464505c HDFS-10239. Fsshell mv fails if port usage doesn't match in src and destination paths. Contributed by Kuhu Shukla. 2016-04-05 09:07:24 -05:00
Vinayakumar B
818d6b799e HDFS-9917. IBR accumulate more objects when SNN was down for sometime. (Contributed by Brahma Reddy Battula) 2016-04-05 09:49:39 +08:00
Colin Patrick Mccabe
f6b1a81812 HDFS-8496. Calling stopWriter() with FSDatasetImpl lock held may block other threads (cmccabe) 2016-04-04 18:02:15 -07:00
Kihwal Lee
a7d1fb0cd2 HDFS-10178. Permanent write failures can happen if pipeline recoveries occur for the first packet. Contributed by Kihwal Lee. 2016-04-04 16:40:00 -05:00
Masatake Iwasaki
154d2532cf HDFS-9599. TestDecommissioningStatus.testDecommissionStatus occasionally fails (Lin Yiqun via iwasakims) 2016-04-05 03:19:48 +09:00
Vinayakumar B
54b2e78fd2 HDFS-10253. Fix TestRefreshCallQueue failure (Contributed by Xiaoyu Yao) 2016-04-03 13:27:49 +05:30
Allen Wittenauer
0a74610d1c HADOOP-11393. Revert HADOOP_PREFIX, go back to HADOOP_HOME (aw) 2016-03-31 07:51:05 -07:00
Colin Patrick Mccabe
37e23ce45c HDFS-10223. peerFromSocketAndKey performs SASL exchange before setting connection timeouts (cmccabe) 2016-03-30 13:37:37 -07:00
Vinayakumar B
09d63d5a19 HDFS-5177. blocksScheduled count should be decremented for abandoned blocks (Contributed by Vinayakumar B) 2016-03-30 14:22:00 +08:00
Arpit Agarwal
ddfe6774c2 HDFS-9439. Support reconfiguring fs.protected.directories without NN restart. (Contributed by Xiaobing Zhou) 2016-03-29 13:55:00 -07:00
Arpit Agarwal
c9307e48b7 HDFS-10228. TestHDFSCLI fails. (Contributed by Akira AJISAKA) 2016-03-29 13:50:08 -07:00
Andrew Wang
f2aec4eb82 HDFS-10197. TestFsDatasetCache failing intermittently due to timeout. Contributed by Lin Yiqun. 2016-03-29 11:54:42 -07:00
Uma Maheswara Rao G
3a4ff7776e HDFS-9694. Make existing DFSClient#getFileChecksum() work for striped blocks. Contributed by Kai Zheng 2016-03-26 19:58:09 -07:00
Lei Xu
fde8ac5d85 Add missing files from HDFS-9005. (lei) 2016-03-25 17:11:35 -07:00
Lei Xu
4fcfea71bf HDFS-9005. Provide support for upgrade domain script. (Ming Ma via Lei Xu) 2016-03-25 17:09:12 -07:00
Andrew Wang
e3bb38d625 HDFS-9405. Warmup NameNode EDEK caches in background thread. Contributed by Xiao Chen. 2016-03-21 11:39:05 -07:00
Sangjin Lee
cd8b6889a7 HDFS-9579. Provide bytes-read-by-network-distance metrics at FileSystem.Statistics level (Ming Ma via sjlee) 2016-03-19 14:02:04 -07:00
Kihwal Lee
63c966a3fb HDFS-9874. Long living DataXceiver threads cause volume shutdown to block. Contributed by Rushabh Shah. 2016-03-18 10:24:59 -05:00
Colin Patrick Mccabe
dc951e606f HDFS-9949. Add a test case to ensure that the DataNode does not regenerate its UUID when a storage directory is cleared (Harsh J via cmccabe) 2016-03-17 10:37:42 -07:00
Zhe Zhang
32d043d9c5 HDFS-9857. Erasure Coding: Rename replication-based names in BlockManager to more generic [part-1]. Contributed by Rakesh R. 2016-03-16 16:53:58 -07:00
Kihwal Lee
d457401784 HDFS-9904. testCheckpointCancellationDuringUpload occasionally fails. Contributed by Lin Yiqun. 2016-03-15 10:52:47 -05:00
Arpit Agarwal
e01c6ea688 HDFS-1477. Support reconfiguring dfs.heartbeat.interval and dfs.namenode.heartbeat.recheck-interval without NN restart. (Contributed by Xiaobing Zhou) 2016-03-10 19:03:55 -08:00
Colin Patrick Mccabe
bd49354c6d HDFS-9934. ReverseXML oiv processor should bail out if the XML file's layoutVersion doesn't match oiv's (cmccabe) 2016-03-10 13:41:06 -08:00
Zhe Zhang
7600e3c48f HDFS-7866. Erasure coding: NameNode manages multiple erasure coding policies. Contributed by Rui Li. 2016-03-08 22:30:30 -08:00
Andrew Wang
a14a6f08ee HDFS-9919. TestStandbyCheckpoints#testNonPrimarySBNUploadFSImage waitForCheckpoint incorrectly. Contributed by Lin Yiqun. 2016-03-08 11:54:50 -08:00
Jing Zhao
743a99f2db HDFS-8786. Erasure coding: use simple replication for internal blocks on decommissioning datanodes. Contributed by Rakesh R. 2016-03-08 10:24:57 -08:00
Masatake Iwasaki
d718fc1ee5 HDFS-9865. TestBlockReplacement fails intermittently in trunk (Lin Yiqun via iwasakims) 2016-03-08 02:07:18 +09:00
Chris Nauroth
2759689d7d HDFS-9239. DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness. Contributed by Chris Nauroth. 2016-03-04 15:29:50 -08:00
Zhe Zhang
3e8099a45a HDFS-9888. Allow reseting KerberosName in unit tests. Contributed by Xiao Chen. 2016-03-04 09:48:05 -08:00
Arpit Agarwal
27941a1811 HDFS-9534. Add CLI command to clear storage policy from a path. (Contributed by Xiaobing Zhou) 2016-03-02 18:35:28 -08:00
Colin Patrick Mccabe
700b0e4019 HDFS-9835. OIV: add ReverseXML processor which reconstructs an fsimage from an XML file (cmccabe) 2016-03-02 17:56:11 -08:00
Xiaoyu Yao
5abf051249 HDFS-9887. WebHdfs socket timeouts should be configurable. Contributed by Austin Donnelly and Chris Douglas. 2016-03-02 09:23:17 -08:00
Akira Ajisaka
27e0681f28 HDFS-9851. NameNode throws NPE when setPermission is called on a path that does not exist. Contributed by Brahma Reddy Battula. 2016-03-02 14:43:03 +09:00
Akira Ajisaka
e2ddf82469 HDFS-9766. TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently. Contributed by Xiao Chen. 2016-03-02 14:17:35 +09:00
Andrew Wang
4abb2fa687 HDFS-9881. DistributedFileSystem#getTrashRoot returns incorrect path for encryption zones. 2016-03-01 16:35:39 -08:00
Kihwal Lee
e76b13c415 HDFS-9880. TestDatanodeRegistration fails occasionally. Contributed by Kihwal Lee. 2016-03-01 17:00:18 -06:00
Kihwal Lee
2c8496ebf3 HDFS-8791. block ID-based DN storage layout can be very slow for datanode on ext4. Contributed by Chris Trezzo. 2016-03-01 15:04:59 -06:00
Jing Zhao
2151716832 HDFS-7964. Add support for async edit logging. Contributed by Daryn Sharp. 2016-02-29 15:34:43 -08:00
Jing Zhao
408f2c807b HDFS-9866. BlockManager#chooseExcessReplicasStriped may weaken rack fault tolerance. Contributed by Jing Zhao. 2016-02-28 14:54:49 -08:00
Tsz-Wo Nicholas Sze
d1d4e16690 HDFS-9710. DN can be configured to send block receipt IBRs in batches. 2016-02-26 15:32:25 -08:00
Karthik Kambatla
c2460dad64 HDFS-9858. RollingFileSystemSink can throw an NPE on non-secure clusters. (Daniel Templeton via kasha) 2016-02-25 16:31:09 -08:00
Zhe Zhang
8808779db3 HDFS-9734. Refactoring of checksum failure report related codes. Contributed by Kai Zheng. 
Change-Id: Ie69a77e3498a360959f8e213c51fb2b17c28b64a
 2016-02-25 09:55:50 -08:00
Zhe Zhang
ccff6035f5 HDFS-9804. Allow long-running Balancer to login with keytab. Contributed by Xiao Chen. 
Change-Id: I54b82e0b11929082057b4d21bfcf285222c94cdb
 2016-02-25 09:24:32 -08:00
Tsz-Wo Nicholas Sze
6979cbfc1f HDFS-9838. Refactor the excessReplicateMap to a class. 2016-02-24 19:42:47 -08:00
Jing Zhao
47b92f2b6f HDFS-9837. BlockManager#countNodes should be able to detect duplicated internal blocks. Contributed by Jing Zhao. 2016-02-24 15:13:56 -08:00
Kihwal Lee
d27d7fc72e HDFS-9395. Make HDFS audit logging consistant. Contributed by Kuhu Shukla. 2016-02-24 12:04:01 -06:00
Zhe Zhang
def754ec06 HDFS-9816. Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]. Contributed by Rui Li. 
Change-Id: I64b57bab4722cdc6e1e3148c3a3a401370249afe
 2016-02-23 14:05:17 -08:00
Zhe Zhang
647a35e996 HDFS-9844. Correct path creation in getTrashRoot to handle root dir. 
Change-Id: I5afe6c7176063892e3e632698c849bd9f4b92906
 2016-02-23 13:29:39 -08:00
Colin Patrick Mccabe
211c78c090 HDFS-9549. TestCacheDirectives#testExceedsCapacity is flaky (Xiao Chen via cmccabe) 2016-02-23 12:01:20 -08:00
Vinayakumar B
342c9572bf HDFS-9425. Expose number of blocks per volume as a metric (Contributed by Brahma Reddy Battula) 2016-02-22 09:29:23 +05:30
Jing Zhao
e54cc29312 HDFS-9818. Correctly handle EC reconstruction work caused by not enough racks. Contributed by Jing Zhao. 2016-02-19 19:02:23 -08:00
Tsz-Wo Nicholas Sze
bbfaf3c271 HADOOP-10865. Add a Crc32 chunked verification benchmark for both directly and non-directly buffer cases. 2016-02-18 11:47:33 -08:00
cnauroth
5d1889a66d HDFS-9711. Integrate CSRF prevention filter in WebHDFS. Contributed by Chris Nauroth. 2016-02-18 10:07:28 -08:00
Andrew Wang
3a23dc683c HDFS-9608. Disk IO imbalance in HDFS with heterogeneous storages. Contributed by Wei Zhou. 2016-02-17 11:29:58 -08:00
Vinayakumar B
2536ece7b6 HDFS-9787. SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false. (Contributed by Guocui Mi) 2016-02-17 10:41:24 +05:30
Junping Du
77ba5add0d HDFS-9456. BlockPlacementPolicyWithNodeGroup should override verifyBlockPlacement(). Contributed by Xiaobing Zhou. 2016-02-16 18:55:55 -08:00
Akira Ajisaka
8ed07bd3e6 HDFS-9765. TestBlockScanner#testVolumeIteratorWithCaching fails intermittently. (aajisaka) 2016-02-15 19:08:03 +09:00
Akira Ajisaka
b0738ae673 HDFS-9773. Remove dead code related to SimulatedFSDataset in tests. Contributed by Brahma Reddy Battula. 2016-02-15 15:35:06 +09:00
Akira Ajisaka
ac5da11399 HDFS-9691. TestBlockManagerSafeMode#testCheckSafeMode fails intermittently. Contributed by Mingliang Liu. 2016-02-15 14:00:09 +09:00
Arpit Agarwal
1de1641f17 HDFS-9801. ReconfigurableBase should update the cached configuration. (Arpit Agarwal) 2016-02-12 12:41:04 -08:00
Jing Zhao
972782d956 HDFS-9754. Avoid unnecessary getBlockCollection calls in BlockManager. Contributed by Jing Zhao. 2016-02-12 11:07:52 -08:00
Karthik Kambatla
33ce7f6c07 HDFS-9780. RollingFileSystemSink doesn't work on secure clusters. (Daniel Templeton via kasha) 2016-02-11 20:37:14 -08:00
Lei Xu
c7fcec24b8 HDFS-9795. OIV Delimited should show which files are ACL-enabled (lei) 2016-02-11 20:33:51 -08:00
Zhe Zhang
0aa8c82894 HDFS-9755. Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2]. Contributed by Rui Li. 
Change-Id: I2100bc27ad484f83c9cb2d2e5bb232f4f74fd286
 2016-02-11 10:14:09 -08:00
Jing Zhao
19adb2bc64 HDFS-9789. Correctly update DataNode's scheduled block size when writing small EC file. Contributed by Jing Zhao. 2016-02-10 21:24:00 -08:00
Karthik Kambatla
bc425a623f HDFS-9637. Tests for RollingFileSystemSink. (Daniel Templeton via kasha) 2016-02-10 10:08:05 -08:00
Akira Ajisaka
802a7edfae HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 2016-02-10 23:49:14 +09:00
Akira Ajisaka
5e565ce7a7 Revert "HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta." 
This reverts commit fe124da5ff.
 2016-02-10 23:43:33 +09:00
Uma Maheswara Rao G
a7fce9ab41 HDFS-9779 . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value. Contributed by Kuhu Shukla. 2016-02-09 15:14:30 -08:00
Zhe Zhang
a0fb2eff9b HDFS-9775. Erasure Coding : Rename BlockRecoveryWork to BlockReconstructionWork. Contributed by Rakesh R. 
Change-Id: I6dfc8efd94fa2bbb4eec0e4730a5a4f92c8a5519
 2016-02-09 14:43:59 -08:00
Allen Wittenauer
401ae4ecdb HDFS-9760. WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler (Ryan Sasson via aw) 2016-02-09 14:15:21 -08:00
zhezhang
dbe49c1bd6 HDFS-9244. Support nested encryption zones. 
Change-Id: I43a13035a8b27956e90967ee82058efb647c3415
 2016-02-08 16:30:51 -08:00
Kihwal Lee
193d27de0a HDFS-9752. Permanent write failures may happen to slow writers during datanode rolling upgrades. Contributed by Walter Su. 2016-02-08 12:16:29 -06:00
Akira Ajisaka
fe124da5ff HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 2016-02-06 19:31:01 +09:00
Tsz-Wo Nicholas Sze
4e5e1c0f99 HDFS-9726. Refactor IBR code to a new class. 2016-02-05 23:17:59 +08:00
Konstantin V Shvachko
81ef1a1f55 HDFS-9503. Use generic option -fs for NNThroughputBenchmark instead of -namenode. Contributed by Mingliang Liu. 2016-02-03 14:59:46 -08:00
zhezhang
a0f5e83f28 HDFS-9658. Erasure Coding: allow to use multiple EC policies in striping related tests. Contributed by Rui Li. 
Change-Id: I5b02f5bde4d343b7529c6a7fe5df73bd99c6cb24
 2016-02-03 14:41:07 -08:00
Jing Zhao
7badf15604 HDFS-9748. Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice. Contributed by Walter Su. 2016-02-03 13:30:09 -08:00
Vinayakumar B
eb2fb943fd HDFS-9740. Use a reasonable limit in DFSTestUtil.waitForMetric() (Contributed by Chang Li) 2016-02-03 11:14:29 +05:30
Vinayakumar B
04375756a5 HDFS-9715. Check storage ID uniqueness on datanode startup (Contributed by Lei (Eddy) Xu) 2016-02-03 07:35:01 +05:30
zhezhang
6d1213860f HDFS-9403. Erasure coding: some EC tests are missing timeout. Contributed by Gao Rui. 
Change-Id: I13c4bc0e566c4f8af5371d26f8e18a749ed41d8b
 2016-02-02 14:03:45 -08:00
zhezhang
4ae543fdcd HDFS-9731. Erasure Coding: Rename BlockECRecoveryCommand to BlockECReconstructionCommand. Contributed by Rakesh R. 
Change-Id: I405365a8395770e494b92bfe9651f4f0366d8f28
 2016-02-02 12:32:08 -08:00
Colin Patrick Mccabe
dd9ebf6eed HDFS-9260. Improve the performance and GC friendliness of NameNode startup and full block reports (Staffan Friberg via cmccabe) 2016-02-02 11:23:00 -08:00
Lei Xu
9d494f0c0e HDFS-9721. Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference. (Xiao Chen via lei) 2016-02-02 10:42:01 -08:00
Vinayakumar B
4d7055e0c8 HDFS-9718. HAUtil#getConfForOtherNodes should unset independent generic keys before initialize (Contributed by DENG FEI) 2016-02-02 12:51:35 +05:30
Yongjun Zhang
34ab50ea92 HDFS-9406. FSImage may get corrupted after deleting snapshot. (Contributed by Jing Zhao, Stanislav Antic, Vinayakumar B, Yongjun Zhang) 2016-02-01 13:56:55 -08:00
Lei Xu
e50aa53eed HDFS-9701. DN may deadlock when hot-swapping under load. (Xiao Chen via lei) 2016-02-01 12:56:53 -08:00
Steve Loughran
af2dccbca5 HDFS-9708. FSNamesystem.initAuditLoggers() doesn't trim classnames (Mingliang Liu via stevel) 2016-02-01 16:01:50 +00:00
Colin Patrick Mccabe
f67149ab08 HDFS-7764. DirectoryScanner shouldn't abort the scan if one directory had an error (Rakesh R via cmccabe) 2016-01-28 19:54:50 -08:00
Tsz-Wo Nicholas Sze
662e17b46a HDFS-9654. Code refactoring for HDFS-8578. 2016-01-28 10:58:03 +08:00
Jing Zhao
ec25c7f9c7 HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu. 2016-01-27 16:34:40 -08:00
Jing Zhao
3a9571308e Revert "HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu." 
This reverts commit 8a91109d16.
 2016-01-27 16:31:19 -08:00
Jing Zhao
8a91109d16 HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu. 2016-01-27 15:48:47 -08:00
Tsz-Wo Nicholas Sze
45c763ad61 HDFS-9690. ClientProtocol.addBlock is not idempotent after HDFS-8071. 2016-01-26 11:20:13 +08:00
Tsz-Wo Nicholas Sze
bd909ed9f2 HDFS-8999. Allow a file to be closed with COMMITTED but not yet COMPLETE blocks. 2016-01-26 10:32:51 +08:00
Jitendra Pandey
e8650fea1f HDFS-9672. o.a.h.hdfs.TestLeaseRecovery2 fails intermittently. Contributed by Mingliang Liu. 2016-01-25 16:08:46 -08:00
Arpit Agarwal
d62b4a4de7 HDFS-9094. Add command line option to ask NameNode reload configuration. (Contributed by Xiaobing Zhou) 2016-01-25 12:17:05 -08:00
Eric Yang
10a2bc0dff HDFS-9653. Added blocks pending deletion report to dfsadmin. 
(Weiwei Yang via eyang)
 2016-01-24 14:19:49 -08:00
Allen Wittenauer
d22c4239a4 HDFS-9525. hadoop utilities need to support provided delegation tokens (HeeSoo Kim via aw) 2016-01-22 12:15:22 -08:00
Jing Zhao
95363bcc7d HDFS-9646. ErasureCodingWorker may fail when recovering data blocks with length less than the first internal block. Contributed by Jing Zhao. 2016-01-22 09:46:02 -08:00
Kihwal Lee
2fd19b9674 HDFS-8898. Create API and command-line argument to get quota and quota usage without detailed content summary. Contributed by Ming Ma. 2016-01-22 09:10:06 -06:00
Masatake Iwasaki
b2ffcc2915 HDFS-9601. NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block (iwasakims) 2016-01-22 12:28:38 +09:00
Jing Zhao
c304890c8c HDFS-9542. Move BlockIdManager from FSNamesystem to BlockManager. Contributed by Jing Zhao. 2016-01-21 11:13:17 -08:00
Kihwal Lee
3616c7b855 HDFS-9634. webhdfs client side exceptions don't provide enough details. Contributed by Eric Payne. 2016-01-21 09:32:57 -06:00
Vinayakumar B
b7372b7166 HDFS-9625. set replication for empty file failed when set storage policy (Contributed by DENG FEI) 2016-01-21 19:17:05 +05:30
Yongjun Zhang
57d0a94305 HDFS-6054. MiniQJMHACluster should not use static port to avoid binding failure in unit test. (Yongjun Zhang) 2016-01-19 22:54:47 -08:00
Andrew Wang
c07f7fa8ff HDFS-9624. DataNode start slowly due to the initial DU command operations. (Lin Yiqun via wang) 2016-01-15 11:28:46 -08:00
Yongjun Zhang
817cc1f02a HDFS-9648. TestStartup.testImageChecksum is broken by HDFS-9569's message change. (Wei-Chiu Chuang via Yongjun Zhang) 2016-01-14 12:36:36 -08:00
Yongjun Zhang
25051c3bd0 HDFS-9569. Log the name of the fsimage being loaded for better supportability. (Yongjun Zhang) 2016-01-12 09:22:22 -08:00
Jing Zhao
9f4bf3bdf9 HDFS-9621. getListing wrongly associates Erasure Coding policy to pre-existing replicated files under an EC directory. Contributed by Jing Zhao. 2016-01-11 11:31:59 -08:00
Lei Xu
fd8065a763 HDFS-9493. Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk. (Tony Wu via lei) 2016-01-08 15:00:30 -08:00
Zhe Zhang
0af2022e6d HDFS-9626. TestBlockReplacement#testBlockReplacement fails occasionally. Contributed by Xiao Chen. 
Change-Id: Ia766a5bfb89f8cf8a48b473bb27c30d5818ebfc0
 2016-01-08 13:14:36 -08:00
Kihwal Lee
38c4c14472 HDFS-9574. Reduce client failures during datanode restart. Contributed by Kihwal Lee. 2016-01-08 11:13:58 -06:00
Vinayakumar B
6702e7d66a HDFS-9619. SimulatedFSDataset sometimes can not find blockpool for the correct namenode (Contributed by Wei-Chiu Chuang) 2016-01-07 14:20:18 +05:30
Vinayakumar B
34cd7cd765 HDFS-9600. do not check replication if the block is under construction (Contributed by Phil Yang) 2016-01-07 11:27:42 +05:30
Arpit Agarwal
67c9780609 HDFS-9498. Move code that tracks blocks with future generation stamps to BlockManagerSafeMode. (Contributed by Mingliang Liu) 2016-01-06 10:30:59 -08:00
cnauroth
84a8147791 HDFS-9376. TestSeveralNameNodes fails occasionally. Contributed by Masatake Iwasaki. 2015-12-29 14:37:03 -08:00
cnauroth
99cf2ecee9 HDFS-7553. fix the TestDFSUpgradeWithHA due to BindException. Contributed by Xiao Chen. 2015-12-29 10:56:59 -08:00
Benoy Antony
8cfd672397 HDFS-9597. BaseReplicationPolicyTest should update data node stats after adding a data node. Contributed by Wei-Chiu Chuang. 2015-12-25 08:55:39 -08:00
Benoy Antony
df83230948 HDFS-9034. StorageTypeStats Metric should not count failed storage. Contributed by Surendra Singh Lilhore. 2015-12-22 15:28:17 -08:00