big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
13,825 Commits 2 Branches 2 Tags 264 MiB
8d202f1258
Commit Graph

5632 Commits

Author SHA1 Message Date
Arpit Agarwal
e325c6ade9 HDFS-9611. DiskBalancer : Replace htrace json imports with jackson. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Tsz-Wo Nicholas Sze
599eca07b4 HDFS-9526. Fix jackson annotation imports. Contributed by Xiaobing Zhou 2016-06-23 18:18:48 -07:00
Tsz-Wo Nicholas Sze
30c6ebd699 HDFS-9449. DiskBalancer: Add connectors. Contributed by Anu Engineer 2016-06-23 18:18:48 -07:00
Tsz-Wo Nicholas Sze
91a5c48143 HDFS-9420. Add DataModels for DiskBalancer. Contributed by Anu Engineer 2016-06-23 18:18:48 -07:00
Xiaoyu Yao
dca298d79e HDFS-10469. Add number of active xceivers to datanode metrics. Contributed by Hanisha Koneru. 2016-06-23 14:13:39 -07:00
Kai Zheng
e6cb07520f HDFS-10460. Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block. Contributed by Rakesh R 2016-06-24 17:39:32 +08:00
Akira Ajisaka
ff07b10803 HDFS-10561. test_native_mini_dfs fails by NoClassDefFoundError. (aajisaka) 2016-06-23 15:48:18 +09:00
Uma Maheswara Rao G
01643020d2 HDFS-10555: Unable to loadFSEdits due to a failure in readCachePoolInfo. Contributed by Uma Maheswara Rao G 2016-06-22 22:06:13 -07:00
Uma Maheswara Rao G
17eae9ebb3 HDFS-10473: Allow only suitable storage policies to be set on striped files. Contributed by Uma Maheswara Rao G 2016-06-22 11:17:43 -07:00
Akira Ajisaka
d433b16ce6 HDFS-7597. DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping. Contributed by Daryn Sharp, Bob Hansen, and Xiao Chen. 2016-06-22 11:00:16 +09:00
Brahma Reddy Battula
f2ac132d6a HDFS-9530. ReservedSpace is not cleared for abandoned Blocks (Contributed by Brahma Reddy Battula) 2016-06-21 15:42:28 +05:30
Colin Patrick Mccabe
46f1602e89 HDFS-10448. CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1 (Yiqun Lin via cmccabe) 2016-06-20 18:25:09 -07:00
Jitendra Pandey
7adc4d1769 HDFS-10538. Remove AsyncDistributedFileSystem. Contributed by Xiaobing Zhou and Jitendra Pandey. 2016-06-20 17:22:55 -07:00
Jitendra Pandey
b7c4cf7129 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:25:30 -07:00
Aaron T. Myers
aa1b583cf9 HDFS-10423. Increase default value of httpfs maxHttpHeaderSize. Contributed by Nicolae Popa. 2016-06-20 13:46:56 -07:00
Colin Patrick Mccabe
5f6b4157a4 HDFS-10328. Add per-cache-pool default replication num configuration (xupeng via cmccabe) 2016-06-20 10:42:50 -07:00
Arpit Agarwal
0319d73c3b HDFS-10437. ReconfigurationProtocol not covered by HDFSPolicyProvider. (Arpit Agarwal) 2016-06-18 16:16:54 -07:00
Kihwal Lee
280069510b HDFS-10493. Add links to datanode web UI in namenode datanodes page. Contributed by Weiwei Yang. 2016-06-17 11:05:44 -05:00
Akira Ajisaka
09e82acaf9 HADOOP-12943. Add -w -r options in dfs -test command. Contributed by Weiwei Yang. 2016-06-17 16:20:38 +09:00
Arpit Agarwal
e983eafb44 HDFS-10532. Typo in RollingUpgrade docs. (Contributed by Yiqun Lin) 2016-06-16 10:10:33 -07:00
Vinayakumar B
cc6c265171 HDFS-10256. Use GenericTestUtils.getTestDir method in tests for temporary directories (Contributed by Vinayakumar B) 2016-06-16 16:47:06 +05:30
Colin Patrick Mccabe
2ca73445f5 HDFS-10525. Fix NPE in CacheReplicationMonitor#rescanCachedBlockMap (Xiao Chen via cmccabe) 2016-06-15 22:47:01 -07:00
Colin Patrick Mccabe
2449db507d HDFS-10505. OIV's ReverseXML processor should support ACLs (Surendra Singh Lilhore via cmccabe) 2016-06-15 22:35:19 -07:00
Ming Ma
b48f27e794 HDFS-9922. Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes. (Chris Trezzo via mingma) 2016-06-15 22:00:52 -07:00
Ming Ma
7d521a29ee HDFS-9016. Display upgrade domain information in fsck. (mingma) 2016-06-14 20:05:50 -07:00
Andrew Wang
709a814fe0 HDFS-10516. Fix bug when warming up EDEK cache of more than one encryption zone. Contributed by Xiao Chen. 2016-06-13 11:50:19 -07:00
Allen Wittenauer
d44f4745b4 HDFS-7987. Allow files / directories to be moved (Ravi Prakash via aw) 2016-06-10 09:02:28 -07:00
Masatake Iwasaki
8ea9bbce26 HDFS-10508. DFSInputStream should set thread's interrupt status after catching InterruptException from sleep. Contributed by Jing Zhao. 2016-06-09 14:52:29 +09:00
Ravi Prakash
ae047655f4 HDFS-10220. A large number of expired leases can make namenode unresponsive and cause failover (Nicolas Fraison via raviprak) 2016-06-08 13:44:22 -07:00
Andrew Wang
1ee9ea0026 Remove redundant TestMiniDFSCluster.testDualClusters. Contributed by Jiayi Zhou. 2016-06-08 12:58:56 -07:00
Jing Zhao
be34e85e68 HDFS-10468. HDFS read ends up ignoring an interrupt. Contributed by Jing Zhao 2016-06-07 10:48:21 -07:00
Akira Ajisaka
e620530301 HDFS-10485. Fix findbugs warning in FSEditLog.java. (aajisaka) 2016-06-07 17:52:55 +09:00
Zhe Zhang
6de9213df1 HDFS-10458. getFileEncryptionInfo should return quickly for non-encrypted cluster. 2016-06-06 15:52:39 -07:00
Tsz-Wo Nicholas Sze
35f255b03b Revert "Revert "HADOOP-13226 Support async call retry and failover."" 
This reverts commit 5360da8bd9.
 2016-06-06 16:31:43 +08:00
Tsz-Wo Nicholas Sze
7e7b1ae037 Revert "Revert "HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou."" 
This reverts commit 8cf47d8589.
 2016-06-06 16:31:38 +08:00
Tsz-Wo Nicholas Sze
db41e6d285 Revert "Revert "HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou"" 
This reverts commit 5ee5912ebd.
 2016-06-06 16:31:34 +08:00
Tsz-Wo Nicholas Sze
b3d81f38da Revert "Revert "HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit b82c74b910.
 2016-06-06 16:31:30 +08:00
Tsz-Wo Nicholas Sze
574dcd34c0 Revert "Revert "HADOOP-13168. Support Future.get with timeout in ipc async calls."" 
This reverts commit e4450d47f1.
 2016-06-06 16:31:23 +08:00
Tsz-Wo Nicholas Sze
cba9a01889 Revert "Revert "HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit f23d5dfc60.
 2016-06-06 16:29:38 +08:00
Tsz-Wo Nicholas Sze
aa20fa150d Revert "Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou"" 
This reverts commit 4d36b221a2.
 2016-06-06 16:28:47 +08:00
Tsz-Wo Nicholas Sze
eded3d109e Revert "Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit 106234d873.
 2016-06-06 16:28:21 +08:00
Andrew Wang
106234d873 Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit fc94810d3f.
 2016-06-03 18:09:55 -07:00
Andrew Wang
4d36b221a2 Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou" 
This reverts commit 1b9f18623a.
 2016-06-03 18:09:18 -07:00
Andrew Wang
f23d5dfc60 Revert "HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit 7251bb922b.
 2016-06-03 18:09:17 -07:00
Andrew Wang
e4450d47f1 Revert "HADOOP-13168. Support Future.get with timeout in ipc async calls." 
This reverts commit 42c22f7e3d.
 2016-06-03 18:09:16 -07:00
Andrew Wang
b82c74b910 Revert "HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit 02d4e478a3.
 2016-06-03 18:09:14 -07:00
Andrew Wang
5ee5912ebd Revert "HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou" 
This reverts commit f4b9bcd87c.
 2016-06-03 18:09:13 -07:00
Andrew Wang
8cf47d8589 Revert "HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou." 
This reverts commit 21890c4239.
 2016-06-03 18:09:12 -07:00
Andrew Wang
5360da8bd9 Revert "HADOOP-13226 Support async call retry and failover." 
This reverts commit 83f2f78c11.
 2016-06-03 18:09:09 -07:00
Andrew Wang
47e0321ee9 HDFS-10481. HTTPFS server should correctly impersonate as end user to open file. Contributed by Xiao Chen. 2016-06-03 17:21:17 -07:00
Allen Wittenauer
99a771cd7a HDFS-7767. Use the noredirect flag in WebHDFS to allow web browsers to upload files via the NN UI (Ravi Prakash via aw) 2016-06-03 17:07:39 -07:00
Allen Wittenauer
15f018434c HDFS-9877. HDFS Namenode UI: Fix browsing directories that need to be encoded (Ravi Prakash via aw) 2016-06-03 17:06:29 -07:00
Andrew Wang
713cb71820 HADOOP-13155. Implement TokenRenewer to renew and cancel delegation tokens in KMS. Contributed by Xiao Chen. 2016-06-03 16:48:54 -07:00
Arpit Agarwal
97e2449477 HDFS-10341. Add a metric to expose the timeout number of pending replication blocks. (Contributed by Akira Ajisaka) 2016-06-02 13:14:45 -07:00
Akira Ajisaka
1df6f5735c HDFS-10471. DFSAdmin#SetQuotaCommand's help msg is not correct. Contributed by Yiqun Lin. 2016-06-03 04:10:32 +09:00
Masatake Iwasaki
aadb77e412 HDFS-10367. TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception. Contributed by Brahma Reddy Battula. 2016-06-02 20:55:04 +09:00
Akira Ajisaka
69555fca06 HDFS-9476. TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail. Contributed by Masatake Iwasaki. 2016-06-02 18:52:47 +09:00
Kai Zheng
d749cf65e1 HDFS-9833. Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data. Contributed by Rakesh R. 2016-06-02 12:56:21 +08:00
Colin Patrick Mccabe
8ceb06e239 HADOOP-13137. TraceAdmin should support Kerberized cluster (Wei-Chiu Chuang via cmccabe) 2016-05-31 17:54:34 -07:00
Colin Patrick Mccabe
c7921c9bdd HDFS-9466. TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky (Wei-Chiu Chuang via cmccabe) 2016-05-31 17:37:52 -07:00
Colin Patrick Mccabe
29d6cadc52 HDFS-10415. TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called (Mingliang Liu via cmccabe) 2016-05-31 16:54:52 -07:00
Tsz-Wo Nicholas Sze
83f2f78c11 HADOOP-13226 Support async call retry and failover. 2016-05-31 16:30:11 -07:00
Kai Zheng
93d8a7f2a2 Revert "HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng" 
This reverts commit a6c79f92d5.
 2016-06-01 08:41:15 +08:00
Chris Nauroth
21890c4239 HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou. 2016-05-27 14:20:18 -07:00
Yongjun Zhang
5ea6fd85c7 HDFS-10276. HDFS should not expose path info that user has no permission to see. (Yuanbo Liu via Yongjun Zhang) 2016-05-27 10:02:02 -07:00
Kai Zheng
916140604f HADOOP-12911. Upgrade Hadoop MiniKDC with Kerby. Contributed by Jiajia Li 2016-05-28 14:23:39 +08:00
Zhe Zhang
8c84a2a93c HDFS-10236. Erasure Coding: Rename replication-based names in BlockManager to more generic [part-3]. Contributed by Rakesh R. 2016-05-26 16:50:55 -07:00
Karthik Kambatla
55c3e2de3d HDFS-10463. TestRollingFileSystemSinkWithHdfs needs some cleanup. (Daniel Templeton via kasha) 2016-05-26 14:24:31 -07:00
Tsz-Wo Nicholas Sze
f4b9bcd87c HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou 2016-05-26 12:15:17 -07:00
Kai Zheng
77202fa103 HADOOP-13010. Refactor raw erasure coders. Contributed by Kai Zheng 2016-05-27 13:23:34 +08:00
Kai Zheng
f69f5ab3b6 HDFS-10434. Fix intermittent test failure of TestDataNodeErasureCodingMetrics. Contributed by Rakesh R 2016-05-27 04:42:53 +08:00
Kai Zheng
a6c79f92d5 HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng 2016-05-27 04:29:42 +08:00
Vinayakumar B
2dab64da2f HDFS-10217. show 'blockScheduled' tooltip in datanodes table. 2016-05-25 12:34:52 +05:30
Tsz-Wo Nicholas Sze
02d4e478a3 HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-24 14:51:27 -07:00
Tsz-Wo Nicholas Sze
15ed080e36 HDFS-9365. Balaner does not work with the HDFS-6376 HA setup. 2016-05-24 12:49:48 -07:00
Karthik Kambatla
57c31a3fef HDFS-9782. RollingFileSystemSink should have configurable roll interval. (Daniel Templeton via kasha) 2016-05-24 10:37:00 -07:00
Allen Wittenauer
4b0f55b6ea HDFS-7766. Add a flag to WebHDFS op=CREATE to not respond with a 307 redirect (Ravi Prakash via aw) 2016-05-23 15:52:01 -07:00
Masatake Iwasaki
12eb5a7b11 HDFS-10439. Update setOwner doc in HdfsPermissionsGuide. Contributed by John Zhuge. 2016-05-22 16:53:30 +09:00
Chris Nauroth
500e946729 HDFS-10438. When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server. Contributed by Chris Nauroth. 2016-05-20 13:01:43 -07:00
Chris Nauroth
bcde1562d2 HDFS-10424. DatanodeLifelineProtocol not able to use under security cluster. Contributed by Chris Nauroth. 2016-05-20 12:47:41 -07:00
Tsz-Wo Nicholas Sze
42c22f7e3d HADOOP-13168. Support Future.get with timeout in ipc async calls. 2016-05-19 15:34:04 -07:00
Kihwal Lee
204de430bf HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 2016-05-19 15:52:32 -05:00
Andrew Wang
010e6ac328 HDFS-2173. saveNamespace should not throw IOE when only one storage directory fails to write VERSION file. Contributed by Andras Bokor. 2016-05-18 14:13:41 -07:00
Lei Xu
cf552aa87b HDFS-10360. DataNode may format directory and lose blocks if current/VERSION is missing. (Wei-Chiu Chuang via lei) 2016-05-18 09:38:37 -07:00
Arpit Agarwal
f4d8fde822 HDFS-9926. MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils. (Contributed by Josh Elser) 2016-05-18 09:20:57 -07:00
Xiaoyu Yao
dd99f5fc79 HDFS-10383. Safely close resources in DFSTestUtil. Contributed by Mingliang Liu. 2016-05-17 16:12:00 -07:00
Yongjun Zhang
16c07cc68a HDFS-10381, DataStreamer DataNode exclusion log message should be warning. (John Zhuge via Yongjun Zhang) 2016-05-17 15:57:33 -07:00
Junping Du
f6ef876fe1 MAPREDUCE-6657. Job history server can fail on startup when NameNode is in start phase. Contributed by Haibo Chen. 2016-05-17 14:41:47 -07:00
Yongjun Zhang
e24fe2641b HDFS-9732, Improve DelegationTokenIdentifier.toString() for better logging. Contributed by Yongjun Zhang. 2016-05-17 14:03:08 -07:00
Kihwal Lee
1356cbe994 Revert "HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu." 
This reverts commit 0942954e8a.
 2016-05-17 15:20:03 -05:00
Kihwal Lee
0942954e8a HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 2016-05-17 15:06:25 -05:00
Colin Patrick Mccabe
7cd5ae62f6 HDFS-10404. Fix formatting of CacheAdmin command usage help text (Yiqun Lin via cmccabe) 2016-05-17 11:10:48 -07:00
Kihwal Lee
4a5819dae2 HDFS-10303. DataStreamer#ResponseProcessor calculates packet ack latency incorrectly. Contributed by Surendra Singh Lilhore. 2016-05-17 08:53:31 -05:00
Akira Ajisaka
9fe5828f05 HDFS-10242. Cannot create space quota of zero. Contributed by Takashi Ohnishi. 2016-05-17 15:53:09 +09:00
Sangjin Lee
61f46be071 HDFS-10208. Addendum for HDFS-9579: to handle the case when client machine can't resolve network path (Ming Ma via sjlee) 2016-05-16 18:49:47 -07:00
Allen Wittenauer
730bc746f9 HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw) 
This commit contains the following JIRA issues:

    HADOOP-12931. bin/hadoop work for dynamic subcommands
    HADOOP-12932. bin/yarn work for dynamic subcommands
    HADOOP-12933. bin/hdfs work for dynamic subcommands
    HADOOP-12934. bin/mapred work for dynamic subcommands
    HADOOP-12935. API documentation for dynamic subcommands
    HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
    HADOOP-13086. enable daemonization of dynamic commands
    HADOOP-13087. env var doc update for dynamic commands
    HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
    HADOOP-13089. hadoop distcp adds client opts twice when dynamic
    HADOOP-13094. hadoop-common unit tests for dynamic commands
    HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
    HADOOP-13107. clean up how rumen is executed
    HADOOP-13108. dynamic subcommands need a way to manipulate arguments
    HADOOP-13110. add a streaming subcommand to mapred
    HADOOP-13111. convert hadoop gridmix to be dynamic
    HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
    HADOOP-13117. clarify daemonization and security vars for dynamic commands
    HADOOP-13120. add a --debug message when dynamic commands have been used
    HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
    HADOOP-13129. fix typo in dynamic subcommand docs
    HADOOP-13151. Underscores should be escaped in dynamic subcommands document
    HADOOP-13153. fix typo in debug statement for dynamic subcommands
 2016-05-16 17:54:45 -07:00
Lei Xu
6a6e74acf5 HDFS-10410. RedundantEditLogInputStream.LOG is set to wrong class. (John Zhuge via lei) 2016-05-16 17:05:46 -07:00
Kai Zheng
ad9441122f HDFS-8449. Add tasks count metrics to datanode for ECWorker. Contributed by Bo Li. 2016-05-15 23:39:09 -07:00
Andrew Wang
45788204ae HDFS-10333. Intermittent org.apache.hadoop.hdfs.TestFileAppend failure in trunk. Contributed by Yiqun Lin. 2016-05-15 22:15:05 -07:00
Andrew Wang
3c5c57af28 HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1. 2016-05-12 18:27:28 -07:00