big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
10,878 Commits 2 Branches 2 Tags 264 MiB
def9136e02
Commit Graph

2443 Commits

Author SHA1 Message Date
Andrew Wang
c7d9ad68e3 HDFS-8191. Fix byte to integer casting in SimulatedFSDataset#simulatedByte. Contributed by Zhe Zhang. 2015-04-24 11:54:25 -07:00
Colin Patrick Mccabe
a8898445dc HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe) 2015-04-23 19:13:27 -07:00
Colin Patrick Mccabe
a0e0a63209 HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm (cmccabe) 2015-04-23 19:00:24 -07:00
Haohui Mai
bcf89ddc7d HDFS-8052. Move WebHdfsFileSystem into hadoop-hdfs-client. Contributed by Haohui Mai. 2015-04-23 17:33:45 -07:00
Tsz-Wo Nicholas Sze
416b84354e HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode. Contributed by surendra singh lilhore 2015-04-23 11:55:06 -07:00
Andrew Wang
26971e52ae HADOOP-11852. Disable symlinks in trunk. 2015-04-23 11:47:01 -07:00
Haohui Mai
6f8003dc7b HDFS-8185. Separate client related routines in HAUtil into a new class. Contributed by Haohui Mai. 2015-04-21 21:59:47 -07:00
Colin Patrick Mccabe
997408eaac HDFS-8133. Improve readability of deleted block check (Daryn Sharp via Colin P. McCabe) 2015-04-21 11:43:24 -07:00
Arpit Agarwal
dfc1c4c303 HDFS-8163. Using monotonicNow for block report scheduling causes test failures on recently restarted systems. (Arpit Agarwal) 2015-04-21 10:58:05 -07:00
Vinayakumar B
8ddbb8dd43 HDFS-7993. Provide each Replica details in fsck (Contributed by J.Andreina) 2015-04-21 15:24:49 +05:30
Arpit Agarwal
c92f6f3605 HDFS-8179. DFSClient#getServerDefaults returns null within 1 hour of system start. (Contributed by Xiaoyu Yao) 2015-04-20 15:42:42 -07:00
Haohui Mai
5c97db07fb HDFS-8169. Move LocatedBlocks and related classes to hdfs-client. Contributed by Haohui Mai. 2015-04-20 00:36:46 -07:00
Arpit Agarwal
8511d80804 HDFS-8152. Refactoring of lazy persist storage cases. (Arpit Agarwal) 2015-04-19 16:09:06 -07:00
Tsuyoshi Ozawa
5459b241c8 HDFS-8043. NPE in MiniDFSCluster teardown. Contributed by Brahma Reddy Battula. 2015-04-19 18:19:36 +09:00
Haohui Mai
76e7264e8d HDFS-8165. Move GRANDFATHER_GENERATION_STAMP and GRANDFATER_INODE_ID to hdfs-client. Contributed by Haohui Mai. 2015-04-16 22:49:30 -07:00
Jitendra Pandey
369ddc67bd HDFS-8153. Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. 2015-04-16 22:13:09 -07:00
Tsz-Wo Nicholas Sze
9595cc003c HDFS-7891. A block placement policy with best rack failure tolerance. Contributed by Walter Su 2015-04-16 18:25:53 -07:00
Tsz-Wo Nicholas Sze
75bbcc8bf3 HDFS-8082. Move dfs.client.read.*, dfs.client.short.circuit.*, dfs.client.mmap.* and dfs.client.hedged.read.* conf from DFSConfigKeys to HdfsClientConfigKeys. 2015-04-16 13:22:31 -07:00
Andrew Wang
2e8ea780a4 HDFS-8142. DistributedFileSystem encryption zone commands should resolve relative paths. Contributed by Rakesh R. 2015-04-16 07:14:44 -07:00
Arpit Agarwal
9e8309a1b2 HDFS-8144. Split TestLazyPersistFiles into multiple tests. (Arpit Agarwal) 2015-04-15 10:25:04 -07:00
Jing Zhao
fddd55279d HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Contributed by Jing Zhao. 2015-04-14 15:05:23 -07:00
cnauroth
fef596df03 HDFS-8055. NullPointerException when topology script is missing. Contributed by Anu Engineer. 2015-04-14 10:19:30 -07:00
cnauroth
d45aa7647b HDFS-6666. Abort NameNode and DataNode startup if security is enabled but block access token is not enabled. Contributed by Vijay Bhat. 2015-04-14 09:59:01 -07:00
Andrew Wang
d60e22152a HDFS-8117. More accurate verification in SimulatedFSDataset: replace DEFAULT_DATABYTE with patterned data. Contributed by Zhe Zhang. 2015-04-13 13:01:10 -07:00
Tsz-Wo Nicholas Sze
7fc50e2525 HDFS-8083. Move dfs.client.write.* conf from DFSConfigKeys to HdfsClientConfigKeys.Write. 2015-04-13 11:43:35 -07:00
cnauroth
f8f5887209 HDFS-7933. fsck should also report decommissioning replicas. Contributed by Xiaoyu Yao. 2015-04-11 13:23:18 -07:00
Tsz-Wo Nicholas Sze
60da0e49e7 HDFS-8084. Move dfs.client.failover.* confs from DFSConfigKeys to HdfsClientConfigKeys.Failover and fix typos in the dfs.http.client.* configuration keys. 2015-04-10 19:38:43 -07:00
Haohui Mai
36e4cd3be6 HDFS-8103. Move BlockTokenSecretManager.AccessMode into BlockTokenIdentifier. Contributed by Haohui Mai. 2015-04-10 16:36:45 -07:00
Tsz-Wo Nicholas Sze
2cc9514ad6 HDFS-8100. Refactor DFSClient.Conf to a standalone class and separates short-circuit related conf to ShortCircuitConf. 2015-04-10 14:48:45 -07:00
Konstantin V Shvachko
0959b67f1a HDFS-8081. Split getAdditionalBlock() into two methods. Contributed by Konstantin Shvachko 2015-04-09 22:00:20 -07:00
Haohui Mai
af9d4fede5 HDFS-8102. Separate webhdfs retry configuration keys from DFSConfigKeys. Contributed by Haohui Mai. 2015-04-09 14:36:27 -07:00
Arun Suresh
922b7ed21d HDFS-8091: ACLStatus and XAttributes should be presented to INodeAttributesProvider before returning to client (asuresh) 2015-04-09 12:38:33 -07:00
Colin Patrick Mccabe
61dc2ea3fe HDFS-8063: Fix intermittent test failures in TestTracing (Masatake Iwasaki via Colin P. McCabe) 2015-04-09 11:28:02 -07:00
Aaron T. Myers
3fe61e0bb0 HDFS-8101. DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS classes at runtime. Contributed by Sean Busbey. 2015-04-09 09:40:08 -07:00
Vinayakumar B
9d8952f97f HDFS-8096. DatanodeMetrics#blocksReplicated will get incremented early and even for failed transfers (Contributed by Vinayakumar B) 2015-04-09 11:58:00 +05:30
Andrew Wang
5a540c3d31 HDFS-8025. Addendum fix for HDFS-3087 Decomissioning on NN restart can complete without blocks being replicated. Contributed by Ming Ma. 2015-04-08 16:09:17 -07:00
Haohui Mai
82d56b337d Revert HDFS-7813. 2015-04-08 16:02:45 -07:00
Haohui Mai
bd4c99bece Revert HDFS-7808. 2015-04-08 15:59:55 -07:00
Andrew Wang
6af0d74a75 HDFS-7725. Incorrect 'nodes in service' metrics caused all writes to fail. Contributed by Ming Ma. 2015-04-08 15:52:06 -07:00
Tsz-Wo Nicholas Sze
c931a3c776 HDFS-8079. Move CorruptFileBlockIterator to a new hdfs.client.impl package. 2015-04-08 11:50:52 -07:00
Arpit Agarwal
608c499841 HDFS-8072. Reserved RBW space is not released if client terminates while writing block. (Arpit Agarwal) 2015-04-08 11:38:21 -07:00
Haohui Mai
ab04ff9efe HDFS-8080. Separate JSON related routines used by WebHdfsFileSystem to a package local class. Contributed by Haohui Mai. 2015-04-07 21:30:57 -07:00
Tsz-Wo Nicholas Sze
4be648b55c HDFS-8079. Move dfs.client.retry.* confs from DFSConfigKeys to HdfsClientConfigKeys.Retry. 2015-04-07 19:48:57 -07:00
Konstantin V Shvachko
75c5454860 HDFS-8071. Redundant checkFileProgress() in PART II of getAdditionalBlock(). Contributed by Konstantin Shvachko. 2015-04-06 22:20:44 -07:00
Colin Patrick Mccabe
ef591b1d6a HDFS-8051. FsVolumeList#addVolume should release volume reference if not put it into BlockScanner. (Lei (Eddy) Xu via Colin P. McCabe) 2015-04-03 16:34:23 -07:00
Colin Patrick Mccabe
023133cef9 HDFS-7996. After swapping a volume, BlockReceiver reports ReplicaNotFoundException (Lei (Eddy) Xu via Colin P. McCabe) 2015-04-03 14:19:46 -07:00
cnauroth
db80e42891 HDFS-8039. Fix TestDebugAdmin#testRecoverLease and testVerfiyBlockChecksumCommand on Windows. Contributed by Xiaoyu Yao. 2015-04-03 10:44:02 -07:00
cnauroth
dc5596c36a HDFS-8034. Fix TestDFSClientRetries#testDFSClientConfigurationLocateFollowingBlockInitialDelay for Windows. Contributed by Xiaoyu Yao. 2015-04-03 10:10:11 -07:00
Tsz-Wo Nicholas Sze
4f412f487c HDFS-7954. TestBalancer#testBalancerWithPinnedBlocks should not be executed on Windows. Contributed by Xiaoyu Yao 2015-04-02 11:11:34 -07:00
Haohui Mai
6ccf4fbf8a HDFS-8008. Support client-side back off when the datanodes are congested. Contributed by Haohui Mai. 2015-04-01 16:54:46 -07:00
Colin Patrick Mccabe
3c7adaaf35 HDFS-7922. ShortCircuitCache#close is not releasing ScheduledThreadPoolExecutors (Rakesh R via Colin P. McCabe) 2015-04-01 16:02:39 -07:00
Colin Patrick Mccabe
c94d594a57 HDFS-8026. Trace FSOutputSummer#writeChecksumChunks rather than DFSOutputStream#writeChunk (cmccabe) 2015-04-01 14:10:10 -07:00
Haohui Mai
796fb26871 Add the missing files for HDFS-8009. 2015-04-01 10:58:32 -07:00
Akira Ajisaka
18a91fe4df HDFS-6945. BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed. (aajisaka) 2015-04-01 09:07:28 +09:00
yliu
3df61f303a HDFS-7997. The first non-existing xattr should also throw IOException. (zhouyingchao via yliu) 2015-03-31 21:17:44 +08:00
Arpit Agarwal
85dc3c14b2 HDFS-7944. Minor cleanup of BlockPoolManager#getAllNamenodeThreads. (Arpit Agarwal) 2015-03-30 21:24:51 -07:00
Arpit Agarwal
1a495fbb48 HDFS-7645. Rolling upgrade is restoring blocks from trash multiple times (Contributed by Vinayakumar B and Keisuke Ogiwara) 2015-03-30 15:25:16 -07:00
Haohui Mai
0967b1d99d Addendum for HDFS-7748. 2015-03-30 13:11:09 -07:00
Haohui Mai
b80457158d HDFS-7748. Separate ECN flags from the Status in the DataTransferPipelineAck. Contributed by Anu Engineer and Haohui Mai. 2015-03-30 11:59:21 -07:00
Kihwal Lee
04ee18ed48 HDFS-7742. Favoring decommissioning node for replication can cause a block to stay 
underreplicated for long periods. Contributed by Nathan Roberts.
 2015-03-30 10:10:11 -05:00
Harsh J
3d9132d434 HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. 2015-03-29 00:45:01 +05:30
Kihwal Lee
60882ab26d HDFS-7990. IBR delete ack should not be delayed. Contributed by Daryn Sharp. 2015-03-27 09:05:17 -05:00
Colin Patrick Mccabe
61df1b27a7 HDFS-7410. Support CreateFlags with append() to support hsync() for appending streams (Vinayakumar B via Colin P. McCabe) 2015-03-26 13:21:09 -07:00
Arpit Agarwal
3d0708bdb0 HDFS-7824. GetContentSummary API and its namenode implementation for Storage Type Quota/Usage. (Contributed by Xiaoyu Yao) 2015-03-26 10:24:11 -07:00
Kihwal Lee
222845632b HDFS-7963. Fix expected tracing spans in TestTracing. Contributed by Masatake Iwasaki. 2015-03-26 08:44:58 -05:00
Kihwal Lee
fc1031af74 HDFS-7928. Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy. Contributed by Rushabh Shah. 2015-03-25 14:42:59 -05:00
Jing Zhao
5e21e4ca37 HDFS-6353. Check and make checkpoint before stopping the NameNode. Contributed by Jing Zhao. 2015-03-25 10:38:00 -07:00
Haohui Mai
80278a5f85 HDFS-7985. WebHDFS should be always enabled. Contributed by Li Lu. 2015-03-24 21:55:56 -07:00
Jitendra Pandey
53a28afe29 HDFS-6826. Plugin interface to enable delegation of HDFS authorization assertions. Contributed by Arun Suresh. 2015-03-24 16:02:40 -07:00
Jing Zhao
a16bfff71b HDFS-7854. Separate class DataStreamer out of DFSOutputStream. Contributed by Li Bo. 2015-03-24 11:06:13 -07:00
Andrew Wang
6413d34986 HDFS-7961. Trigger full block report after hot swapping disk. Contributed by Eddy Xu. 2015-03-24 09:07:02 -07:00
Vinayakumar B
c6c396fcd6 HDFS-3325. When configuring 'dfs.namenode.safemode.threshold-pct' to a value greater or equal to 1 there is mismatch in the UI report (Contributed by J.Andreina) 2015-03-24 12:12:01 +05:30
Andrew Wang
50ee8f4e67 HDFS-7960. The full block report should prune zombie storages even if they're not empty. Contributed by Colin McCabe and Eddy Xu. 2015-03-23 22:00:34 -07:00
cnauroth
2c238ae4e0 HDFS-7917. Use file to replace data dirs in test to simulate a disk failure. Contributed by Lei (Eddy) Xu. 2015-03-23 16:29:51 -07:00
Kihwal Lee
75ead273be HDFS-6841. Use Time.monotonicNow() wherever applicable instead of Time.now(). Contributed by Vinayakumar B 2015-03-20 14:02:09 -05:00
Jing Zhao
d368d3647a HDFS-7957. Truncate should verify quota before making changes. Contributed by Jing Zhao. 2015-03-20 11:50:24 -07:00
Jing Zhao
a6a5aae472 HDFS-7829. Code clean up for LocatedBlock. Contributed by Takanobu Asanuma. 2015-03-20 10:50:19 -07:00
Yongjun Zhang
15612313f5 HDFS-7835. make initial sleeptime in locateFollowingBlock configurable for DFSClient. Contributed by Zhihai Xu. 2015-03-20 09:12:42 -07:00
yliu
e37ca221bf HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) 2015-03-19 23:23:19 +08:00
Kihwal Lee
e79be0ee12 HDFS-7816. Unable to open webhdfs paths with "+". Contributed by Haohui Mai 2015-03-19 08:01:01 -05:00
Jing Zhao
c7c71cdba5 HDFS-7587. Edit log corruption can happen if append fails with a quota violation. Contributed by Jing Zhao. 2015-03-18 18:53:50 -07:00
Colin Patrick Mccabe
43b41f2241 HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade starts (Zhe Zhang via Colin P. McCabe) 2015-03-18 18:48:54 -07:00
Jing Zhao
bee5a6a64a HDFS-7943. Append cannot handle the last block with length greater than the preferred block size. Contributed by Jing Zhao. 2015-03-18 18:40:59 -07:00
Colin Patrick Mccabe
8234fd0e10 HDFS-7054. Make DFSOutputStream tracing more fine-grained (cmccabe) 2015-03-18 18:14:54 -07:00
Haohui Mai
8c40e88d5d HDFS-7945. The WebHdfs system on DN does not honor the length parameter. Contributed by Haohui Mai. 2015-03-18 16:19:46 -07:00
Arpit Agarwal
c239b6d376 HDFS-7914. TestJournalNode#testFailToStartWithBadConfig fails when the default dfs.journalnode.http-address port 8480 is in use. (Contributed by Xiaoyu Yao) 2015-03-18 14:04:53 -07:00
Arpit Agarwal
e817cedcdc HDFS-7951. Fix NPE for TestFsDatasetImpl#testAddVolumeFailureReleasesInUseLock on Linux. (Contributed by Xiaoyu Yao) 2015-03-18 12:45:46 -07:00
Arpit Agarwal
d462c62755 HDFS-7950. Fix TestFsDatasetImpl#testAddVolumes failure on Windows. (Contributed by Xiaoyu Yao) 2015-03-18 12:35:30 -07:00
Arpit Agarwal
fdd58aa32f HDFS-7948. TestDataNodeHotSwapVolumes#testAddVolumeFailures failed on Windows. (Contributed by Xiaoyu Yao) 2015-03-18 12:26:00 -07:00
Arpit Agarwal
5b322c6a82 HDFS-7946. TestDataNodeVolumeFailureReporting NPE on Windows. (Contributed by Xiaoyu Yao) 2015-03-17 21:34:31 -07:00
Vinayakumar B
018893e81e HDFS-5356. MiniDFSCluster should close all open FileSystems when shutdown() (Contributed by Rakesh R) 2015-03-17 15:32:34 +05:30
Konstantin V Shvachko
ce5de93a58 HDFS-7886. Fix TestFileTruncate falures. Contributed by Plamen Jeliazkov and Konstantin Shvachko. 2015-03-16 12:54:04 -07:00
Colin Patrick Mccabe
bc9cb3e271 HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) 2015-03-14 22:36:46 -07:00
Colin Patrick Mccabe
32741cf3d2 Revert "HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) 
This reverts commit 5aa892ed48.
 2015-03-13 18:40:20 -07:00
Colin Patrick Mccabe
5aa892ed48 HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) 2015-03-13 18:29:49 -07:00
Konstantin V Shvachko
6acb7f2110 HDFS-7903. Cannot recover block after truncate and delete snapshot. Contributed by Plamen Jeliazkov. 2015-03-13 13:12:51 -07:00
Kihwal Lee
d324164a51 HDFS-7435. PB encoding of block reports is very inefficient. Contributed by Daryn Sharp. 2015-03-13 14:23:37 -05:00
Brandon Li
f446669afb HDFS-7926. NameNode implementation of ClientProtocol.truncate(..) is not idempotent. Contributed by Tsz Wo Nicholas Sze 2015-03-13 10:42:22 -07:00
Colin Patrick Mccabe
b49c3a1813 HDFS-7722. DataNode#checkDiskError should also remove Storage when error is found. (Lei Xu via Colin P. McCabe) 2015-03-12 12:00:18 -07:00
Tsz-Wo Nicholas Sze
6dae6d12ec HDFS-6833. DirectoryScanner should not register a deleting block with memory of DataNode. Contributed by Shinichi Yamashita 2015-03-13 02:25:32 +08:00