big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9,018 Commits 2 Branches 2 Tags 264 MiB
2a51494ce1
Commit Graph

3118 Commits

Author SHA1 Message Date
arp
e9edafc731 Merge branch 'trunk' into HDFS-6581 2014-09-25 11:06:25 -07:00
cnauroth
e0b1dc514f HDFS-7118. Improve diagnostics on storage directory rename operations by using NativeIO#renameTo in Storage#rename. Contributed by Chris Nauroth. 2014-09-25 10:03:59 -07:00
arp
e8e7fbe81a HDFS-6932. Balancer and Mover tools should ignore replicas on RAM_DISK. (Contributed by Xiaoyu Yao) 2014-09-24 21:08:59 -07:00
arp
feda4733a8 HDFS-7143. Fix findbugs warnings in HDFS-6581 branch. (Contributed by Tsz Wo Nicholas Sze) 2014-09-24 21:06:56 -07:00
arp
b1000fbba4 Merge branch 'trunk' into HDFS-6581 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
 2014-09-24 20:08:32 -07:00
Jing Zhao
428a76663a HDFS-7140. Add a tool to list all the existing block storage policies. Contributed by Jing Zhao. 2014-09-24 19:11:16 -07:00
Andrew Wang
63198bfba7 HDFS-7139. Unit test for creating encryption zone on root path. (Zhe Zhang via wang) 2014-09-24 16:40:20 -07:00
Jing Zhao
073bbd805c HDFS-7081. Add new DistributedFileSystem API for getting all the existing storage policies. Contributed by Jing Zhao. 2014-09-24 10:05:40 -07:00
Allen Wittenauer
cb74f39697 HDFS-6534. Fix build on macosx: HDFS parts (Binglin Chang via aw) 2014-09-24 08:22:02 -07:00
arp
bfc7b7e6a0 Merge branch 'trunk' into HDFS-6581 2014-09-23 16:40:35 -07:00
cnauroth
b93d9603a2 HDFS-7130. TestDataTransferKeepalive fails intermittently on Windows. Contributed by Chris Nauroth. 2014-09-23 14:35:33 -07:00
Andrew Wang
f48686a1ad HDFS-7132. hdfs namenode -metadataVersion command does not honor configured name dirs. Contributed by Charles Lamb. 2014-09-23 14:10:02 -07:00
Allen Wittenauer
3dc28e2052 HADOOP-11092. hadoop shell commands should print usage if not given a class (aw) 2014-09-23 12:24:23 -07:00
cnauroth
a1fd804a31 HDFS-7126. TestEncryptionZonesWithHA assumes Unix path separator for KMS key store path. Contributed by Xiaoyu Yao. 2014-09-23 08:21:46 -07:00
arp
8b139b0800 HDFS-6990. Add unit test for evict/delete RAM_DISK block with open handle. (Contributed by Xiaoyu Yao) 2014-09-22 18:10:28 -07:00
arp
889122bd55 Merge branch 'trunk' into HDFS-6581 2014-09-22 18:00:58 -07:00
Colin Patrick Mccabe
7b8df93ce1 HDFS-7001. Tests in TestTracing should not depend on the order of execution. (iwasakims via cmccabe) 2014-09-22 17:21:58 -07:00
Colin Patrick Mccabe
912ad32b03 HDFS-7106. Reconfiguring DataNode volumes does not release the lock files in removed volumes. (cnauroth via cmccabe) 2014-09-22 11:51:31 -07:00
cnauroth
26cba7f35f HDFS-7115. TestEncryptionZones assumes Unix path separator for KMS key store path. Contributed by Xiaoyu Yao. 2014-09-22 09:07:07 -07:00
Andrew Wang
1737950d0f HDFS-6987. Move CipherSuite xattr information up to the encryption zone root. Contributed by Zhe Zhang. 2014-09-21 21:29:46 -07:00
cnauroth
b6fd5ccbce HDFS-7110. Skip tests related to short-circuit read on platforms that do not currently implement short-circuit read. Contributed by Chris Nauroth. 2014-09-21 12:36:57 -07:00
cnauroth
0795fe3721 HDFS-7109. TestDataStorage does not release file locks between tests. Contributed by Chris Nauroth. 2014-09-20 21:01:13 -07:00
cnauroth
47491f9a11 HDFS-7107. Avoid Findbugs warning for synchronization on AbstractNNFailoverProxyProvider#fallbackToSimpleAuth. Contributed by Chris Nauroth. 2014-09-20 20:53:09 -07:00
arp
8c9860f7c9 HDFS-7071. Undo accidental commit of binary file editsStored. 2014-09-20 17:11:11 -07:00
arp
486a76a39b HDFS-7071. Updated editsStored and editsStored.xml to bump layout version and add LazyPersist flag. (Contributed by Xiaoyu Yao and Arpit Agarwal) 2014-09-20 16:58:58 -07:00
arp
e257b6dbb1 Merge branch 'trunk' into HDFS-6581 2014-09-20 16:13:17 -07:00
arp
50b321068d HDFS-7108. Fix unit test failures in SimulatedFsDataset. (Arpit Agarwal) 2014-09-20 16:12:17 -07:00
Jing Zhao
84a0a629d3 HDFS-7095. Archival Storage: TestStorageMover often fails in Jenkins. Contributed by Jing Zhao. 2014-09-20 13:44:53 -07:00
arp
b2d5ed36bc HDFS-7100. Make eviction scheme pluggable. (Arpit Agarwal) 2014-09-20 13:25:23 -07:00
arp
09dab88d3e Merge branch 'trunk' into HDFS-6581 2014-09-20 10:34:45 -07:00
cnauroth
f85cc14eb4 HDFS-7073. Allow falling back to a non-SASL connection on DataTransferProtocol in several edge cases. Contributed by Chris Nauroth. 2014-09-19 21:23:26 -07:00
cnauroth
332e2e23ba HDFS-7105. Fix TestJournalNode#testFailToStartWithBadConfig to match log output change. Contributed by Ray Chiang. 2014-09-19 21:04:55 -07:00
Kihwal Lee
9e35571965 HDFS-7046. HA NN can NPE upon transition to active. Contributed by 
Kihwal Lee.
 2014-09-19 17:07:06 -05:00
arp
222bf0fe67 Merge branch 'trunk' into HDFS-6581 2014-09-19 10:02:27 -07:00
Kihwal Lee
bf27b9ca57 HDFS-7065. Pipeline close recovery race can cause block corruption. 
Contributed by Kihwal Lee.
 2014-09-19 09:00:13 -05:00
Kihwal Lee
f02d934fed HDFS-6948. DN rejects blocks if it has older UC block. Contributed by 
Eric Payne.
 2014-09-19 08:50:43 -05:00
arp
f8bbf80067 Merge branch 'trunk' into HDFS-6581 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsLocatedFileStatus.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeLayoutVersion.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
 2014-09-18 22:26:42 -07:00
arp
e79c98c11f HDFS-7091. Add forwarding constructor for INodeFile for existing callers. (Arpit Agarwal) 2014-09-18 19:20:25 -07:00
Andrew Wang
8e73084491 HDFS-6840. Clients are always sent to the same datanode when read is off rack. (wang) 2014-09-18 17:49:17 -07:00
Andrew Wang
20a076bafc HDFS-6970. Move startFile EDEK retries to the DFSClient. (wang) 2014-09-18 17:35:24 -07:00
Colin Patrick Mccabe
fe38d2e9b5 HDFS-6727. Refresh data volumes on DataNode based on configuration changes (Lei Xu via Colin Patrick McCabe) 2014-09-18 16:52:46 -07:00
Tsz-Wo Nicholas Sze
98f7bfe993 Merge branch 'HDFS-6584' into trunk 2014-09-19 07:11:19 +08:00
arp
22295b4783 Merge branch 'trunk' into HDFS-6581 2014-09-18 15:59:37 -07:00
Andrew Wang
70be56d093 HDFS-7003. Add NFS Gateway support for reading and writing to encryption zones. (clamb via wang) 2014-09-18 14:57:25 -07:00
arp
0b10e5a0f7 Merge branch 'trunk' into HDFS-6581 2014-09-18 12:10:45 -07:00
Colin Patrick Mccabe
570b8b468e HDFS-7047. Expose FileStatus#isEncrypted in libhdfs (cmccabe) 2014-09-18 11:42:23 -07:00
arp
5e4627d0fb HDFS-7084. FsDatasetImpl#copyBlockFiles debug log can be improved. (Contributed by Xiaoyu Yao) 2014-09-18 11:13:53 -07:00
arp
849ccfa699 Merge branch 'trunk' into HDFS-6581 2014-09-18 11:06:40 -07:00
Jing Zhao
76ccb80fad HDFS-7088. Archival Storage: fix TestBalancer and TestBalancerWithMultipleNameNodes. Contributed by Tsz Wo Nicholas Sze. 2014-09-18 10:54:01 -07:00
Tsz-Wo Nicholas Sze
2d2b0009e6 Merge branch 'trunk' into HDFS-6584 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
 2014-09-18 13:00:29 +08:00
Andrew Wang
0ecefe6017 HDFS-7078. Fix listEZs to work correctly with snapshots. (wang) 2014-09-17 21:28:05 -07:00
Andrew Wang
10e8602f32 HDFS-7004. Update KeyProvider instantiation to create by URI. (wang) 2014-09-17 20:14:40 -07:00
arp
4eab083b1b HDFS-7080. Fix finalize and upgrade unit test failures. (Arpit Agarwal) 2014-09-17 15:25:04 -07:00
arp
a186d514bd Merge branch 'trunk' into HDFS-6581 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
 2014-09-17 15:03:55 -07:00
Jing Zhao
911979c8ab Merge changes from trunk 2014-09-17 13:12:45 -07:00
Colin Patrick Mccabe
e3803d002c HDFS-6843. Create FileStatus isEncrypted() method (clamb via cmccabe) 2014-09-17 12:55:35 -07:00
arp
24f8156886 Merge branch 'trunk' into HDFS-6581 2014-09-17 12:10:50 -07:00
Andrew Wang
ea4e2e843e HDFS-6705. Create an XAttr that disallows the HDFS admin from accessing a file. (clamb via wang) 2014-09-17 11:23:47 -07:00
Jing Zhao
b014e83bc5 HDFS-6864. Archival Storage: add user documentation. Contributed by Tsz Wo Nicholas Sze. 2014-09-17 09:40:17 -07:00
Alejandro Abdelnur
e14e71d5fe HADOOP-11096. KMS: KeyAuthorizationKeyProvider should verify the keyversion belongs to the keyname on decrypt. (tucu) 2014-09-16 23:20:35 -07:00
Tsz-Wo Nicholas Sze
bd2e409de1 HDFS-7072. Fix TestBlockManager and TestStorageMover. Contributed by Jing Zhao 2014-09-17 13:48:32 +08:00
arp
a24a9331d4 Merge branch 'trunk' into HDFS-6581 2014-09-16 18:18:09 -07:00
Jing Zhao
432b863719 Merge changes from trunk 2014-09-16 17:58:39 -07:00
Jing Zhao
2e4033e872 HDFS-7052. Archival Storage: Add Mover into hdfs script. Contributed by Jing Zhao. 2014-09-16 17:52:28 -07:00
Andrew Wang
a044407b5c HDFS-6789. TestDFSClientFailover.testFileContextDoesntDnsResolveLogicalURI and TestDFSClientFailover.testDoesntDnsResolveLogicalURI failing on jdk7. Contributed by Akira Ajisaka. 2014-09-16 17:02:06 -07:00
Andrew Wang
33ce887c20 HDFS-6851. Refactor EncryptionZoneWithId and EncryptionZone. (clamb via wang) 2014-09-16 16:41:09 -07:00
Benoy Antony
02adf7185d HDFS-6799. The invalidate method in SimulatedFSDataset failed to remove (invalidate) blocks from the file system. Contributed by Megasthenis Asteris. 2014-09-16 16:24:23 -07:00
Allen Wittenauer
7971c97ec1 HADOOP-11022. User replaced functions get lost 2-3 levels deep (e.g., sbin) (aw) 2014-09-16 16:06:23 -07:00
arp
4603e4481f HDFS-7064. Fix unit test failures in HDFS-6581 branch. (Contributed by Xiaoyu Yao) 2014-09-16 15:45:07 -07:00
Alejandro Abdelnur
3e85f5b605 HDFS-7006. Test encryption zones with KMS. (Anthony Young-Garner and tucu) 2014-09-16 14:36:07 -07:00
arp
e0d7fb48fb Merge branch 'trunk' into HDFS-6581 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
 2014-09-16 14:32:57 -07:00
Colin Patrick Mccabe
56119fec96 HDFS-6880. Adding tracing to DataNode data transfer protocol (iwasakims via cmccabe) 2014-09-16 13:59:32 -07:00
Kihwal Lee
0c26412be4 HDFS-6965. NN continues to issue block locations for DNs with full 
disks. Contributed by Rushabh Shah.
 2014-09-16 09:04:54 -05:00
arp
a7814e1aa4 Merge branch 'trunk' into HDFS-6581 2014-09-15 15:30:19 -07:00
arp
a4dcbaa332 HDFS-7066. LazyWriter#evictBlocks misses a null check for replicaState. (Contributed by Xiaoyu Yao) 2014-09-15 15:28:17 -07:00
Kihwal Lee
2f847b3357 HDFS-7059. HAadmin transtionToActive with forceActive option can show 
confusing message. Contributed by Rushabh Shah.
 2014-09-15 16:27:22 -05:00
Jing Zhao
ece3ecabdf Merge remote-tracking branch 'origin/trunk' into HDFS-6584 2014-09-15 10:45:45 -07:00
Andrew Wang
43b0303008 HDFS-7032. Add WebHDFS support for reading and writing to encryption zones. Contributed by Charles Lamb. 2014-09-15 10:23:57 -07:00
Jing Zhao
2689b6ca72 HDFS-7062. Archival Storage: skip under construction block for migration. Contributed by Jing Zhao. 2014-09-15 10:16:56 -07:00
Andrew Wang
fc741b5d78 HDFS-7061. Add test to verify encryption zone creation after NameNode restart without saving namespace. Contributed by Stephen Chu. 2014-09-14 23:49:01 -07:00
Tsz-Wo Nicholas Sze
dba52ce0bb Merge branch 'trunk' into HDFS-6584 2014-09-13 14:56:48 +08:00
arp
9f22fb8c9a HDFS-6978. Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal) 2014-09-12 22:13:18 -07:00
arp
7aeda47bd9 Merge branch 'trunk' into HDFS-6581 2014-09-12 11:45:11 -07:00
cnauroth
d6ce26986e HDFS-7051. TestDataNodeRollingUpgrade#isBlockFileInPrevious assumes Unix file path separator. Contributed by Chris Nauroth. 2014-09-11 20:58:15 -07:00
Jing Zhao
dc27ee9ae9 HDFS-7039. Archival Storage: fix Balancer tests. Contributed by Tsz Wo Nicholas Sze. 2014-09-11 14:19:18 -07:00
Jing Zhao
8de20a1987 Merge trunk into HDFS-6584 2014-09-11 13:04:35 -07:00
Jing Zhao
0d85f7e591 HDFS-7034. Archival Storage: Fix TestBlockPlacement and TestStorageMover. Contributed by Jing Zhao. 2014-09-11 13:00:43 -07:00
cnauroth
80ac6aabce HDFS-7042. Upgrade fails for Windows HA cluster due to file locks held during rename in JournalNode. Contributed by Chris Nauroth. 2014-09-11 12:16:19 -07:00
Andrew Wang
1e684995d7 HDFS-7045. Fix NameNode deadlock when opening file under /.reserved path. Contributed by Yi Liu. 2014-09-11 10:59:09 -07:00
Andrew Wang
d1dab6cc63 HDFS-6621. Hadoop Balancer prematurely exits iterations. Contributed by Rafal Wodjdyla and Benjamin Bowman. 2014-09-11 10:32:20 -07:00
Andrew Wang
bf64fce78c HDFS-6966. Add additional unit tests for encryption zones. Contributed by Stephen Chu. 2014-09-11 10:26:13 -07:00
Tsz-Wo Nicholas Sze
70dfe9cfab Merge branch 'trunk' into HDFS-6584 2014-09-11 15:23:38 +08:00
Jing Zhao
db41a1b7b9 Merge changes from trunk 2014-09-09 22:47:50 -07:00
Alejandro Abdelnur
bbff44cb03 HDFS-6776. Using distcp to copy data between insecure and secure cluster via webdhfs doesn't work. (yzhangal via tucu) 2014-09-09 22:16:42 -07:00
Tsz-Wo Nicholas Sze
4124905aa5 HDFS-7028: FSDirectory should not get storage policy id from symlinks 2014-09-10 12:34:15 +08:00
arp
0d2f6b4600 Merge branch 'trunk' into HDFS-6581 2014-09-09 18:07:18 -07:00
Konstantin V Shvachko
05af0ff4be Revert HDFS-6940. 2014-09-09 17:30:10 -07:00
cnauroth
2749fc677c Newly moved block replica been invalidated and deleted in TestBalancer. Contributed by Binglin Chang. 2014-09-09 11:32:54 -07:00
arp
61ccd50362 Merge branch 'trunk' into HDFS-6581 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeLayoutVersion.java
 2014-09-08 22:44:04 -07:00
arp
f949f6b548 HDFS-6981. Fix DN upgrade with layout version change. (Arpit Agarwal) 2014-09-08 21:20:55 -07:00