big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
20,541 Commits 2 Branches 2 Tags 264 MiB
9dbb2b67c6
Commit Graph

1046 Commits

Author SHA1 Message Date
Yiqun Lin
1312f9ae4c HADOOP-15391. Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules. 2018-04-18 16:04:00 +08:00
Inigo Goiri
78860372bd YARN-8134. Support specifying node resources in SLS. Contributed by Abhishek Modi. 2018-04-17 13:11:34 -07:00
Aaron Fabbri
b0aff8a962
HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran. 2018-04-10 23:55:38 -07:00
Xiao Chen
7c9cdad6d0 HDFS-13056. Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo. 2018-04-10 21:31:48 -07:00
Aaron Fabbri
ea3849f0cc
HADOOP-14759 S3GuardTool prune to prune specific bucket entries. Contributed by Gabor Bota. 2018-04-05 20:23:17 -07:00
Steve Loughran
5a174f8ac6 HADOOP-14758. S3GuardTool.prune to handle UnsupportedOperationException. 
Contributed by Gabor Bota.
 2018-04-03 16:31:34 +01:00
Sammi Chen
6542d17ea4 HADOOP-14999. AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu. 2018-03-30 20:23:05 +08:00
Sean Mackrory
9d7a9031a5 HADOOP-15342. Updating ADLS connector to use the current SDK version (2.2.7). Contributed by Atul Sikaria. 2018-03-29 10:13:40 -06:00
Chris Douglas
081c350188 HADOOP-15320. Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake. Contributed by Shanyu Zhao 2018-03-28 11:58:59 -07:00
Haibo Chen
fcea5a4d79 YARN-7794. SLSRunner is not loading timeline service jars, causing failure. (Yufei Gu via Haibo Chen) 2018-03-26 14:30:11 -07:00
Sean Mackrory
82665a7887 HADOOP-15299. Bump Jackson 2 version to Jackson 2.9.x. 2018-03-26 14:05:15 -06:00
Sean Mackrory
7ce6b41509 HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota. 2018-03-20 21:12:20 -07:00
Sammi Chen
d67a5e2dec HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu. 2018-03-19 15:02:37 +08:00
Wangda Tan
f47659fb97 MAPREDUCE-7047. Make HAR tool support IndexedLogAggregtionController. (Xuan Gong via wangda) 
Change-Id: Ice5ae1c706f2476361997dcbb29f2c33c77d4f0c
 2018-03-15 13:26:01 -07:00
Steve Loughran
1976e0066e HADOOP-15209. DistCp to eliminate needless deletion of files under already-deleted directories. 
Contributed by Steve Loughran.
 2018-03-15 18:05:14 +00:00
fang zhenyi
5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-15 16:12:07 +09:00
Chris Douglas
45cccadd2e HDFS-12780. Fix spelling mistake in DistCpUtils.java. Contributed by Jianfei Jiang 2018-03-13 11:08:11 -07:00
Chris Douglas
45d1b0fdcc HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 2018-03-12 20:05:39 -07:00
Steve Loughran
dd05871b8b HADOOP-15297. Make S3A etag => checksum feature optional. 
Contributed by Steve Loughran.
 2018-03-12 14:01:42 +00:00
Steve Loughran
7ef4d942dd HADOOP-15273.distcp can't handle remote stores with different checksum algorithms. 
Contributed by Steve Loughran.
 2018-03-08 11:24:06 +00:00
Steve Loughran
3bd6b1fd85 HADOOP-15292. Distcp's use of pread is slowing it down. 
Contributed by Virajith Jalaparti.
 2018-03-08 11:15:46 +00:00
Steve Loughran
e0307e53e2 HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled. 
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d8d8544d715e67ee77cd3f017c21699)
 2018-03-07 18:51:10 +00:00
Steve Loughran
2e1e049bd0 HADOOP-15288. TestSwiftFileSystemBlockLocation doesn't compile. 
Contributed by Steve Loughran.
 2018-03-05 18:12:38 +00:00
Steve Loughran
8110d6a0d5 HADOOP-13761. S3Guard: implement retries for DDB failures and throttling; translate exceptions. 
Contributed by Aaron Fabbri.
 2018-03-05 14:06:20 +00:00
Gergely Novák
e99a1e7767
MAPREDUCE-7023. TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-02 16:15:04 -08:00
Ray Chiang
923e1773e1 HADOOP-14652. Update metrics-core version to 3.2.4. (rchiang) 2018-03-01 15:36:10 -08:00
fang zhenyi
29233c3c4a
HADOOP-15274. Move hadoop-openstack to slf4j 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-01 15:08:30 -08:00
Steve Loughran
edc9f1451b HADOOP-15263. hadoop cloud-storage module to mark hadoop-common as provided; add azure-datalake. 
Contributed by Steve Loughran
 2018-02-28 14:42:25 +00:00
Weiwei Yang
7af4f34de5 YARN-7929. Support to set container execution type in SLS. (Jiandan Yang via Weiwei Yang) 2018-02-28 17:58:09 +08:00
Arpit Agarwal
2fa7963c3d HADOOP-15254. Correct the wrong word spelling 'intialize'. Contributed by fang zhenyi. 2018-02-24 14:41:55 -08:00
Xuan Gong
68ce193efc MAPREDUCE-7027: HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails. Contributed by Gergely Novák 2018-02-23 14:37:26 -08:00
Carlo Curino
84cea0011f YARN-7732. Support Generic AM Simulator from SynthGenerator. (Contributed by Young Chen via curino) 2018-02-20 17:00:34 -08:00
fang zhenyi
4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-02-18 22:19:39 +09:00
Steve Loughran
7ac88244c5 HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. 
Contributed by Steve Loughran.
 2018-02-16 16:37:06 +00:00
Steve Loughran
481d79fedc HADOOP-13972. ADLS to support per-store configuration. 
Contributed by Sharad Sonker.

(cherry picked from commit 050f5287b79324b7f6231b879c0bfc608203b980)
 2018-02-15 16:27:31 +00:00
Steve Loughran
9a013b255f HADOOP-15176. Enhance IAM Assumed Role support in S3A client. 
Contributed by Steve Loughran

(cherry picked from commit 96c047fbb98c2378eed9693a724d4cbbd03c00fd)
 2018-02-15 15:57:10 +00:00
Steve Loughran
b27ab7dd81 HADOOP-15076. Enhance S3A troubleshooting documents and add a performance document. 
Contributed by Steve Loughran.

(cherry picked from commit c761e658f6594c4e519ed39ef36669de2c5cee15)
 2018-02-15 14:57:56 +00:00
Steve Loughran
c9a373fb14 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran.

(cherry picked from commit 58a2120e8a31307f19551f87be4e81d4fb626de1)
 2018-02-15 14:26:00 +00:00
Steve Loughran
f20dc0d577 HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions. 
Contributed by Andras Bokor.
 2018-02-14 16:20:14 +00:00
Steve Loughran
8cf88fcd1f HADOOP-15187. Remove ADL mock test dependency on REST call invoked from Java SDK. 
Contributed by Vishwajeet Dusane.

(cherry picked from commit bd5ab5912564d2d687651b01f552b8e4ca8c145a)
 2018-02-12 15:13:00 +00:00
Wangda Tan
60f9e60b3b Preparing for 3.2.0 development 
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
 2018-02-11 11:17:38 +08:00
Anu Engineer
4304fcd5bd HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 2018-02-06 13:43:45 -08:00
Arpit Agarwal
d4e13a4647 HADOOP-15198. Correct the spelling in CopyFilter.java. Contributed by Mukul Kumar Singh. 2018-02-02 11:37:51 -08:00
Sunil G
9b81cb0537 YARN-7779. Display allocation tags in RM web UI and expose same through REST API. Contributed by Weiwei Yang. 2018-01-31 01:30:17 -08:00
Arun Suresh
38af237969 YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Steve Loughran
7fd287b4af HADOOP-15186. Allow Azure Data Lake SDK dependency version to be set on the command line. 
Contributed by Vishwajeet Dusane.

(cherry picked from commit 95a96b13e2a54e01ea6c6933045d912998477da3)
 2018-01-29 09:48:14 -08:00
Jason Lowe
ff8378eb1b YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:47:19 -06:00
Yufei Gu
16be42d309 YARN-7798. Refactor SLS Reservation Creation. Contributed by Young Chen. 2018-01-25 13:06:01 -08:00
Steve Loughran
f63d13f10d HADOOP-15185. Update adls connector to use the current version of ADLS SDK. 
Contributed by Atul Sikaria.
 2018-01-23 09:09:11 -08:00
Steve Loughran
1093a73689 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits. 
Contributed by Aaron Fabbri
 2018-01-18 13:13:58 +00:00
Steve Loughran
f274fe33ea Revert "HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits." 
This reverts commit 35ad9b1dd2.
 2018-01-18 12:35:57 +00:00
Aaron Fabbri
268ab4e027
HADOOP-15141 Support IAM Assumed roles in S3A. Contributed by Steve Loughran. 2018-01-17 00:05:24 -08:00
Sammi Chen
9195a6e302 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 2018-01-17 15:55:59 +08:00
Steve Loughran
a0c71dcc33 HADOOP-15079. ITestS3AFileOperationCost#testFakeDirectoryDeletion failing 
after OutputCommitter patch.
Contributed by Steve Loughran
 2018-01-15 11:33:32 +00:00
Mukul Kumar Singh
2e1e9017aa
HADOOP-15172. Fix the javadoc warning in WriteOperationHelper.java 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-01-15 15:45:51 +09:00
Steve Loughran
1a09da7400 HADOOP-15163. Fix S3ACommitter documentation 
Contributed by Alessandro Andrioni.

(cherry picked from commit 100e8a1ae1d930dde084af7d1281e491c7f124ec)
 2018-01-10 15:37:07 +00:00
Steve Loughran
b62a5ece95 HADOOP-15161. s3a: Stream and common statistics missing from metrics 
Contributed by Sean Mackrory
 2018-01-09 18:46:52 +00:00
Steve Loughran
52babbb4a0 HADOOP-15086. NativeAzureFileSystem file rename is not atomic. 
Contributed by Thomas Marquardt
 2017-12-22 11:39:55 +00:00
Steve Loughran
c8ff0cc304 HADOOP-13282. S3 blob etags to be made visible in S3A status/getFileChecksum() calls. 
Contributed by Steve Loughran
 2017-12-21 14:58:58 +00:00
Steve Loughran
ef450df443 HADOOP-15113. NPE in S3A getFileStatus: null instrumentation on using closed instance. 
Contributed by Steve Loughran.
 2017-12-21 14:15:53 +00:00
Akira Ajisaka
5ab632baf5
HDFS-12949. Fix findbugs warning in ImageWriter.java. 2017-12-21 10:04:34 +09:00
Steve Loughran
1ba491ff90 HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive 2017-12-20 18:25:33 +00:00
Steve Loughran
35ad9b1dd2 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits. 
Contributed by Aaron Fabbri
 2017-12-18 21:19:06 +00:00
Chris Douglas
4b3a785914 HDFS-12903. [READ] Fix closing streams in ImageWriter. Contributed by Virajith Jalaparti 2017-12-15 17:51:42 -08:00
Chris Douglas
e515103a83 Revert "HDFS-12903. [READ] Fix closing streams in ImageWriter" 
This reverts commit c1bf2654b0e9118985b8518b0254eac4dd302a2f.
 2017-12-15 17:51:42 -08:00
Virajith Jalaparti
8239e3afb3 HDFS-12712. [9806] Code style cleanup 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
80c3fec3a1 HDFS-12912. [READ] Fix configuration and implementation of LevelDB-based alias maps 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
c89b29bd42 HDFS-12893. [READ] Support replication of Provided blocks with non-default topologies. 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
0f6aa9564c HDFS-12905. [READ] Handle decommissioning and under-maintenance Datanodes with Provided storage. 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
962b5e722b HDFS-12903. [READ] Fix closing streams in ImageWriter 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
4531588a94 HDFS-11640. [READ] Datanodes should use a unique identifier when reading from external stores 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
fb996a32a9 HDFS-12894. [READ] Skip setting block count of ProvidedDatanodeStorageInfo on DN registration update 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
9c35be86e1 HDFS-12713. [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata. Contributed by Ewan Higgs 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
a027055dd2 HDFS-12885. Add visibility/stability annotations. Contributed by Chris Douglas 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
352f994b64 HDFS-12665. [AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb). Contributed by Ewan Higgs. 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
4d59dabb7f HDFS-12809. [READ] Fix the randomized selection of locations in {{ProvidedBlocksBuilder}}. 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
3d3be87e30 HDFS-12778. [READ] Report multiple locations for PROVIDED blocks 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
3b1d30301b HDFS-12775. [READ] Fix reporting of Provided volumes 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
6cd80b2521 HDFS-12779. [READ] Allow cluster id to be specified to the Image generation tool 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
90d1b47a2a HDFS-12776. [READ] Increasing replication for PROVIDED files should create local replicas 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
87dc026bee HDFS-12789. [READ] Image generation tool does not close an opened stream 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
c293cc8e9b HDFS-12671. [READ] Test NameNode restarts when PROVIDED is configured 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
71d0a82571 HDFS-12607. [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
98f5ed5aa3 HDFS-11902. [READ] Merge BlockFormatProvider and FileRegionProvider. 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
d6a9a89973 HDFS-12605. [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
17052c4aff HDFS-12584. [READ] Fix errors in image generation tool from latest rebase 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
aca023b72c HDFS-12289. [READ] HDFS-12091 breaks the tests for provided block reads 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
546b95f484 HDFS-11673. [READ] Handle failures of Datanode with PROVIDED storage 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
4851f06bc2 HDFS-11791. [READ] Test for increasing replication of provided files. 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
aa5ec85f7f HDFS-11663. [READ] Fix NullPointerException in ProvidedBlocksBuilder 2017-12-15 17:51:37 -08:00
Virajith Jalaparti
d65df0f273 HDFS-11190. [READ] Namenode support for data stored in external stores. 2017-12-15 17:51:37 -08:00
Virajith Jalaparti
8da3a6e314 HDFS-10706. [READ] Add tool generating FSImage from external store 2017-12-15 17:51:37 -08:00
Kai Zheng
f86c81d923 HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu 2017-12-14 11:14:52 +08:00
Kai Zheng
28792b6b7f Revert "HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu" 
This reverts commit 479d6a5792262c977025c26fd4960574b0db6847
 2017-12-14 11:05:11 +08:00
Surendra Singh Lilhore
00129c5314 HDFS-12833. Distcp : Update the usage of delete option for dependency with update and overwrite option. Contributed by usharani. 2017-12-12 00:28:02 +05:30
Kai Zheng
ce04340ec7 HADOOP-15104. AliyunOSS: change the default value of max error retry. Contributed by Jinhu Wu 2017-12-08 23:03:54 +08:00
Kai Zheng
73b86979d6 HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu 2017-12-06 12:01:36 +08:00
Aaron Fabbri
6555af81a2
HADOOP-14475 Metrics of S3A don't print out when enabled. Contributed by Younger and Sean Mackrory. 2017-12-05 11:06:32 -08:00
Steve Loughran
3150c019ae HADOOP-15071 S3a troubleshooting docs to add a couple more failure modes. 
Contributed by Steve Loughran
 2017-12-05 15:05:41 +00:00
Steve Loughran
0ea182d0fa HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0. 
Contributed by Rajesh Balamohan
 2017-11-28 11:52:59 +00:00
Rohith Sharma K S
3cd75845da MAPREDUCE-7014. Fix java doc errors in jdk1.8. Contributed by Steve Loughran. 2017-11-27 22:01:00 +05:30
Steve Loughran
de8b6ca5ef HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints. 
Contributed by Steve Loughran and Ryan Blue.
 2017-11-22 15:28:12 +00:00
Steve Loughran
c326fc89b0 HADOOP-15024 Support user agent configuration and include that & Hadoop version information to oss server. 
Contributed by Sammi Chen.
 2017-11-20 18:56:42 +00:00
Akira Ajisaka
6903cf096e HADOOP-13514. Upgrade maven surefire plugin to 2.20.1 
Signed-off-by: Allen Wittenauer <aw@apache.org>
 2017-11-19 12:39:37 -08:00
Kai Zheng
18621af7ae HADOOP-14993. AliyunOSS: Override listFiles and listLocatedStatus. Contributed Genmao Yu 2017-11-14 17:58:37 +08:00
Daniel Templeton
ba8136615a YARN-7413. Support resource type in SLS (Contributed by Yufei Gu via Daniel Templeton) 
Change-Id: Ic0a897c123c5d2f57aae757ca6bcf1dad7b90d2b
 2017-11-09 12:09:48 -08:00
John Zhuge
bf6a660232 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 2017-11-09 10:16:12 -08:00
Subru Krishnan
f2df6b8983 HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru). 2017-11-08 18:07:12 -08:00
Arun Suresh
53c0fb7efe HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh) 2017-11-02 17:15:33 -07:00
Akira Ajisaka
cc3f3eca40
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi. 2017-11-02 18:32:24 +09:00
Chris Douglas
139cc752d2 HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li 2017-10-27 21:59:27 -07:00
Yufei Gu
99880d0a16 YARN-6505. Define the strings used in SLS JSON input file format. (Contributed by Gergely Novak) 2017-10-27 14:41:37 -07:00
Subru Krishnan
625039ef20 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 2017-10-25 15:51:27 -07:00
Wangda Tan
281d83604d YARN-7318. Fix shell check warnings of SLS. (Gergely Novák via wangda) 2017-10-20 15:54:15 -07:00
Steve Loughran
f36cbc8475 HADOOP-14942. DistCp#cleanup() should check whether jobFS is null. 
Contributed by Andras Bokor.
 2017-10-20 22:27:04 +01:00
Haibo Chen
46eb1033a8 YARN-6546. SLS is slow while loading 10k queues. (Yufei Gu via Haibo Chen) 2017-10-17 16:04:19 -07:00
Steve Loughran
9fcc3a1fc8 HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled. 
Contributed by Santhosh G Nayak
 2017-10-16 16:01:47 +01:00
Steve Loughran
3de574413c HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB. 
Contributed by Varada Hemeswari.
 2017-10-11 18:06:43 +01:00
Steve Loughran
572cdb5463 HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB 
Contributed by Kannapiran Srinivasan.
 2017-10-06 17:43:38 +01:00
Haibo Chen
c071aad5da YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen) 2017-10-05 15:50:13 -07:00
Steve Loughran
9288206cb3
HADOOP-14845. Azure wasb: getFileStatus not making any auth check. 
Contributed by Sivaguru Sankaridurg
 2017-10-05 15:05:23 +01:00
Yufei Gu
79e37dc6db YARN-7211. AMSimulator in SLS does't work due to responseId refactoring.(Contributed by Botong Huang via Yufei Gu) 2017-10-03 15:10:50 -07:00
Akira Ajisaka
c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 2017-10-04 07:01:54 +09:00
John Zhuge
8aca46e53f HADOOP-14651. Update okhttp version to 2.7.5. Contributed by Ray Chiang and John Zhuge. 2017-09-29 12:15:06 -07:00
Steve Loughran
a530e7ab3b
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB 
Contributed by Varada Hemeswari
 2017-09-28 19:52:56 +01:00
Aaron Fabbri
47011d7dd3
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. Contributed by Steve Loughran 2017-09-25 15:59:38 -07:00
Jason Lowe
b133dc5700 MAPREDUCE-6967. gridmix/SleepReducer should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 2017-09-22 15:21:42 -05:00
Steve Loughran
2d2d97fa7d
HADOOP-14553. Add (parallelized) integration tests to hadoop-azure 
Contributed by Steve Loughran
 2017-09-15 17:03:01 +01:00
Aaron Fabbri
49467165a5
HADOOP-14738 Remove S3N and obsolete bits of S3A; rework docs. Contributed by Steve Loughran. 2017-09-14 14:10:48 -07:00
ChenSammi
e0b3c644e1 HDFS-12414. Ensure to use CLI command to enable/disable erasure coding policy. Contributed by Sammi Chen 2017-09-14 09:15:29 +08:00
John Zhuge
a4661850c1 HADOOP-14844. Remove requirement to specify TenantGuid for MSI Token Provider. Contributed by Atul Sikaria. 2017-09-08 11:51:03 -07:00
Steve Loughran
5bbca80428
HADOOP-13421. Switch to v2 of the S3 List Objects API in S3A. 
Contributed by Aaron Fabbri
 2017-09-08 12:07:02 +01:00
Steve Loughran
83449abb40
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter. 
Contributed by Yonger.
 2017-09-07 19:55:56 +01:00
Steve Loughran
13eda50003
HADOOP-14520. WASB: Block compaction for Azure Block Blobs. 
Contributed by Georgi Chalakov
 2017-09-07 18:35:03 +01:00
Xiaoyu Yao
63720ef574 HADOOP-14839. DistCp log output should contain copied and deleted files and directories. Contributed by Yiqun Lin. 2017-09-05 23:34:55 -07:00
John Zhuge
50506e90a8 HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge. 2017-09-05 23:26:57 -07:00
Steve Loughran
792eff9ea7
HADOOP-14820 Wasb mkdirs security checks inconsistent with HDFS. 
Contributed by Sivaguru Sankaridurg
 2017-09-05 14:16:57 +01:00
Andrew Wang
0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Allen Wittenauer
a3fee475f7
HADOOP-14414. Calling maven-site-plugin directly for docs profile is unnecessary (Andras Bokor via aw) 2017-09-01 08:38:30 -07:00
Steve Loughran
621b43e254
HADOOP-13345 HS3Guard: Improved Consistency for S3A. 
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.
 2017-09-01 14:13:41 +01:00
John Zhuge
7ecc6dbed6 HADOOP-14824. Update ADLS SDK to 2.2.2 for MSI fix. Contributed by Atul Sikaria. 2017-08-31 21:13:22 -07:00
Allen Wittenauer
71bbb86d69
HADOOP-14670. Increase minimum cmake version for all platforms 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-30 21:47:24 -07:00
Haibo Chen
d04f85f387 MAPREDUCE-6892. Issues with the count of failed/killed tasks in the jhist file. (Peter Bacsko via Haibo Chen) 2017-08-30 10:07:48 -07:00
Yufei Gu
26fafc3597 YARN-7077. TestAMSimulator and TestNMSimulator fail. (Contributed by Akira Ajisaka via Yufei Gu) 2017-08-29 16:33:51 -07:00
Steve Loughran
9374f38820
HADOOP-14583. wasb throws an exception if you try to create a file and there's no parent directory 
Contributed by Esfandiar Manii and Thomas Marquardt.
 2017-08-29 19:03:44 +01:00
Steve Loughran
021974f4cb
HADOOP-14802. Add support for using container saskeys for all accesses. 
Contributed by Sivaguru Sankaridurg
 2017-08-29 19:02:43 +01:00
Andrew Wang
f29a0fc288 HDFS-12303. Change default EC cell size to 1MB for better performance. Contributed by Wei Zhou. 2017-08-25 14:14:23 -07:00
Arpit Agarwal
8b7cbe3840 HADOOP-14729. Upgrade JUnit 3 test cases to JUnit 4. Contributed by Ajay Kumar. 2017-08-25 09:55:46 -07:00
Steve Loughran
27ab5f7385
HADOOP-14787. AliyunOSS: Implement the createNonRecursive operator. 
Contributed by Genmao Yu
 2017-08-22 11:55:48 +01:00
Kai Zheng
267e19a09f HADOOP-14194. Aliyun OSS should not use empty endpoint as default. Contributed by Genmao Yu 2017-08-21 13:36:28 +08:00
Arun Suresh
8410d862d3 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 2017-08-20 07:54:09 -07:00
Steve Loughran
c6b4e656b7
HADOOP-14769. WASB: delete recursive should not fail if a file is deleted. 
Contributed by Thomas Marquardt
 2017-08-18 14:13:40 +01:00
Andrew Wang
dd7916d3cd HDFS-12250. Reduce usage of FsPermissionExtension in unit tests. Contributed by Chris Douglas. 2017-08-17 09:35:36 -07:00
Chris Douglas
645a8f2a4d HADOOP-14726. Mark FileStatus::isDir as final 2017-08-14 21:57:20 -07:00
John Zhuge
7769e96149 HADOOP-14627. Support MSI and DeviceCode token provider in ADLS. Contributed by Atul Sikaria. 2017-08-13 00:22:34 -07:00
Allen Wittenauer
4222c97108
HADOOP-10392. Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem) (ajisakaa via aw) 2017-08-11 09:25:56 -07:00
Steve Loughran
54356b1e83
HADOOP-14183. Remove service loader config file for wasb fs. 
Contributed by Esfandiar Manii.
 2017-08-10 16:46:33 +01:00
Steve Loughran
f4e1aa0508
HADOOP-14715. TestWasbRemoteCallHelper failing. 
Contributed by Esfandiar Manii.
 2017-08-08 23:37:47 +01:00
Chris Douglas
55a181f845 HADOOP-14730. Support protobuf FileStatus in AdlFileSystem. 2017-08-07 21:31:28 -07:00
Steve Loughran
d91b7a8451
HADOOP-14722. Azure: BlockBlobInputStream position incorrect after seek. 
Contributed by Thomas Marquardt
 2017-08-06 20:19:23 +01:00
Steve Loughran
7fc324aabd
HADOOP-14126. Remove jackson, joda and other transient aws SDK dependencies from hadoop-aws. 
Contributed by Steve Loughran

(cherry picked from commit ced547d5f0dbea571cbc472c5f55fe89d5900a6f)
 2017-08-04 11:09:08 +01:00
Sean Mackrory
1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Steve Loughran
778d4edd9a
HADOOP-14660 wasb: improve throughput by 34% when account limit exceeded. 
Contributed by Thomas Marquardt.
 2017-08-01 21:36:28 +01:00
John Zhuge
f735ad1b67 HADOOP-14678. AdlFilesystem#initialize swallows exception when getting user name. Contributed by John Zhuge. 2017-07-28 09:46:54 -07:00
Jitendra Pandey
a92bf39e23 HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. Contributed by Thomas Marquardt. 2017-07-25 16:26:48 -07:00
Jitendra Pandey
f2921e51f0 HADOOP-14518. Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Contributed by Georgi Chalakov. 2017-07-24 23:01:01 -07:00
Ravi Prakash
94ca52ae9e HADOOP-14597. Native compilation broken with OpenSSL-1.1.0. Contributed by Ravi Prakash. 2017-07-24 16:01:45 -07:00
Wei-Chiu Chuang
44350fdf49 HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default). Contributed by Bharat Viswanadham. 2017-07-20 18:23:13 -07:00
Haibo Chen
0ba8cda135 YARN-6685. Add job count in to SLS JSON input format. (Yufei Gu via Haibo Chen) 2017-07-20 08:15:46 -07:00
Jitendra Pandey
2843c688bc HADOOP-14642. wasb: add support for caching Authorization and SASKeys. Contributed by Sivaguru Sankaridurg. 2017-07-19 00:13:06 -07:00
Jitendra Pandey
b0e78ae085 HADOOP-14640. Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak. 2017-07-17 02:27:55 -07:00
Steve Loughran
7d272ea124
HADOOP-14581. Restrict setOwner to list of user when security is enabled in wasb. 
Contributed by Varada Hemeswari

(cherry picked from commit 1e69e5260351effc8077d1bdc397cec57cf1ff1b)
 2017-07-12 10:37:39 +01:00
Steve Loughran
d670c3a4da
HADOOP-14535 wasb: implement high-performance random access and seek of block blobs. 
Contributed by Thomas Marquardt
 2017-07-11 21:34:27 +01:00
Haibo Chen
147df300bf YARN-5067 Support specifying resources for AM containers in SLS. (Yufei Gu via Haibo Chen) 2017-06-30 17:03:44 -07:00
Mingliang Liu
38996fdcf0 HADOOP-14443. Azure: Support retry and client side failover for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak 2017-06-30 16:53:48 -07:00
Andrew Wang
af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00
Mingliang Liu
72993b33b7 HADOOP-14596. AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error. Contributed by Steve Loughran 
Change-Id: I49173bf6163796903d64594a8ca8a4bd26ad2bfc
 2017-06-29 17:07:52 -07:00
Mingliang Liu
b08cc97396 HADOOP-14601. Azure: Reuse ObjectMapper. Contributed by Mingliang Liu 2017-06-29 09:22:00 -07:00
Allen Wittenauer
20ba86d66a MAPREDUCE-6536. hadoop-pipes doesn't use maven properties for openssl 
Signed-off-by: Ravi Prakash <raviprak@apache.org>
 2017-06-28 17:33:44 -07:00
Mingliang Liu
990aa34de2 HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code. Contributed by Steve Loughran 2017-06-28 14:18:59 -07:00
Steve Loughran
ee243e5289
HADOOP-14190. Add more on S3 regions to the s3a documentation. 
Contributed by Steve Loughran
 2017-06-28 10:22:13 +01:00
Mingliang Liu
7e031c2c18 HADOOP-14546. Azure: Concurrent I/O does not work when secure.mode is enabled. Contributed by Thomas 2017-06-27 17:32:07 -07:00
Mingliang Liu
686a634f01 HADOOP-14573. regression: Azure tests which capture logs failing with move to SLF4J. Contributed by Steve Loughran 2017-06-27 16:48:47 -07:00
Mingliang Liu
bc4dfe9c9c HADOOP-14594. ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions. Contributed by Mingliang Liu 2017-06-27 13:19:14 -07:00
Steve Loughran
48f4a229a9
HADOOP-14461 Azure: handle failure gracefully in case of missing account access key. 
Contributed by Mingliang Liu.
 2017-06-26 13:39:47 +01:00
Akira Ajisaka
a594f96fb1
HADOOP-14549. Use GenericTestUtils.setLogLevel when available in hadoop-tools. Contributed by wenxin he. 2017-06-26 16:41:00 +09:00
Steve Loughran
c6bd73c6c5
HADOOP-14547. [WASB] the configured retry policy is not used for all storage operations. 
Contributed by Thomas.
 2017-06-23 21:38:41 +01:00
Jason Lowe
dd65eea74b HADOOP-8143. Change distcp to have -pb on by default. Contributed by Mithun Radhakrishnan 2017-06-20 09:53:47 -05:00
Akira Ajisaka
3369540653
HADOOP-14296. Move logging APIs over to slf4j in hadoop-tools. 2017-06-20 13:18:26 +09:00
Mingliang Liu
f214a9961f HADOOP-14494. ITestJets3tNativeS3FileSystemContract tests NPEs in teardown if store undefined. Contributed by Steve Loughran 2017-06-15 11:16:16 -07:00
Mingliang Liu
325163f23f HADOOP-14512. WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename. Contributed by Duo Xu 2017-06-09 13:07:11 -07:00
Mingliang Liu
df35e4cc3b HADOOP-14500. Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails. Contributed by Rajesh Balamohan 2017-06-07 13:54:52 -07:00
Carlo Curino
b65100c14b YARN-6547. Enhance SLS-based tests leveraging invariant checker. 2017-06-06 17:19:15 -07:00
Mingliang Liu
536f057158 HADOOP-14491. Azure has messed doc structure. Contributed by Mingliang Liu 2017-06-06 11:09:28 -07:00
Mingliang Liu
6b5285bbcb HADOOP-14472. Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently. Contributed by Mingliang Liu 2017-06-06 11:06:49 -07:00
Mingliang Liu
5fd9742c83 HADOOP-14478. Optimize NativeAzureFsInputStream for positional reads. Contributed by Rajesh Balamohan 2017-06-05 15:56:43 -07:00
Mingliang Liu
6aeda55bb8 HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu 2017-06-05 11:26:56 -07:00
Mingliang Liu
ece33208b8 HADOOP-14460. Azure: update doc for live and contract tests. Contributed by Mingliang Liu 2017-06-01 11:52:11 -07:00
Akira Ajisaka
6e6e96583f
HADOOP-14466. Remove useless document from TestAliyunOSSFileSystemContract.java. Contributed by Chen Liang. 2017-06-01 13:08:01 +09:00
Akira Ajisaka
07e60f85d8
HADOOP-14458. Add missing imports to TestAliyunOSSFileSystemContract.java. Contributed by Mingliang Liu. 2017-05-30 15:11:10 +09:00