big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
17,834 Commits 2 Branches 2 Tags 264 MiB
31db977cb3
Commit Graph

863 Commits

Author SHA1 Message Date
Steve Loughran
7ac88244c5 HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. 
Contributed by Steve Loughran.
 2018-02-16 16:37:06 +00:00
Steve Loughran
481d79fedc HADOOP-13972. ADLS to support per-store configuration. 
Contributed by Sharad Sonker.

(cherry picked from commit 050f5287b79324b7f6231b879c0bfc608203b980)
 2018-02-15 16:27:31 +00:00
Steve Loughran
9a013b255f HADOOP-15176. Enhance IAM Assumed Role support in S3A client. 
Contributed by Steve Loughran

(cherry picked from commit 96c047fbb98c2378eed9693a724d4cbbd03c00fd)
 2018-02-15 15:57:10 +00:00
Steve Loughran
b27ab7dd81 HADOOP-15076. Enhance S3A troubleshooting documents and add a performance document. 
Contributed by Steve Loughran.

(cherry picked from commit c761e658f6594c4e519ed39ef36669de2c5cee15)
 2018-02-15 14:57:56 +00:00
Steve Loughran
c9a373fb14 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran.

(cherry picked from commit 58a2120e8a31307f19551f87be4e81d4fb626de1)
 2018-02-15 14:26:00 +00:00
Steve Loughran
f20dc0d577 HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions. 
Contributed by Andras Bokor.
 2018-02-14 16:20:14 +00:00
Steve Loughran
8cf88fcd1f HADOOP-15187. Remove ADL mock test dependency on REST call invoked from Java SDK. 
Contributed by Vishwajeet Dusane.

(cherry picked from commit bd5ab5912564d2d687651b01f552b8e4ca8c145a)
 2018-02-12 15:13:00 +00:00
Wangda Tan
60f9e60b3b Preparing for 3.2.0 development 
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
 2018-02-11 11:17:38 +08:00
Anu Engineer
4304fcd5bd HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 2018-02-06 13:43:45 -08:00
Arpit Agarwal
d4e13a4647 HADOOP-15198. Correct the spelling in CopyFilter.java. Contributed by Mukul Kumar Singh. 2018-02-02 11:37:51 -08:00
Sunil G
9b81cb0537 YARN-7779. Display allocation tags in RM web UI and expose same through REST API. Contributed by Weiwei Yang. 2018-01-31 01:30:17 -08:00
Arun Suresh
38af237969 YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Steve Loughran
7fd287b4af HADOOP-15186. Allow Azure Data Lake SDK dependency version to be set on the command line. 
Contributed by Vishwajeet Dusane.

(cherry picked from commit 95a96b13e2a54e01ea6c6933045d912998477da3)
 2018-01-29 09:48:14 -08:00
Jason Lowe
ff8378eb1b YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:47:19 -06:00
Yufei Gu
16be42d309 YARN-7798. Refactor SLS Reservation Creation. Contributed by Young Chen. 2018-01-25 13:06:01 -08:00
Steve Loughran
f63d13f10d HADOOP-15185. Update adls connector to use the current version of ADLS SDK. 
Contributed by Atul Sikaria.
 2018-01-23 09:09:11 -08:00
Steve Loughran
1093a73689 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits. 
Contributed by Aaron Fabbri
 2018-01-18 13:13:58 +00:00
Steve Loughran
f274fe33ea Revert "HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits." 
This reverts commit 35ad9b1dd2.
 2018-01-18 12:35:57 +00:00
Aaron Fabbri
268ab4e027
HADOOP-15141 Support IAM Assumed roles in S3A. Contributed by Steve Loughran. 2018-01-17 00:05:24 -08:00
Sammi Chen
9195a6e302 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 2018-01-17 15:55:59 +08:00
Steve Loughran
a0c71dcc33 HADOOP-15079. ITestS3AFileOperationCost#testFakeDirectoryDeletion failing 
after OutputCommitter patch.
Contributed by Steve Loughran
 2018-01-15 11:33:32 +00:00
Mukul Kumar Singh
2e1e9017aa
HADOOP-15172. Fix the javadoc warning in WriteOperationHelper.java 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-01-15 15:45:51 +09:00
Steve Loughran
1a09da7400 HADOOP-15163. Fix S3ACommitter documentation 
Contributed by Alessandro Andrioni.

(cherry picked from commit 100e8a1ae1d930dde084af7d1281e491c7f124ec)
 2018-01-10 15:37:07 +00:00
Steve Loughran
b62a5ece95 HADOOP-15161. s3a: Stream and common statistics missing from metrics 
Contributed by Sean Mackrory
 2018-01-09 18:46:52 +00:00
Steve Loughran
52babbb4a0 HADOOP-15086. NativeAzureFileSystem file rename is not atomic. 
Contributed by Thomas Marquardt
 2017-12-22 11:39:55 +00:00
Steve Loughran
c8ff0cc304 HADOOP-13282. S3 blob etags to be made visible in S3A status/getFileChecksum() calls. 
Contributed by Steve Loughran
 2017-12-21 14:58:58 +00:00
Steve Loughran
ef450df443 HADOOP-15113. NPE in S3A getFileStatus: null instrumentation on using closed instance. 
Contributed by Steve Loughran.
 2017-12-21 14:15:53 +00:00
Akira Ajisaka
5ab632baf5
HDFS-12949. Fix findbugs warning in ImageWriter.java. 2017-12-21 10:04:34 +09:00
Steve Loughran
1ba491ff90 HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive 2017-12-20 18:25:33 +00:00
Steve Loughran
35ad9b1dd2 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits. 
Contributed by Aaron Fabbri
 2017-12-18 21:19:06 +00:00
Chris Douglas
4b3a785914 HDFS-12903. [READ] Fix closing streams in ImageWriter. Contributed by Virajith Jalaparti 2017-12-15 17:51:42 -08:00
Chris Douglas
e515103a83 Revert "HDFS-12903. [READ] Fix closing streams in ImageWriter" 
This reverts commit c1bf2654b0e9118985b8518b0254eac4dd302a2f.
 2017-12-15 17:51:42 -08:00
Virajith Jalaparti
8239e3afb3 HDFS-12712. [9806] Code style cleanup 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
80c3fec3a1 HDFS-12912. [READ] Fix configuration and implementation of LevelDB-based alias maps 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
c89b29bd42 HDFS-12893. [READ] Support replication of Provided blocks with non-default topologies. 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
0f6aa9564c HDFS-12905. [READ] Handle decommissioning and under-maintenance Datanodes with Provided storage. 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
962b5e722b HDFS-12903. [READ] Fix closing streams in ImageWriter 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
4531588a94 HDFS-11640. [READ] Datanodes should use a unique identifier when reading from external stores 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
fb996a32a9 HDFS-12894. [READ] Skip setting block count of ProvidedDatanodeStorageInfo on DN registration update 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
9c35be86e1 HDFS-12713. [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata. Contributed by Ewan Higgs 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
a027055dd2 HDFS-12885. Add visibility/stability annotations. Contributed by Chris Douglas 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
352f994b64 HDFS-12665. [AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb). Contributed by Ewan Higgs. 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
4d59dabb7f HDFS-12809. [READ] Fix the randomized selection of locations in {{ProvidedBlocksBuilder}}. 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
3d3be87e30 HDFS-12778. [READ] Report multiple locations for PROVIDED blocks 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
3b1d30301b HDFS-12775. [READ] Fix reporting of Provided volumes 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
6cd80b2521 HDFS-12779. [READ] Allow cluster id to be specified to the Image generation tool 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
90d1b47a2a HDFS-12776. [READ] Increasing replication for PROVIDED files should create local replicas 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
87dc026bee HDFS-12789. [READ] Image generation tool does not close an opened stream 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
c293cc8e9b HDFS-12671. [READ] Test NameNode restarts when PROVIDED is configured 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
71d0a82571 HDFS-12607. [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED 2017-12-15 17:51:39 -08:00