big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Projects Wiki Activity
23,830 Commits 2 Branches 2 Tags
Commit Graph

23830 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mehakmeet Singh
d1ba6c963d HADOOP-17020. Improve RawFileSystem Performance (#2063) 
Contributed by : Mehakmeet Singh

Co-authored-by: Rajesh Balamohan
Co-authored-by: Mehakmeet Singh
 2020-06-17 16:16:30 +01:00
Szilard Nemeth
ec913398a9 YARN-10281. Redundant QueuePath usage in UserGroupMappingPlacementRule and AppNameMappingPlacementRule. Contributed by Gergely Pollak 2020-06-17 14:36:08 +02:00
Vinayakumar B
c1ef247dc6
YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars (#2075) 2020-06-17 09:29:49 +05:30
Uma Maheswara Rao G
120ee793fc HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 785b1def959fab6b8b7ffff66410bcd240feee13)
 2020-06-16 20:02:44 -07:00
Ayush Saxena
bee2846bee HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena 
(cherry picked from commit cc671b16f7b0b7c1ed7b41b96171653dc43cf670)
 2020-06-16 16:54:40 -07:00
Uma Maheswara Rao G
418580446b HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 76fa0222f0d2e2d92b4a1eedba8b3e38002e8c23)
 2020-06-16 16:54:01 -07:00
Uma Maheswara Rao G
0b5e202614 HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ed83c865dd0b4e92f3f89f79543acc23792bb69c)
 2020-06-16 16:53:38 -07:00
Uma Maheswara Rao G
8e71e85af7 HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 4734c77b4b64b7c6432da4cc32881aba85f94ea1)
 2020-06-16 16:53:10 -07:00
Abhishek Das
5b248de42d HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 
(cherry picked from commit ce4ec7445345eb94c6741d416814a4eac319f0a6)
 2020-06-16 16:52:29 -07:00
Uma Maheswara Rao G
544996c857 HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac4a2e11d98827c7926a34cda27aa7bcfd3f36c1)
 2020-06-16 16:50:57 -07:00
Stephen O'Donnell
0b9e5ea592 HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 730a39d1388548f22f76132a6734d61c24c3eb72)
 2020-06-16 16:02:07 -07:00
Szilard Nemeth
8be302a3b8 YARN-10274. Merge QueueMapping and QueueMappingEntity. Contributed by Gergely Pollak 2020-06-16 18:25:47 +02:00
Szilard Nemeth
52efe48d79 YARN-10292. FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler. Contributed by Benjamin Teke 2020-06-16 18:01:39 +02:00
Eric Yang
d73cdb1c86
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 81d8a887b0406380e469c76ed2e41022a6372dd7)
 2020-06-15 11:12:08 +09:00
Takanobu Asanuma
3b8418250f HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina. 
(cherry picked from commit f41a144077fc0e2d32072e0d088c1abd1897cee5)
 2020-06-15 09:17:40 +09:00
Vinayakumar B
534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30
Eric Badger
fcd7ce53b5 YARN-10312. Add support for yarn logs -logFile to retain backward compatibility. 
Contributed by Jim Brennan.

(cherry picked from commit fed6fecd3a9e24efc20f9221505da35a7e1949c7)
 2020-06-11 21:11:20 +00:00
Szilard Nemeth
e35f619841 YARN-10296. Make ContainerPBImpl#getId/setId synchronized. Contributed by Benjamin Teke 2020-06-10 18:00:21 +02:00
Ayush Saxena
043628dcf1 HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang. 2020-06-10 12:09:34 +05:30
Eric E Payne
a7526ba9f7 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 
(cherry picked from commit 56247db3022705635580c4d2f8b0abde109f954f)
 2020-06-09 18:51:46 +00:00
Steve Loughran
5e290e702f
HADOOP-17050. S3A to support additional token issuers 
Contributed by Steve Loughran.

S3A delegation token providers will be asked for any additional
token issuers, an array can be returned,
each one will be asked for tokens when DelegationTokenIssuer collects
all the tokens for a filesystem.

Change-Id: I1bd3035bbff98cbd8e1d1ac7fc615d937e6bb7bb
 2020-06-09 14:43:02 +01:00
Ayush Saxena
2d4faa39e8 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 
*Added missed test file.
 2020-06-09 18:47:24 +05:30
Eric Badger
890617c7ac Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)." 
This reverts commit 741fcf2c63c639edfc66834088378afd473c9ce6.
 2020-06-08 20:25:02 +00:00
Mingliang Liu
fa723aa7f8
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:31:42 -07:00
Mingliang Liu
543075b845
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-08 10:38:17 -07:00
Szilard Nemeth
ac307fe20d YARN-10284. Add lazy initialization of LogAggregationFileControllerFactory in LogServlet. Contributed by Adam Antal 2020-06-05 12:40:57 +02:00
Toshihiro Suzuki
ec8f3714e0 HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.

(cherry picked from commit 545a0a147c5256c44911ba57b4898e01d786d836)
 2020-06-05 11:17:13 +01:00
Szilard Nemeth
a266e32d82 YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed by Andras Gyori 2020-06-05 09:50:43 +02:00
Akira Ajisaka
9cfc0e50fa
HADOOP-17056. Addendum patch: Fix typo 
(cherry picked from commit 5157118bd7f3448949da885e323c163828c35aee)
 2020-06-04 16:35:27 +09:00
Akira Ajisaka
0b25913384
HADOOP-17062. Fix shelldocs path in Jenkinsfile (#2049) 
(cherry picked from commit 704409d53bf7ebf717a3c2e988ede80f623bbad3)
 2020-06-04 06:05:51 +09:00
Mehakmeet Singh
1714589609
HADOOP-17016. Adding Common Counters in ABFS (#1991). 
Contributed by: Mehakmeet Singh.

Change-Id: Ib84e7a42f28e064df4c6204fcce33e573360bf42
 2020-06-03 20:02:44 +01:00
Steve Loughran
8a642caca8
HADOOP-16568. S3A FullCredentialsTokenBinding fails if local credentials are unset. (#1441) 
Contributed by Steve Loughran.

Move the loading to deployUnbonded (where they are required) and add a safety check when a new DT is requested

Change-Id: I03c69aa2e16accfccddca756b2771ff832e7dd58
 2020-06-03 17:08:52 +01:00
Mike
cf84bec6e3 HADOOP-14566. Add seek support for SFTP FileSystem. (#1999) 
Contributed by Mikhail Pryakhin
 2020-06-03 11:38:49 +01:00
Akira Ajisaka
c88bf7acc1
HADOOP-17056. shelldoc fails in hadoop-common. (#2045) 
In the docker build image, skip GPG verification when downloading
Yetus tarball via yetus-wrapper.

(cherry picked from commit 9c290c08db4361de29f392b0569312c2623b8321)
 2020-06-03 18:03:03 +09:00
Szilard Nemeth
f65f64e8ae YARN-10254. CapacityScheduler incorrect User Group Mapping after leaf queue change. Contributed by Gergely Pollak 2020-06-02 18:32:06 +02:00
Dhiraj
910d88eeed
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 10:50:22 -07:00
Prabhu Joseph
07b8963aa3 YARN-10259. Fix reservation logic in Multi Node Placement. 
Reviewed by Wangda Tan.

(cherry picked from commit 6ce295b78737aca8103912121d54f318cb5d36ef)
 2020-06-01 18:24:38 +05:30
Mukund Thakur
b0c9e4f1b5
HADOOP-16900. Very large files can be truncated when written through the S3A FileSystem. 
Contributed by Mukund Thakur and Steve Loughran.

This patch ensures that writes to S3A fail when more than 10,000 blocks are
written. That upper bound still exists. To write massive files, make sure
that the value of fs.s3a.multipart.size is set to a size which is large
enough to upload the files in fewer than 10,000 blocks.

Change-Id: Icec604e2a357ffd38d7ae7bc3f887ff55f2d721a
 2020-06-01 12:01:13 +01:00
Jonathan Hung
7a323a45aa YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 
(cherry picked from commit c30c23cb665761e997bcfc1dc00908f70b069fa2)
 2020-05-29 09:25:08 -07:00
S O'Donnell
90f57965e9 HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 
(cherry picked from commit 19f26a020e2e5cec2ceb28d796c63c83bc8ac506)
 2020-05-29 14:49:40 +01:00
S O'Donnell
b803efbdce HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 2020-05-29 11:44:48 +01:00
Wanqiang Ji
9b84a637b7
HADOOP-17055. Remove residual code of Ozone (#2039) 
(cherry picked from commit d9838f2d42eaadd0769167847af4e8f2963817fb)
 2020-05-29 16:50:10 +09:00
Ayush Saxena
6b7040a1cb HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 2020-05-28 13:04:22 +05:30
Wilfred Spiegelenburg
586d9427e3
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false 
Contributed by Tarun Parimi.

(cherry picked from commit 10db97df1c8562a9e29b00e60d5bde1773c09188)
 2020-05-28 13:36:21 +10:00
Szilard Nemeth
94358dbc3c YARN-10108. FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 
(cherry picked from commit ac4540dd8e2872b113a61107b0a4a66beeec8649)
 2020-05-27 13:07:17 +02:00
Szilard Nemeth
2c60bcb1c2 Revert "FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak" 
This reverts commit 9ce913a7bf720944d0305960bbf1899461b8628b.
Reverting because of missing Jira-ID in the commit message
 2020-05-27 13:07:01 +02:00
Ayush Saxena
718c8a5868 HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun. 2020-05-24 18:06:19 +05:30
Takanobu Asanuma
c44d8b8bdf HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f0eb093775f0fe4f39ef4be639ad752)
 2020-05-23 17:33:39 +09:00
Szilard Nemeth
9ce913a7bf FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 
(cherry picked from commit ac4540dd8e2872b113a61107b0a4a66beeec8649)
 2020-05-22 14:48:53 +02:00
Masatake Iwasaki
4d30c395f7 HADOOP-17040. Fix intermittent failure of ITestBlockingThreadPoolExecutorService. (#2020) 
(cherry picked from commit 968531463375ebf29ba3186c13b5f8685df10d25)
 2020-05-22 18:53:53 +09:00