big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,735 Commits 2 Branches 2 Tags 264 MiB
56350664a7
Commit Graph

23735 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mukund Thakur
56350664a7
HADOOP-13873. log DNS addresses on s3a initialization. 
Contributed by Mukund Thakur.

If you set the log org.apache.hadoop.fs.s3a.impl.NetworkBinding
to DEBUG, then when the S3A bucket probe is made -the DNS address
of the S3 endpoint is calculated and printed.

This is useful to see if a large set of processes are all using
the same IP address from the pool of load balancers to which AWS
directs clients when an AWS S3 endpoint is resolved.

This can have implications for performance: if all clients
access the same load balancer performance may be suboptimal.

Note: if bucket probes are disabled, fs.s3a.bucket.probe = 0,
the DNS logging does not take place.

Change-Id: I21b3ac429dc0b543f03e357fdeb94c2d2a328dd8
 2020-04-17 14:15:38 +01:00
Ayush Saxena
37d6582223 HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena. 2020-04-17 13:32:02 +05:30
Stephen O'Donnell
3481895f8a HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-04-16 20:18:25 -07:00
Eric Yang
053e39e1e4 MAPREDUCE-7273. Fixed Job History server token renewal. 
Contributed by Peter Bacsko via eyang
 2020-04-16 20:17:21 -04:00
Mukund Thakur
8505840c26
HADOOP-16979. S3Guard auth mode should be set to false by default in integration tests. (#1958) Contributed by Mukund Thakur. 2020-04-16 13:02:30 +02:00
Ayush Saxena
cc5c1da7c1 HDFS-15277. Parent directory in the explorer does not support all path formats. Contributed by Jianfei Jiang. 2020-04-15 23:01:58 +05:30
Vinayakumar B
4d24d99e85
HADOOP-16985. Handle release package related issues (#1957) 2020-04-15 20:38:19 +05:30
Szilard Nemeth
4db598e0e6 YARN-9999. TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup. Contributed by Benjamin Teke 2020-04-15 13:18:44 +02:00
Prabhu Joseph
94f7470c71 YARN-10233. Fix YARN UI2 Daemon Logs 
Contributed by Akhil PB.
 2020-04-15 13:39:37 +05:30
Szilard Nemeth
55fcbcb5c2 YARN-10234. FS-CS converter: don't enable auto-create queue property for root. Contributed by Peter Bacsko 2020-04-15 07:24:04 +02:00
sunlisheng
ae474e1ae5 HADOOP-16982. Update Netty to 4.1.48.Final. Contributed by Lisheng Sun. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-04-15 06:41:32 +09:00
bilaharith
0ad0102678
HADOOP-16855. Changing wildfly dependency scope in hadoop-azure to compile 
Contributed by Biliharith
 2020-04-14 19:17:12 +01:00
Mukund Thakur
7b2d84d19c
HADOOP-16465 listLocatedStatus() optimisation (#1943) 
Contributed by Mukund Thakur

Optimize S3AFileSystem.listLocatedStatus() to perform list
operations directly and then fallback to head checks for files
 2020-04-14 17:19:51 +01:00
Steve Loughran
aeeebc5e79
HADOOP-16941. ITestS3GuardOutOfBandOperations.testListingDelete failing on versioned bucket (#1919) 
Contributed by Steve Loughran.

Removed the failing probe and replacing with two probes which will fail
on both versioned and unversioned buckets.
 2020-04-14 10:56:00 +01:00
Prabhu Joseph
c791b0e90e YARN-10219. Fix YARN Native Service Placement Constraints with Node Attributes. 
Contributed by Eric Yang.
 2020-04-14 12:34:53 +05:30
Ayush Saxena
3edbe8708a HADOOP-16958. NPE when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL. Contributed by Ctest. 2020-04-14 02:13:42 +05:30
Eric E Payne
11d17417ce MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein) 2020-04-13 18:20:07 +00:00
Ayush Saxena
8d49229c37 HDFS-15247. RBF: Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 2020-04-12 12:05:50 +05:30
Ayush Saxena
2ea5adf2de HADOOP-16967. TestSequenceFile#testRecursiveSeqFileCreate fails in subsequent run. Contributed by Ctest. 2020-04-12 11:31:07 +05:30
Ayush Saxena
275c478330 HDFS-15261. RBF: Add Block Related Metrics. Contributed by Ayush Saxena. 2020-04-09 23:44:36 +05:30
Jonathan Hung
23481ad378 YARN-10212. Create separate configuration for max global AM attempts. Contributed by Bilwa S T 2020-04-09 11:06:11 -07:00
Ayush Saxena
5746533cde HADOOP-16962. Making getBoolean log warning message for unrecognized value. Contributed by Ctest. 2020-04-09 23:30:28 +05:30
Wei-Chiu Chuang
061afcdf30
HDFS-15269. NameNode should check the authorization API version only … (#1945) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-04-09 09:19:35 -07:00
Szilard Nemeth
c04a484aed YARN-9995. Code cleanup in TestSchedConfCLI. Contributed by Bilwa S T. 2020-04-09 15:46:25 +02:00
Sunil G
6fb29d0f17 YARN-10226. NPE in Capacity Scheduler while using %primary_group queue mapping. Contributed by Peter Bacsko. 2020-04-09 18:51:42 +05:30
Szilard Nemeth
e2c87b7917 MAPREDUCE-7266. historyContext doesn't need to be a class attribute inside JobHistoryServer. Contributed by Siddharth Ahuja 2020-04-09 12:38:54 +02:00
Szilard Nemeth
8e4517f251 YARN-5277. When localizers fail due to resource timestamps being out, provide more diagnostics. Contributed by Siddharth Ahuja 2020-04-09 12:31:01 +02:00
Akira Ajisaka
e53d472bb0
HADOOP-14836. Upgrade maven-clean-plugin to 3.1.0 (#1933) 2020-04-09 01:47:31 +09:00
Ayush Saxena
1189af4746 HDFS-15263. Fix the logic of scope and excluded scope in Network Topology. Contributed by Ayush Saxena. 2020-04-08 15:55:14 +05:30
Wilfred Spiegelenburg
2214005c0f
YARN-10063. Add container-executor arguments --http/--https to usage. Contributed by Siddharth Ahuja 2020-04-08 11:30:03 +10:00
Steve Loughran
20eec95867
HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 (#1936) 
Contributed by Steve Loughran.

This strips out all the -p preservation options which have already been
processed when uploading a file before deciding whether or not to query
the far end for the status of the (existing/uploaded) file to see if any
other attributes need changing.

This will avoid 404 caching-related issues in S3, wherein a newly created
file can have a 404 entry in the S3 load balancer's cache from the
probes for the file's existence prior to the upload.

It partially addresses a regression caused by HADOOP-8143,
"Change distcp to have -pb on by default" that causes a resurfacing
of HADOOP-13145, "In DistCp, prevent unnecessary getFileStatus call when
not preserving metadata"
 2020-04-07 17:55:55 +01:00
Szilard Nemeth
bffb43b00e YARN-10207. CLOSE_WAIT socket connection leaks during rendering of (corrupted) aggregated logs on the JobHistoryServer Web UI. Contributed by Siddharth Ahuja 2020-04-07 17:03:17 +02:00
Szilard Nemeth
45362a9f4c YARN-10001. Add explanation of unimplemented methods in InMemoryConfigurationStore. Contributed by Siddharth Ahuja 2020-04-07 15:51:55 +02:00
Toshihiro Suzuki
c12ddbd1de
HDFS-15249 ThrottledAsyncChecker is not thread-safe. (#1922) 2020-04-07 14:38:09 +09:00
Inigo Goiri
0b855b9f35 HDFS-15256. Fix typo in DataXceiverServer#run(). Contributed by Lisheng Sun. 2020-04-06 13:05:33 -07:00
Akira Ajisaka
ab7495dc7d
MAPREDUCE-7269. TestNetworkedJob fails (#1940) 2020-04-06 17:57:37 +09:00
Takanobu Asanuma
b89d875f7b HADOOP-16954. Add -S option in "Count" command to show only Snapshot Counts. Contributed by hemanthboyina. 2020-04-06 11:03:10 +09:00
Akira Ajisaka
e6455cc864
HADOOP-16949. pylint fails in the build environment (#1926) 2020-04-04 16:52:04 +09:00
Masatake Iwasaki
8f8be6b92a HADOOP-16647. Support OpenSSL 1.1.1 LTS. Contributed by Rakesh Radhakrishnan. 2020-04-04 07:20:51 +09:00
Akira Ajisaka
1695d8d59c
HDFS-15258. RBF: Mark Router FSCK unstable. (#1934) 2020-04-04 00:13:41 +09:00
Ayush Saxena
ff50ecdc10 HADOOP-16952. Add .diff to gitignore. Contributed by Ayush Saxena. 2020-04-03 13:57:02 +05:30
Prabhu Joseph
7a39da3537 YARN-10120. Amendment fix for Java Doc. 
Contributed by Bilwa S T.
 2020-04-03 13:07:41 +05:30
Wanqiang Ji
f8e598f278
MAPREDUCE-7268. Fix TestMapreduceConfigFields (#1935) 2020-04-03 10:59:07 +09:00
Prabhu Joseph
25361b077b YARN-10120. Https Support in Router WebServiceClient. 
Contributed by Bilwa S T.
 2020-04-02 18:11:43 +05:30
Takanobu Asanuma
736659e0e1 HDFS-15252. HttpFS: setWorkingDirectory should not accept invalid paths. Contributed by hemanthboyina. 2020-04-02 19:21:02 +09:00
Ayush Saxena
4a3eb10972 HDFS-15051. RBF: Impose directory level permissions for Mount entries. Contributed by Xiaoqiao He. 2020-04-02 11:06:01 +05:30
He Xiaoqiao
d3b5951572 HDFS-15242. Add metrics for operations hold lock times of FsDatasetImpl. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-01 16:36:46 -07:00
Jason Lowe
c613296dc8 MAPREDUCE-7241. FileInputFormat listStatus with less memory footprint. Contributed by Zhihua Deng 2020-04-01 07:46:33 -05:00
Akira Ajisaka
c162648aff
YARN-10202. Fix documentation about NodeAttributes. Contributed by Sen Zhao. 2020-04-01 16:05:47 +09:00
Mehakmeet Singh
c734d247b1
HADOOP-16910 : ABFS Streams to update FileSystem.Statistics counters on IO. (#1918). Contributed by Mehakmeet Singh. 2020-03-31 14:49:09 +02:00