big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HADOOP-17869. fs.s3a.connection.maximum should be bigger than fs.s3a.threads.max (#3337).

Browse Source 
The value of `fs.s3a.connection.maximum` has been increased to 96

Contributed by Dongjoon Hyun
This commit is contained in:
Dongjoon Hyun 2021-08-30 10:30:43 -07:00 committed by GitHub
parent 50dda774f1
commit 265a48e245
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
View File

@ -1381,7 +1381,7 @@
<property> <property>
<name>fs.s3a.connection.maximum</name> <name>fs.s3a.connection.maximum</name>
<value>48</value> <value>96</value>
<description>Controls the maximum number of simultaneous connections to S3. <description>Controls the maximum number of simultaneous connections to S3.
This must be bigger than the value of fs.s3a.threads.max so as to stop This must be bigger than the value of fs.s3a.threads.max so as to stop
threads being blocked waiting for new HTTPS connections. threads being blocked waiting for new HTTPS connections.

2
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
View File

@ -147,7 +147,7 @@ private Constants() {
// number of simultaneous connections to s3 // number of simultaneous connections to s3
public static final String MAXIMUM_CONNECTIONS = "fs.s3a.connection.maximum"; public static final String MAXIMUM_CONNECTIONS = "fs.s3a.connection.maximum";
public static final int DEFAULT_MAXIMUM_CONNECTIONS = 48; public static final int DEFAULT_MAXIMUM_CONNECTIONS = 96;
// connect to s3 over ssl? // connect to s3 over ssl?
public static final String SECURE_CONNECTIONS = public static final String SECURE_CONNECTIONS =