big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool. (#6372)

Browse Source 
HADOOP-19015.  Increase fs.s3a.connection.maximum to 500 to minimize the risk of Timeout waiting for connection from the pool

Contributed By: Mukund Thakur
This commit is contained in:
Mukund Thakur 2024-01-16 17:06:28 -06:00 committed by GitHub
parent d378853790
commit 7b1570e2f1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 4 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
View File

@ -1530,7 +1530,7 @@
<property> <property>
<name>fs.s3a.connection.maximum</name> <name>fs.s3a.connection.maximum</name>
<value>200</value> <value>500</value>
<description>Controls the maximum number of simultaneous connections to S3. <description>Controls the maximum number of simultaneous connections to S3.
This must be bigger than the value of fs.s3a.threads.max so as to stop This must be bigger than the value of fs.s3a.threads.max so as to stop
threads being blocked waiting for new HTTPS connections. threads being blocked waiting for new HTTPS connections.
@ -1538,6 +1538,7 @@
</description> </description>
</property> </property>
<property> <property>
<name>fs.s3a.connection.ssl.enabled</name> <name>fs.s3a.connection.ssl.enabled</name>
<value>true</value> <value>true</value>

2
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
View File

@ -173,7 +173,7 @@ private Constants() {
* Future releases are likely to increase this value. * Future releases are likely to increase this value.
* Keep in sync with the value in {@code core-default.xml} * Keep in sync with the value in {@code core-default.xml}
*/ */
public static final int DEFAULT_MAXIMUM_CONNECTIONS = 200; public static final int DEFAULT_MAXIMUM_CONNECTIONS = 500;
/** /**
* Configuration option to configure expiration time of * Configuration option to configure expiration time of

2
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
View File

@ -226,7 +226,7 @@ The choice is yours. Generally recovery is better, but sometimes fail-fast is mo
| Property | Default | V2 | Meaning | | Property | Default | V2 | Meaning |
|-----------------------------------------|---------|:----|-------------------------------------------------------| |-----------------------------------------|---------|:----|-------------------------------------------------------|
| `fs.s3a.connection.maximum` | `200` | | Connection pool size | | `fs.s3a.connection.maximum` | `500` | | Connection pool size |
| `fs.s3a.connection.keepalive` | `false` | `*` | Use TCP keepalive on open channels | | `fs.s3a.connection.keepalive` | `false` | `*` | Use TCP keepalive on open channels |
| `fs.s3a.connection.acquisition.timeout` | `60s` | `*` | Timeout for waiting for a connection from the pool. | | `fs.s3a.connection.acquisition.timeout` | `60s` | `*` | Timeout for waiting for a connection from the pool. |
| `fs.s3a.connection.establish.timeout` | `30s` | | Time to establish the TCP/TLS connection | | `fs.s3a.connection.establish.timeout` | `30s` | | Time to establish the TCP/TLS connection |