big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HADOOP-19279. ABFS: Disabling Apache Http Client as Default Http Client for ABFS Driver(#7055)

Browse Source 
As part of work done under HADOOP-19120 [ABFS]: ApacheHttpClient adaptation as network library - ASF JIRA
Apache HTTP Client was introduced as an alternative Network Library that can be used with ABFS Driver. Earlier JDK Http Client was the only supported network library.

Apache HTTP Client was found to be more helpful in terms of controls and knobs it provides to manage the Network aspects of the driver better. Hence, the default Network Client was made to be used with the ABFS Driver.

Recently while running scale workloads, we observed a regression where some unexpected wait time was observed while establishing connections. A possible fix has been identified and we are working on getting it fixed.
There was also a possible NPE scenario was identified on the new network client code.

Until we are done with the code fixes and revalidated the whole Apache client flow, we would like to make JDK Client as default client again. The support will still be there, but it will be disabled behind a config.

Contributed by: manika137
This commit is contained in:
manika137 2024-09-21 03:08:56 +05:30 committed by GitHub
parent ee2e5ac4e4
commit 6bcc254123
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java
View File

@ -169,7 +169,7 @@ public final class FileSystemConfigurations {
public static final long THOUSAND = 1000L; public static final long THOUSAND = 1000L;
public static final HttpOperationType DEFAULT_NETWORKING_LIBRARY public static final HttpOperationType DEFAULT_NETWORKING_LIBRARY
= HttpOperationType.APACHE_HTTP_CLIENT; = HttpOperationType.JDK_HTTP_URL_CONNECTION;
public static final int DEFAULT_APACHE_HTTP_CLIENT_MAX_IO_EXCEPTION_RETRIES = 3; public static final int DEFAULT_APACHE_HTTP_CLIENT_MAX_IO_EXCEPTION_RETRIES = 3;

4
hadoop-tools/hadoop-azure/src/site/markdown/abfs.md
View File

@ -878,8 +878,8 @@ ABFS Driver can use the following networking libraries:
The networking library can be configured using the configuration `fs.azure.networking.library` The networking library can be configured using the configuration `fs.azure.networking.library`
while initializing the filesystem. while initializing the filesystem.
Following are the supported values: Following are the supported values:
- `APACHE_HTTP_CLIENT` : Use Apache HttpClient [Default] - `JDK_HTTP_URL_CONNECTION` : Use JDK networking library [Default]
- `JDK_HTTP_URL_CONNECTION` : Use JDK networking library - `APACHE_HTTP_CLIENT` : Use Apache HttpClient
#### <a href="ahc_networking_conf"></a>ApacheHttpClient networking layer configuration Options: #### <a href="ahc_networking_conf"></a>ApacheHttpClient networking layer configuration Options: