big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,212 Commits 2 Branches 2 Tags 264 MiB
Java 92.9%
C++ 2.8%
C 1.8%
JavaScript 1.1%
Shell 0.5%
Other 0.6%
e31a636e92
Go to file
Sneha Vijayarajan e31a636e92
HADOOP-17215: Support for conditional overwrite. 
Contributed by Sneha Vijayarajan

DETAILS:

    This change adds config key "fs.azure.enable.conditional.create.overwrite" with
    a default of true.  When enabled, if create(path, overwrite: true) is invoked
    and the file exists, the ABFS driver will first obtain its etag and then attempt
    to overwrite the file on the condition that the etag matches. The purpose of this
    is to mitigate the non-idempotency of this method.  Specifically, in the event of
    a network error or similar, the client will retry and this can result in the file
    being created more than once which may result in data loss.  In essense this is
    like a poor man's file handle, and will be addressed more thoroughly in the future
    when support for lease is added to ABFS.

TEST RESULTS:

    namespace.enabled=true
    auth.type=SharedKey
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 42
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

    namespace.enabled=true
    auth.type=OAuth
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
2020-09-19 01:28:44 +00:00
.github HADOOP-15184. Add GitHub pull request template. (#1419) 2019-09-11 11:10:11 +09:00
dev-support HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277) 2020-09-16 16:23:57 +09:00
hadoop-assemblies HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
hadoop-build-tools HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-18 14:33:43 +05:30
hadoop-client-modules YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars (#2075) 2020-06-17 09:26:41 +05:30
hadoop-cloud-storage-project HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) 2020-07-22 13:40:20 +09:00
hadoop-common-project HADOOP-15136. Correct typos in filesystem.md (#2314) 2020-09-18 18:28:14 +01:00
hadoop-dist Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-hdfs-project HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 2020-09-18 17:28:11 +05:30
hadoop-mapreduce-project [UI1] Provide a way to hide Tools section in Web UIv1. Contributed by Andras Gyori 2020-08-31 16:00:39 +02:00
hadoop-maven-plugins Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-minicluster HDFS-15331. Remove invalid exclusions that minicluster dependency on HDFS (#1996). Contributed by Wanqiang Ji 2020-05-06 02:06:38 +05:30
hadoop-project YARN-10386. Create new JSON schema for Placement Rules. Contributed by Peter Bacsko and Benjamin Teke 2020-08-28 17:09:31 +02:00
hadoop-project-dist Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-tools HADOOP-17215: Support for conditional overwrite. 2020-09-19 01:28:44 +00:00
hadoop-yarn-project YARN-9333. TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently. Contributed by Peter Bacsko 2020-09-17 19:57:19 +02:00
licenses HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
licenses-binary HADOOP-15993. Upgrade Kafka to 2.4.0 in hadoop-kafka module. (#1796) 2020-01-09 16:24:58 +09:00
.asf.yaml HADOOP-17234. Add .asf.yaml to allow Github to Jira integration. (#2253). Contributed by Ayush Saxena. 2020-08-28 17:22:46 +05:30
.gitattributes HADOOP-13598. Add eol=lf for unix format files in .gitattributes. Contributed by Yiqun Lin. 2016-09-14 11:14:31 +09:00
.gitignore YARN-10407. Add phantomjsdriver.log to gitignore. (#2244) 2020-09-01 10:44:55 +09:00
BUILDING.txt HADOOP-17240. Fix wrong command line for setting up CentOS 8. (#2268) 2020-09-02 15:12:03 +09:00
Jenkinsfile HADOOP-17262. Switch to Yetus main branch. (#2304) 2020-09-15 11:12:00 +09:00
LICENSE-binary HADOOP-17049. javax.activation-api and jakarta.activation-api define overlapping classes (#2027) 2020-05-22 11:19:11 +09:00
LICENSE.txt YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-12-09 01:25:10 +00:00
NOTICE-binary HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
NOTICE.txt HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
pom.xml upate the hadoop.version property in the root pom.xml and hadoop.assemblies.version in hadoop-project/pom.xml (see HADOOP-15369) 2020-03-29 23:44:20 +05:30
README.txt HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
start-build-env.sh HADOOP-16849. start-build-env.sh behaves incorrectly when username is numeric only. Contributed by Jihyun Cho. 2020-02-12 14:06:23 +09:00

README.txt 

For the latest information about Hadoop, please visit our website at:

   http://hadoop.apache.org/

and our wiki, at:

   https://cwiki.apache.org/confluence/display/HADOOP/