49df838995
Contains:
HADOOP-16474. S3Guard ProgressiveRenameTracker to mark destination
dirirectory as authoritative on success.
HADOOP-16684. S3guard bucket info to list a bit more about
authoritative paths.
HADOOP-16722. S3GuardTool to support FilterFileSystem.
This patch improves the marking of newly created/import directory
trees in S3Guard DynamoDB tables as authoritative.
Specific changes:
* Renamed directories are marked as authoritative if the entire
operation succeeded (HADOOP-16474).
* When updating parent table entries as part of any table write,
there's no overwriting of their authoritative flag.
s3guard import changes:
* new -verbose flag to print out what is going on.
* The "s3guard import" command lets you declare that a directory tree
is to be marked as authoritative
hadoop s3guard import -authoritative -verbose s3a://bucket/path
When importing a listing and a file is found, the import tool queries
the metastore and only updates the entry if the file is different from
before, where different == new timestamp, etag, or length. S3Guard can get
timestamp differences due to clock skew in PUT operations.
As the recursive list performed by the import command doesn't retrieve the
versionID, the existing entry may in fact be more complete.
When updating an existing due to clock skew the existing version ID
is propagated to the new entry (note: the etags must match; this is needed
to deal with inconsistent listings).
There is a new s3guard command to audit a s3guard bucket/path's
authoritative state:
hadoop s3guard authoritative -check-config s3a://bucket/path
This is primarily for testing/auditing.
The s3guard bucket-info command also provides some more details on the
authoritative state of a store (HADOOP-16684).
Change-Id: I58001341c04f6f3597fcb4fcb1581ccefeb77d91
78 lines
2.7 KiB
XML
78 lines
2.7 KiB
XML
<!--
|
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
|
contributor license agreements. See the NOTICE file distributed with
|
|
this work for additional information regarding copyright ownership.
|
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
|
(the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
-->
|
|
<FindBugsFilter>
|
|
|
|
<!-- same code as in FileSystem is triggering the same warning. -->
|
|
<Match>
|
|
<Class name="org.apache.hadoop.fs.s3native.S3xLoginHelper" />
|
|
<Method name="checkPath" />
|
|
<Bug pattern="ES_COMPARING_STRINGS_WITH_EQ" />
|
|
</Match>
|
|
<!-- Redundant null check makes code clearer, future-proof here. -->
|
|
<Match>
|
|
<Class name="org.apache.hadoop.fs.s3a.S3AFileSystem" />
|
|
<Method name="s3Exists" />
|
|
<Bug pattern="RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE" />
|
|
</Match>
|
|
|
|
<!--
|
|
This extends the serializable S3Object, so findbug checks
|
|
serializability. It is never serialized however, so its
|
|
warnings are false positives.
|
|
-->
|
|
<Match>
|
|
<Class name="org.apache.hadoop.fs.s3a.InconsistentS3Object" />
|
|
<Bug pattern="SE_TRANSIENT_FIELD_NOT_RESTORED" />
|
|
</Match>
|
|
<Match>
|
|
<Class name="org.apache.hadoop.fs.s3a.InconsistentS3Object" />
|
|
<Bug pattern="SE_NO_SERIALVERSIONID" />
|
|
</Match>
|
|
|
|
<!--
|
|
findbugs gets confused by lambda expressions in synchronized methods
|
|
and considers references to fields to be unsynchronized.
|
|
As you can't disable the methods individually, we have to disable
|
|
them for the entire class.
|
|
-->
|
|
<Match>
|
|
<Class name="org.apache.hadoop.fs.s3a.S3AInputStream"/>
|
|
<Bug pattern="IS2_INCONSISTENT_SYNC"/>
|
|
</Match>
|
|
<!--
|
|
findbugs reporting RV ignored. Not true.
|
|
"Return value of S3AReadOpContext.getReadInvoker() ignored,
|
|
but method has no side effect"
|
|
-->
|
|
<Match>
|
|
<Class name="org.apache.hadoop.fs.s3a.S3AInputStream"/>
|
|
<Method name="reopen"/>
|
|
<Bug pattern="RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT"/>
|
|
</Match>
|
|
<Match>
|
|
<Class name="org.apache.hadoop.fs.s3a.S3AFileSystem"/>
|
|
<Method name="openFileWithOptions"/>
|
|
<Bug pattern="RV_RETURN_VALUE_IGNORED_BAD_PRACTICE"/>
|
|
</Match>
|
|
<Match>
|
|
<Class name="org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo"/>
|
|
<Method name="run"/>
|
|
<Bug pattern="SF_SWITCH_FALLTHROUGH"/>
|
|
</Match>
|
|
|
|
</FindBugsFilter>
|