Commit Graph

336 Commits

Author SHA1 Message Date
Christian Kamm
e774c6c739
Discovery: Ensure selective sync lists are sorted #6958 2020-12-15 10:58:34 +01:00
Olivier Goffart
62ec4c9330
Discovery: Handle the blacklistFiles from the server capabilities
Issue #434

Ideally one could add the blacklist to the exlucde reggexp, but this
is simpler
2020-12-15 10:58:30 +01:00
Olivier Goffart
fd410a5a84
SyncEngine: Ensure that the paths passed to the discovery ends with slashes
This was making the tx.pl test fail
2020-12-15 10:58:28 +01:00
Christian Kamm
b30f79edf6
vfs: Ensure SyncOptions::_vfs is never null
- Create a VfsOff derived class
- Make it a shared pointer shared with Folder::_vfs
2020-12-15 10:58:26 +01:00
Christian Kamm
d5a3604d39
vfs: Reenable local metadata updating of vfs files 2020-12-15 10:58:23 +01:00
Christian Kamm
2b20985875
winvfs: initial work
Done by ckamm and dschmidt
2020-12-15 10:58:21 +01:00
Olivier Goffart
69de2d5180
Sync: optimize by removing setFileRecordMetadata
Inh most case we already have a record from before, so avoid doing a useless
lookup in the database.
In owncloudpropagator.cpp, directories do not have a checksum so no need
to call a function that preserves it
2020-12-15 10:58:19 +01:00
Olivier Goffart
b79e57d1c1
Discovery: fix double emission of finished in case of error 2020-12-15 10:58:17 +01:00
Christian Kamm
ce420d77a8
Discovery: Don't rebuild invalidFilname regex each call 2020-12-15 10:58:13 +01:00
Olivier Goffart
75f66ddaa1
Local discovery: always recurse within touched directory
If the file system watcher tells us a directory was modified, we should
recurse into it because it means it is probably a new directory

Issue #6804
2020-12-15 10:58:11 +01:00
Olivier Goffart
b86e1efc9a
Remove the backup deteciton code which was used for server < 9.1 2020-12-15 10:58:10 +01:00
Olivier Goffart
c009dae1ce
New discovery algorithm: fixups
Adapt reviews from ckamm in https://github.com/owncloud/client/pull/6738#pullrequestreview-164623532

- SyncJournalFileRecord: initialize everything inline
- Add more comments
- And some ENFORCE
2020-12-15 10:58:08 +01:00
Olivier Goffart
c8eff3da2d
New Discovery algorithm: Remove the sync cleanup phase
Since we do not recurse within some directories, many files are not seen.

The stale entry will cleanup by themself as the sync engine try to remove
the files that are already removed.
Should we need to actually do this cleanup, it should be dotected in the
discovery.
2020-12-15 10:58:08 +01:00
Olivier Goffart
52dcfcb166
New Propagation algorithm: Fetch, and emit, the root etag
Remove the feature to concatenate etags as servers that don't
have a root etag are no longer suported
2020-12-15 10:58:07 +01:00
Christian Kamm
6b571b609c
Conflicts: Rename BasePath to InitialBasePath #6709 2020-12-15 10:58:05 +01:00
Olivier Goffart
3a06a7978f
New discovery algo: data finger print 2020-12-15 10:58:04 +01:00
Olivier Goffart
c3f745fa76
Conflict handling: add the OC-ConflictBasePath header
Issue #6702
2020-12-15 10:58:04 +01:00
Olivier Goffart
732069aaa7
SyncEngine: Fix the "direction" of the "all file delted" message when the server is reset
Using the direction of the "first" item is not enough as it might be a directory
which is a UPDATE_META_DATA,  or there can have been a few changes locally as well.

As reported on https://github.com/owncloud/client/issues/6317#issuecomment-412163113
2020-12-15 10:58:02 +01:00
Olivier Goffart
9401273daf
New discovery algorithm: Call FolderDiscovered 2020-12-15 10:58:02 +01:00
Olivier Goffart
a36ed56f01
New Discovery algorithm: Refactor a bit the way the signal are emited 2020-12-15 10:58:02 +01:00
Olivier Goffart
b6487181c6
New Discovery Phase: fix a few FIXME
Some FIXME were already fixed.
2020-12-15 10:58:02 +01:00
Olivier Goffart
ca32eb1cf7
More cleanup of csync remains 2020-12-15 10:58:01 +01:00
Olivier Goffart
ac24cdbde6
New Discovery Algo: Permsission check 2020-12-15 10:58:01 +01:00
Olivier Goffart
0a6d2d0f46
Remove csync_update and csync_reconcile
I guess some other csync utilities can also be remove dnow but
that will be for later
2020-12-15 10:58:00 +01:00
Olivier Goffart
75a1f2d089
Convert p7.pl to a C++ test
This is just a translation of test/scripts/txpl/t7.pl to C++ using the test
framework.
2020-12-15 10:58:00 +01:00
Olivier Goffart
f4a5167812
New Discovery Algo: Support the DatabaseAndFilesystem mode for local discovery 2020-12-15 10:57:59 +01:00
Olivier Goffart
710934bdbd
New disco algo: Fix TestSyncFileStatusTracker 2020-12-15 10:57:59 +01:00
Olivier Goffart
35e40b58ca
New Disco algorithm: Type change (file to dir) 2020-12-15 10:57:59 +01:00
Olivier Goffart
22d989e272
New discovery algorithm: Fix directory deletion
TestAllFilesDeleted passes
2020-12-15 10:57:59 +01:00
Olivier Goffart
d54e00488a
New Discovery algorithm
Some error handling. In particular for the case where there is a conflict
between files and directories.

SyncEngineTest and SyncMoveTest passes
2020-12-15 10:57:58 +01:00
Olivier Goffart
a384a2d1cb
New Discovery algorithm: Handle of move within a moved directory 2020-12-15 10:57:58 +01:00
Olivier Goffart
f9a9be59e6
New discovery algo: Remote move
TestSyncMove::testRemoteChangeInMovedFolder
2020-12-15 10:57:57 +01:00
Olivier Goffart
92ddc60900
Handle Encoding Problems
TestSyncEngine now passes
2020-12-15 10:57:57 +01:00
Olivier Goffart
501da58b10
Ignore Hidden Files 2020-12-15 10:57:57 +01:00
Olivier Goffart
8bf69cf0cd
Port the invalid napme regexp
TestSyncEngine::testInvalidFilenameRegex
2020-12-15 10:57:57 +01:00
Olivier Goffart
e934f6b27b
New discovery algo
Make TestSyncEngine::testSelectiveSyncBug pass
2020-12-15 10:57:57 +01:00
Olivier Goffart
ec681ab2a5
New discovery algorithm: Initial work.
SyncEngineTest testFileDownload is passing
2020-12-15 10:57:56 +01:00
Kevin Ottens
ba0d465e47
Revert "Merge pull request #1454 from nextcloud/syncjournal-del-prio"
This reverts commit d9fd9cfef2, reversing
changes made to 2dcf594fc6.
2020-12-15 10:57:56 +01:00
Olivier Goffart
cf3846c565
csync: refactor csync_s::error_string to avoid valgrind error
The problem here is that we were sometimes allocating the error_string with
qstrdup, which need to be released with delete[] and not free().

Simplify the code by using QString instead.

```
==7230== Mismatched free() / delete / delete []
==7230==    at 0x4C2E10B: free (vg_replace_malloc.c:530)
==7230==    by 0x57C2321: csync_s::reinitialize() (csync.cpp:247)
==7230==    by 0x548130F: OCC::SyncEngine::finalize(bool) (syncengine.cpp:1212)
==7230==    by 0x5481223: OCC::SyncEngine::handleSyncError(csync_s*, char const*) (syncengine.cpp:746)
==7230==    by 0x5483E78: OCC::SyncEngine::slotDiscoveryJobFinished(int) (syncengine.cpp:965)
==7230==    by 0x5495E34: QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<int>, void, void (OCC::SyncEngine::*)(int)>::call(void (OCC::SyncEngine::*)(int), OCC::SyncEngine*, void**) (qobjectdefs_impl.h:134)
==7230==    by 0x5495D92: void QtPrivate::FunctionPointer<void (OCC::SyncEngine::*)(int)>::call<QtPrivate::List<int>, void>(void (OCC::SyncEngine::*)(int), OCC::SyncEngine*, void**) (qobjectdefs_impl.h:167)
==7230==    by 0x5495CB5: QtPrivate::QSlotObject<void (OCC::SyncEngine::*)(int), QtPrivate::List<int>, void>::impl(int, QtPrivate::QSlotObjectBase*, QObject*, void**, bool*) (qobjectdefs_impl.h:396)
==7230==    by 0xA9BF2E1: QObject::event(QEvent*) (in /usr/lib/libQt5Core.so.5.11.0)
==7230==    by 0x64BE983: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib/libQt5Widgets.so.5.11.0)
==7230==    by 0x64C625A: QApplication::notify(QObject*, QEvent*) (in /usr/lib/libQt5Widgets.so.5.11.0)
==7230==    by 0xA994BC8: QCoreApplication::notifyInternal2(QObject*, QEvent*) (in /usr/lib/libQt5Core.so.5.11.0)
==7230==  Address 0x225b2640 is 0 bytes inside a block of size 50 alloc'd
==7230==    at 0x4C2DC6F: operator new[](unsigned long) (vg_replace_malloc.c:423)
==7230==    by 0xA7E8FC8: qstrdup(char const*) (in /usr/lib/libQt5Core.so.5.11.0)
==7230==    by 0x53F5750: OCC::DiscoveryJob::remote_vio_opendir_hook(char const*, void*) (discoveryphase.cpp:666)
==7230==    by 0x57E1278: csync_vio_opendir(csync_s*, char const*) (csync_vio.cpp:39)
==7230==    by 0x57D718F: csync_ftw(csync_s*, char const*, int (*)(csync_s*, std::unique_ptr<csync_file_stat_s, std::default_delete<csync_file_stat_s> >), unsigned int) (csync_update.cpp:674)
==7230==    by 0x57C1B05: csync_update(csync_s*) (csync.cpp:109)
==7230==    by 0x53F5BCC: OCC::DiscoveryJob::start() (discoveryphase.cpp:718)
==7230==    by 0x54B8F74: OCC::DiscoveryJob::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) (moc_discoveryphase.cpp:494)
==7230==    by 0xA9BF2E1: QObject::event(QEvent*) (in /usr/lib/libQt5Core.so.5.11.0)
==7230==    by 0x64BE983: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib/libQt5Widgets.so.5.11.0)
==7230==    by 0x64C625A: QApplication::notify(QObject*, QEvent*) (in /usr/lib/libQt5Widgets.so.5.11.0)
==7230==    by 0xA994BC8: QCoreApplication::notifyInternal2(QObject*, QEvent*) (in /usr/lib/libQt5Core.so.5.11.0)
==7230==
```
2020-12-15 10:57:55 +01:00
Christian Kamm
aa6f5f59c4
Rename Placeholders to Virtual Files in code #6531 2020-12-15 10:57:53 +01:00
Christian Kamm
ddeeecd061
Rename "placeholder" feature 2020-12-15 10:57:53 +01:00
Christian Kamm
40f2303166
SyncOptions: Add default placeholder suffix
Otherwise each test has to set this up anew.
2020-12-15 10:57:49 +01:00
Olivier Goffart
91f53521ae
Placeholder: The extension is now a branding option 2020-12-15 10:57:49 +01:00
Christian Kamm
b1de184bc8
Placeholders: Safe migration to older client versions
Now the db entries for placeholders will have the full placeholder
paths. That way older clients will, on remote discovery, delete the
placeholders and download the real files.
2020-12-15 10:57:48 +01:00
Olivier Goffart
1049fb74d9
Placeholders: Move the placeholder option from the account to the folder 2020-12-15 10:57:48 +01:00
Christian Kamm
f5cf03b6a6
On-demand downloading: Placeholder-file based prototype
- Controled by an option.
- New remote files start out as ItemTypePlaceholder, are created with a
  .owncloud extension.
- When their db entry is set to ItemTypePlaceholderDownload the next
  sync run will download them.
- Files that aren't in the placeholder state sync as usual.
- See test cases in testsyncplaceholders.

Missing:
- User ui for triggering placeholder file download
- Maybe: Going back from file to placeholder?
2020-12-15 10:57:43 +01:00
Christian Kamm
00e901f5a7
SocketAPI: Introduce conflict resolution actions #6252
For conflicts generally as well as new files in read-only directories
the context menu will now present delete and move options.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-10-22 16:40:10 +02:00
Kevin Ottens
80cc196f6f Enable bugprone-narrowing-conversions clang-tidy check
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-09-01 06:37:03 +00:00
Kevin Ottens
c8ea848596 Enable the bugprone-branch-clone clang-tidy check
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-09-01 06:37:03 +00:00
Kevin Ottens
866ffc2a6b Enable the modernize-loop-convert check on clang-tidy
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-13 12:55:14 +00:00