Commit Graph

61 Commits

Author SHA1 Message Date
Christian Kamm
b2a8ffc577 Abort: Fix crash with early aborts
_chunkCount could be 0, leading to a floating point exception

I also added initializers for several uninitialized integers in the
upload jobs.
2017-10-17 09:44:52 +02:00
Piotr Mrowczynski
e10775d34f Fix paused sync file move issue #5949
Dont abort final chunk immedietally

Use sync and async aborts
2017-10-17 09:44:52 +02:00
Olivier Goffart
ce04a76b17 PropagateUpload: Fix another non-normalized connect that appears while profiling 2017-09-25 12:49:55 +02:00
Helmut K. C. Tessarek
709aa27031 remove qt4 code 2017-09-15 07:11:05 +02:00
Olivier Goffart
3cea550d08 PropagateUpload: Remove the Qt workaround against data corruption
This was only for Qt <= 5.4, which we don't really support anymore.
And even then, the data corruption is happens very seldomly anyway.
2017-07-17 10:12:48 +02:00
Christian Kamm
5d90b48790 PropagateUpload: Put upload error handling in one function 2017-07-12 09:04:27 +02:00
Piotr Mrowczynski
d1e00099dc Classify chunked items correctly. Issue #5850 2017-07-03 14:41:53 +02:00
Christian Kamm
8160963110 Compare the hash of files with identical mtime/size #5589
* For conflicts where mtime and size are identical:

  a) If there's no remote checksum, skip (unchanged)
  b) If there's a remote checksum that's a useful hash, create a
     PropagateDownload job and compute the local hash. If the hashes
     are identical, don't download the file and just update metadata.

* Avoid exposing the existence of checksumTypeId beyond the database
  layer. This makes handling checksums easier in general because they
  can usually be treated as a single blob.

  This change was prompted by the difficulty of producing file_stat_t
  entries uniformly from PROPFINDs and the database.
2017-06-15 13:54:16 +02:00
Christian Kamm
c8d0f788e0 Apply clang-format 2017-05-17 12:26:27 +02:00
Jocelyn Turcotte
b7553d5bdf Upgrade some qCDebug to qCInfo or qCWarning
Use qCInfo for anything that has general value for support and
development. Use qCWarning for any recoverable error and qCCritical
for anything that could result in data loss or would identify a serious
issue with the code.

Issue #5647
2017-05-11 17:22:59 +02:00
Jocelyn Turcotte
4ad190a558 Use Qt logging categories for logging
This gives more insight about the logs and allow setting fine-tuned
logging rules. The categories are set to only output Info by default
so this allows us to provide more concise logging while keeping the
ability to extract more information for a specific category when
developping or debugging customer issues.

Issue #5647
2017-05-11 17:22:59 +02:00
Kaustubh Welankar
70c3514996 Removed Magic Numbers in propagator #5685 (#5705) 2017-04-20 11:14:53 +02:00
Christian Kamm
35af03b2e5 Improve http error messages; cleanup
By default QNetworkReply::errorString() often produces messages like
   "Error downloading <url> - server replied: <reason>"
but the "downloading" part invariably confuses people since the
error might very well have been produced by a PUT request.

This commit produces clearer error messages for HTTP errors.

Additionally:
* Remove some unnecessary null checks from slots connected to
  network job signals and document that these signals never send
  null replies.
* There was a bug where AbstractNetworkJob::_timedout wasn't
  set when derived classes overrode slotTimeout. We now ensure
  it's always set by disallowing overrides of slotTimeout.
  Instead it now calls onTimedOut, which allows custom handling.
* Several subclasses declared errorString, isTimedOut. Move
  these to AbstractNetworkJob.
* Unify handling of OC-ErrorString (via the new, general
  Job::errorString)
* Add documentation in various places.
2017-04-04 09:27:37 +02:00
Christian Kamm
e86499d990 dynamic chunking: cleanup, fixes, improvements
* make target duration a client option instead of a capability
* simplify algorithm for determining chunk size significantly
* preserve chunk size for the whole propagation, not just per upload
* move options to SyncOptions to avoid depending on ConfigFile
  in the propagator
* move chunk-size adjustment to after a chunk finishes, not when
  a new chunk starts
2017-03-28 11:32:10 +02:00
Piotr M
88e0f97798 dynamic chunking prototype 2017-03-27 13:43:39 +02:00
Christian Kamm
4a1a5fa076 AbstractNetworkJob: Improve redirect handling #5555
* For requests:
  - reuse the original QNetworkRequest, so headers and attributes
    are the same as in the original request
  - determine the original http method from the reply and the request
    attributes
  - keep the original request body around such that it can be sent
    again in case the request is redirected

* Simplify the interface that is used for creating new requests in
  AbstractNetworkJob.
2017-03-07 13:18:01 +01:00
Jocelyn Turcotte
f6665ccc81 Pack boolean members in PropagateUploadFileCommon #5457 2017-01-26 17:13:54 +01:00
Jocelyn Turcotte
f1e9be4fa8 Don't use the StopWatch for each job in production builds #5456
This accounted for more than 10% of the peak memory usage while only
providing value through the debugging log.
2017-01-26 17:13:54 +01:00
Jocelyn Turcotte
9db23d4df1 Remove SyncFileItem::_requestDuration #5456
This will save a QElapsedTimer in each job that was used only for
.owncloudsync.log (for which the request duration doesn't bring much
value).
2017-01-26 17:13:54 +01:00
Olivier Goffart
d6fdda8efa ChunkingNG: add '0' padding on the filename (#5476)
The server sorts the chunk by name alphabetically. So if we want to keep
the chunk in order, we need to add a few '0' in front of the chunk name
2017-01-20 16:03:50 +01:00
Christian Kamm
29b39acfbe PropagatorJob: Access propagator through function #5453 2017-01-17 14:47:38 +01:00
ckamm
d76e0ec6d8 Reset stuck chunked uploads eventually #5344 (#5443)
Previously this wasn't happening for errors that were not
NormalErrors because they don't end up in the blacklist.

This revises the resetting logic to be independent of the
error blacklist and make use of UploadInfo::errorCount
instead.

412 errors should reset chunked uploads because they might be
indicative of a checksum error.

Additionally, server bugs might require that additional
errors cause an upload reset. To allow that, a new capability
is added that can be used to advise the client about this.
2017-01-13 14:44:45 +01:00
Olivier Goffart
15f2b911d9 ChunkingNG: remove stale files when resuming 2016-11-04 16:43:01 +01:00
Olivier Goffart
0960058842 Merge remote-tracking branch 'origin/master' into chunking-ng 2016-10-20 09:25:03 +02:00
Markus Goetz
8c5ea8dc90 Propagator: Also upload more in parallel (#5230)
Fix for https://github.com/owncloud/client/issues/4986#issuecomment-227071801

This is before we have bundling at some point.
2016-10-14 14:23:55 +02:00
Olivier Goffart
7c75a39bc1 Chunking-NG: Some fixup after feedback from the pull request #5102 2016-09-10 12:30:14 +02:00
Olivier Goffart
818b5854ce Chunking-NG: Qt4 compile 2016-08-31 10:28:44 +02:00
Olivier Goffart
fad387b6b8 Chunking-Ng: Resume 2016-08-31 10:28:15 +02:00
Olivier Goffart
a1558100b8 WIP: new chunking algorithm
Current limitations of this WiP
 - No resuming implemented yet
 - No parallel chunks
 - Hackish way to get the webdav paths
2016-08-31 10:28:15 +02:00
Olivier Goffart
4f3f642da6 Upload: refactor the upload in two classes so the new chuning can be implemented 2016-08-31 10:28:15 +02:00
Olivier Goffart
d2bde5489f Propagator, remove the QNAM in the name as the alternative has been removed a long time ago 2016-08-31 10:25:06 +02:00
Olivier Goffart
d12c0939b9 Upload: Chunk size can be changed in the config file
Added a new "chunkSize" entry in the General group of the owncloud.cfg
which can be set to the size, in bytes, of the chunks.

This allow user with hude bandwidth to select more optimal chunk size

Issue #4354
2016-01-15 13:16:52 +01:00
Christian Kamm
abf5a5ad1e Propagation: Fix dir <-> file changes propagating to server #4302
* Ensure every time a file becomes a directory or the other way around
  the item is flagged as INSTRUCTION_TYPE_CHANGE.
* Delete the badly-typed entity if necessary in the propagation jobs.
2016-01-08 05:38:08 +01:00
Christian Kamm
a25f094c4c Checksums: Don't reupload if size and checksum are unchanged #3235
* Compute the content checksum (in addition to the optional
  transmission checksum) during upload (.eml files only)

* Add hook to compute and compare the checksum in csync_update

* Add content checksum to database, remove transmission checksum
2015-11-23 11:59:56 +01:00
Christian Kamm
0354289795 Checksums: Improvement in async computation and validation #3735 2015-10-28 09:56:39 +01:00
Phil Davis
f0e17fd9c0 libtypos comment and message typos for master 2015-10-05 09:05:09 +05:45
Daniel Molkentin
66e8aaeabc Use doxygen style everywhere 2015-06-29 18:56:09 +02:00
Daniel Molkentin
0735aa1fbd Structure developer documentation
- rename target "doc-dev"
- group into modules
- move to doc/dev
2015-06-29 18:43:21 +02:00
Olivier Goffart
6b27d2e703 Merge remote-tracking branch 'origin/1.8' 2015-05-26 12:28:31 +02:00
Klaas Freitag
8f277e46d6 TransmissionChecksumming: Remove SyncFileItem dependency. 2015-05-21 15:51:48 +02:00
Klaas Freitag
5b5a636cc1 Make TransmissionChecksumValidator child of the job for auto delete.
That way no explicit memory management is needed as the allocation
is freed when the job (parent) is deleted automatically.
2015-05-20 16:54:04 +02:00
Klaas Freitag
c7f759fedf Propagator: Use the TransmissionChecksumValidator class. 2015-05-19 17:09:40 +02:00
Klaas Freitag
3701fbcbfe PropagateUpload: Add checksum calculation if required by config.
If the config file has an transmissionChecksum entry, a checksum
is added to the PUT requests in a header.
2015-05-19 17:09:39 +02:00
Daniel Molkentin
b54f2b3b31 Merge remote-tracking branch 'origin/1.9'
Conflicts:
	VERSION.cmake
	src/libsync/propagatedownload.cpp
	src/libsync/propagateupload.cpp
2015-05-12 10:03:38 +02:00
Jocelyn Turcotte
1053153ec4 Fix the Qt4 build
Following a6500d8068
Qt4's moc doesn't expand macros and wouldn't evaluate QT_VERSION_CHECK.
2015-05-07 14:49:01 +02:00
Jocelyn Turcotte
a6500d8068 Work around the Qt PUT corruption bug with Qt < 5.4.2 #2425
Since QNonContiguousByteDeviceThreadForwardImpl::reset will
call UploadDevice::reset with a BlockingQueuedConnection, this
allows us to reset the HTTP channel along with its buffers
before they get the chance to be reused with a subsequent request.
2015-05-06 16:56:50 +02:00
Christian Kamm
509b83e73e Time estimation: Avoid a progress reset before finish. #2328
The current algorithm doesn't care much, but resetting progress
to 0 just before completing a job is confusing anyway.
2015-04-22 10:45:41 +02:00
Jocelyn Turcotte
12d7484b85 Pass SyncFileItems by pointer instead of by value
Use a QSharedPointer to keep the same ownership and
continue passing the SyncFileItems as a const& when
ownership isn't taken. This allows sharing the same
allocations between the jobs and the result vectors.

This saves about 20MB of memory (off 120MB) once all
jobs are created.
2015-04-18 14:26:40 +02:00
Jocelyn Turcotte
3c9acdf724 Fix a crash when accessing a dangling UploadDevice pointer #2984
This is largely a guess, but this is the only place where we use
a QIODevice to push data through QNAM and that the QIODevice isn't
a direct child of the QNetworkReply.

Fix the issue by making sure that we don't go back to the event loop
and possibly handle network events between the destruction of the
upload QIODevice and the QNetworkReply, which might lead to QNAM
dereferencing a dangling QIODevice pointer.
2015-03-23 16:14:54 +01:00
Olivier Goffart
97c221d860 PropagateUpload: add a few comments 2015-03-04 12:00:55 +01:00