SyncBack Pro - Temporary Tracking File

jmgroud Posted messages 41 Status Member -  
jmgroud Posted messages 41 Status Member -

Hello

Context

I have been using SyncBack Pro for many years and I have just set up file hashing in order to try to implement file renaming recognition.

So far, renaming a file on the source or target platform means that SyncBack generates a deletion (old file name) and the creation of a new file (new name). Everything is working perfectly; it’s quite “logical,” but not optimized for transfers, as a rename results in the complete download of the file (even though it is already present).

After modifying the hashing calculation parameter, I restarted the synchronization and I now have an additional step that consists of calculating the hash values for all the files involved in this backup.

Problem

I have about 20,000 files, some of which exceed 10 GB, so it takes "some time" to perform this operation (which will certainly be faster on the next run).

In order to estimate the total time SyncBack takes to do all this work, do you have any idea about the temporary tracking file that is stored on the disk during the calculation? I have no estimation information in the interface nor the number of files processed (and/or to be processed), making it impossible to estimate.

Thank you in advance for your responses.

Best regards


3 answers

  1. brucine Posted messages 24845 Registration date   Status Member Last intervention   4 169
     

    Hello,

    I have a poor understanding of the context: the essence of an incremental backup is to copy from the source to the target the files that have a more recent timestamp and to delete from the target those that are no longer found in the source. The procedure is quite fast except for the first time, even in the presence of a large volume of files (personally, I don’t use Syncback for this purpose but a simple command line).

    Obviously, if a user has renamed a file in the source, even without modifying it, the result will be that this file in its old version will disappear from both the source and the target. The most abnormal situation occurs when this file has not been renamed but copied under a different name; the normal course of action in this situation seems to me to restrict the user from such a procedure.

    Now, if we have a good reason to keep, say, the last 3 daily backups of the source, it is not an incremental backup that needs to be done, but a script that will automatically copy the latest source and then delete from the target all backup folders older than 3 days, even compressing them on the fly to save space, though this will still take up considerable space.

    In any case, checking not only the date attribute but also calculating the hash of each file will take considerable time.

    Moreover, as far as I know, which I understand poorly, a file does not by default have a hash fingerprint that needs to be assigned to it, only those that can be downloaded from the Internet seeming to be concerned to ensure they match the original, and should this possible verification not take place when the file is created in the source and not during its transfer, especially since this fingerprint primarily concerns the content of the file but not its name?

    0
  2. jmgroud Posted messages 41 Status Member 18
     

    Thank you for your response brucine

    There are other software options, I know, and I also use others for different reasons.

    I won't explain here the reasons for which I would use SyncBackPro profiles to exchange with other machines. I have about twenty profiles to manage different situations on various Windows and Linux machines in different geographic locations. I would spend too much time on it...

    I know very well how long it takes for a machine and "a lot" of time to calculate an MD5 or other (the transfer also takes time...) and I know that the same algorithms must, of course, be used on both sides to "trust" this method.

    That said, I find SyncBackPro very well designed even if nothing is ever perfect, of course.
    It has many options, not always easy to master, but it allows for a very wide range of possibilities for file exchanges between two machines.

    I am therefore testing the hashing option which, for some profiles, seems interesting.
    There are still questions pending regarding the calculation methodology but also the system for storing values (from the local machine on which SyncBack is running, but also and especially the remote one since SyncBack is not installed on that machine).

    I wouldn't be surprised to see SyncBack manage all these aspects in a "very intelligent" way to optimize security and execution time of the profiles, but I will see...

    I will therefore return to my initial question: How can I know the current status of the profile being executed?

    I indeed thought about the possible presence of a tracking file stored on disk (hoping that SyncBack doesn't manage everything in memory...) but maybe there is another way...

    Jean-Marie


    0
    1. brucine Posted messages 24845 Registration date   Status Member Last intervention   4 169
       

      The log files are, as you know, located at C:\Users\xxx\AppData\Local\2BrightSparks\SyncBack, but there is no evidence that they are timestamped or that operations are indeed only redirected to the console, nor is there any indication, even if the file processing log were timestamped, that the operations are sequential (processing all attributes on one side, hashing on the other), otherwise there is no way to know the specific time dedicated to this last operation (unless you decompile the software...).

      We can obviously know, but without predicting, how long a regular profile backup took when running only that one (or for all by the difference between the log creation time and its last modification time), but I suppose that's not what you want to know.

      0
  3. jmgroud Posted messages 41 Status Member 18
     

    Actually, I no longer remembered by heart where the log files were located... Thanks for the reminder, I will go check, you never know...

    Knowing the exact duration in advance is anyway not possible, but it will only be an estimate after calculating the time taken for a certain number of GB locally (I will do a rule of three).

    Moreover, this will not be sufficient anyway because I am currently only considering the computing time for my local files.
    I am not yet considering the files on the target machine because I don’t know how SyncBack will proceed. I imagine a transfer to a temporary file on my SyncBack machine (with timestamp information), then a hash calculation before storage in its database, but nothing is certain of course...

    I will see all this in "a few hours or days or weeks" ;-) depending on the total time taken for my 20,000 local files...
    In any case, it’s quite a test...
    I am willing to wait a few weeks, but if I have to count in "months," I will think about it with my partners to see if it’s all worth it :-)

    Thanks anyway for your very interesting remarks.

    For your information, the SyncBack log folder does not seem to contain the "temporary logs" of the running profiles.
    I searched all the files (using the Windows search option with "*") in the folder and sorted by modification date and found only files dated yesterday (my completed profiles).
    Everything must be stored elsewhere...


    0