File syncronization timeout

Again one of the things I see asked often.

My answer usually:

“Restarting the process will not help you.

I have the same problem. The problem is that the syncronization functionality is not optimized for high amounts of files, and when you click synchronize the application takes ALL the files from ALL portals, caches that list (my problem came at around 10 000 files on one host), and for each file extracts that list from the cache, deserializes it and creates a new object (well copy) of the one from cache (also a veeeery expensive operation).

All of this works for a small amount of files, but for a large one (like I had on one host) it lasted for me for about 30-40 minutes each time for the host, and for a portal syncronize (for a portal which had 1200 files) it took about 5-6 minutes.”

which is usually followed by another one:

“Well I found it best to do 2 things:

1. Not use the snycronization at all - do everything through file manager (so without FTP)

2. Re-program the syncronization routine - I’ve actually done this, and have cut times on mine 1200 files from 6 minutes to 8-10 seconds of snycronization. The only problem is that it contains one bug to which I haven’t yet had time to give to solve it (some files still remain on the file system and do not end up in the db, I’m not sure that this is my bug, maybe even it’s present in the core syncronization itself) 

hope this helps “

I have pasted this here so that:

a) People googling can find it (I know I’ve spent some time searching the internet)
b) So that I can reference it in my answers to the forums