OK #selfhosting and #linux and other friends here's one for you. My wife's work has Google Drive subscription. My wife wants her own historical archives of her drives so if people delete or damage files they can roll back to a week ago or whatever. we tried this with #rclone but shes got terabyte drives with hundreds of thousands of files in folder hierarchies (biology data from decades of students).
Discussion
Loading...
we created our own oauth token so we dont share the giant rclone default one. but by default the token gets 10 transactions a second and this means it takes days to search the drive to synchronize them. but the drive is in active use so its changing every day. we need enough transaction quota we can rclone sync the entire drive in say 2 hours overnight at 2am.
how do we go about getting that quota upped to like 1000/s?
is that even a thing mortals can do? rather than software developer "partners"
my next option is to do a scheduled google takeout... but thats ugly
Might not be possible. If it's for archiving, is there a reason it needs to sync so fast?
Yeah the drives are changing all day every day (tens of people in the lab using Drive sync to their laptops editing files etc). So if it doesn't sync overnight then it's just continuously out of sync. Like suppose it takes 4 days to run... By the time you've finished you gotta run it again because of hundreds of newly changed files.
We used to just share the drive via SMB. But SMB broke normal kerberos encryption a couple years back, and if you weren't running a samba domain controller you couldn't make it work. Then I tried running NFS4 and got a small number of students Mac's working with that, but different versions of MacOS had different bugs wrt NFS4. So at some point she just moved to Google Drive since people knew how to use it, but it became a giant lock-in bullshit thing, and rclone couldn't help.
we were able to do like 120 transactions per second using rclone's default token... but the rclone site says please use your own token rather than share the quota with thousands of users... So then we got our own token, but it defaults to 10x fewer transactions per second! so things went from syncing over like 2 days, to syncing now over like 20 days??? Screw that.
1+ more replies (not shown)