Commit Graph

259 Commits

Author SHA1 Message Date
ansuz
4495d99e49 use the standard method of checking whether metadata is valid 2020-12-08 16:45:57 +05:30
ansuz
6bc103be5f delay cache eviction each time the metadata is requested 2020-12-08 16:43:50 +05:30
ansuz
d4f94f3091 briefly cache server metadata in-memory to avoid repeated reads 2020-12-08 16:34:28 +05:30
ansuz
820f2a0054 Merge branch 'soon' into staging 2020-12-08 11:14:53 +05:30
ansuz
93d56e5e3d profile worker commands by total running time in seconds 2020-12-08 10:44:46 +05:30
ansuz
cd77b5ffc8 Merge branch 'soon' into staging 2020-11-19 13:18:08 +05:30
yflory
bd4b44476d Archive and restore UI in the admin panel 2020-11-18 16:49:40 +01:00
ansuz
e185008846 complete uploads in child processes
also fix a nasty race condition for unowned file uploads
2020-11-18 18:26:21 +05:30
ansuz
1f1b40b83d move some implictly global state to env.js 2020-11-03 16:48:31 +05:30
ansuz
5cf8ca70e6 don't pin falsey document ids 2020-11-03 16:48:31 +05:30
ansuz
034472d3dd move some implictly global state to env.js 2020-10-27 12:47:15 +05:30
ansuz
f7bd3bdc23 don't pin falsey document ids 2020-10-26 17:34:41 +05:30
ansuz
fbc9edd795 rename latest admin commands and implement metadata getter/remover 2020-10-20 17:12:26 +05:30
ansuz
488ec93ece allow expert admins to get and clear cached channel indices 2020-10-20 17:07:55 +05:30
ansuz
faa7ebf399 Fix some bugs with integrated eviction
1. implement SET_LAST_EVICTION as an admin command, not a decree
2. expect a return value from Env.evictInactive and expose it via Env.evictionReport
2020-10-15 13:15:39 +05:30
ansuz
a2c0d2165b implement a few more admin decrees 2020-10-13 11:36:30 +05:30
ansuz
abd84665ae lint compliance 2020-10-13 11:00:45 +05:30
ansuz
6ec5171518 add support for changing a few more Env parameters at runtime 2020-10-13 10:56:40 +05:30
ansuz
f2ec9cbe33 load premium and customLimits to avoid evicting them even if inactive 2020-10-12 19:18:12 +05:30
ansuz
854e4c06ff report lastEviction in an admin INSTANCE_STATUS 2020-10-12 17:52:15 +05:30
ansuz
471a9a33ac take note of a minor error with pin RPCs 2020-10-12 17:51:38 +05:30
ansuz
6d13a785c6 downgrade a non-critical XXX to a FIXME 2020-10-12 17:42:41 +05:30
ansuz
b5d3a10dc2 guard against possible typeErrors from unvalidated config input 2020-10-09 13:28:13 +05:30
ansuz
f4f803ccd9 reuse Keys.canonicalize to accept any pubkey format 2020-10-08 19:07:00 +05:30
ansuz
914139dbfa lint compliance 2020-10-08 19:07:00 +05:30
yflory
51e1b9b043 Fix custom limit not applied when using a raw edPublic 2020-10-08 19:07:00 +05:30
ansuz
aa35e092f7 update an inaccurate comment 2020-10-08 19:07:00 +05:30
ansuz
928b7e3583 implement GET_LIMITS admin rpc
...and label stored usage limits according to their source
2020-10-08 19:07:00 +05:30
ansuz
b62d4387a6 drop support for SET_DEFAULT_STORAGE_LIMIT
it's redundant now that we support persistent admin decrees
and it was never exposed via the UI anyway
2020-10-08 19:07:00 +05:30
ansuz
0c5db31fbc refactor quota logic for external usage 2020-10-08 19:07:00 +05:30
ansuz
bd6234c5bc lint compliance and some notes 2020-10-08 19:07:00 +05:30
ansuz
1781ee2585 implement 'UPDATE_DEFAULT_STORAGE' as a decree 2020-10-08 19:07:00 +05:30
ansuz
7aa7d5978f implement persistence of config changes from admin panel 2020-10-08 19:07:00 +05:30
ansuz
fcfcb3dca6 fix history trim for channels with access lists 2020-10-07 18:44:24 +05:30
ansuz
6ada5fd751 always consider premium accounts to be active 2020-09-29 18:48:23 +05:30
ansuz
d9219f4ecc add a note for later 2020-09-25 14:27:22 +05:30
ansuz
5749b6fd7b add an admin RPC to archive documents 2020-09-25 14:24:26 +05:30
ansuz
95778cc7c9 respond with ENOENT instead of E_NO_OWNERS when archiving a file that does not exist 2020-07-23 09:30:04 -04:00
yflory
04af753a81 lint compliance 2020-07-01 12:10:12 +02:00
ansuz
ebcc9a069b add server-side support for the new format of public signing keys 2020-06-25 13:14:26 -04:00
ansuz
d35b015c3f log some info on successful file uploads 2020-04-28 17:12:03 -04:00
ansuz
9ed26cfeb0 lint compliance 2020-04-15 12:40:18 -04:00
ansuz
74567e0cf4 return process memory usage in GET CACHE STATS RPC 2020-04-14 15:53:58 -04:00
ansuz
e527867e2e queue blob and channel deletions per-user 2020-04-09 16:29:52 -04:00
ansuz
bd3e18d1a8 only cache a user's pinned channels if there are no errors when loading them 2020-04-08 12:08:26 -04:00
ansuz
bef18a9320 hotfix 2020-04-08 10:28:26 -04:00
ansuz
3d8e78a268 fix overly broad config interpretation for daily check 2020-03-30 18:27:14 -04:00
ansuz
e6ec891d9a improve admin rpc for diagnosing memory used by the cache 2020-03-30 18:26:04 -04:00
ansuz
6d0dee979a allow admins to inspect index/metadata caches 2020-03-27 19:59:26 -04:00
ansuz
b0179eaad9 drop XXX note 2020-03-27 14:25:07 -04:00
ansuz
9dbd32758a improve worker logging and move blob deletion to worker processes as well 2020-03-27 13:38:27 -04:00
ansuz
cb53bd1c15 lint compliance 2020-03-26 14:44:37 -04:00
ansuz
14feef1757 calculate pin list checksums in a worker 2020-03-25 18:37:42 -04:00
ansuz
70a0d4efb4 move more database reads into the database worker 2020-03-25 17:43:57 -04:00
ansuz
4ba36a9173 load user pins in the database worker 2020-03-25 11:39:14 -04:00
ansuz
471e374533 compute metadata in the same child process that builds indexes 2020-03-24 17:43:15 -04:00
ansuz
479b76f848 lint compliance 2020-03-24 17:40:07 -04:00
ansuz
40251948d4 check authenticated rpc signatures in separate threads 2020-03-24 14:31:40 -04:00
ansuz
90899aa2f4 start reviewing and removing XXX notes 2020-03-24 13:03:49 -04:00
ansuz
631ea54b49 lint compliance 2020-03-17 15:40:06 -04:00
ansuz
67dd4a608c Merge remote-tracking branch 'origin/scaling' into scaling2 2020-03-17 15:36:58 -04:00
ansuz
b2fcde87d8 implement an admin command to update the default storage limit without a restart 2020-03-17 10:11:52 -04:00
yflory
0d636dabc9 Check signature for history keeper in a different process 2020-03-17 13:29:53 +01:00
ansuz
2a9780abcf lint compliance 2020-03-16 12:29:26 -04:00
ansuz
2d47e7e2cf implement an admin rpc to fetch ulimit data 2020-03-12 12:09:44 -04:00
ansuz
81c7416095 lint compliance 2020-03-12 11:40:46 -04:00
ansuz
f644c24362 add an admin-rpc to count the number of open files 2020-03-12 11:35:16 -04:00
ansuz
1230057970 disable IS_PAD_PINNED 2020-03-12 11:31:08 -04:00
ansuz
6d8cdca7a3 add 'rejected' field to metadata when getMetadata requests are blocked 2020-03-05 12:09:18 -05:00
ansuz
9d8bb43d03 finally untangle metadata and index caches 2020-03-04 12:56:33 -05:00
ansuz
32d769447a begin standardizing our method of streaming lines from files 2020-03-04 11:38:14 -05:00
ansuz
b126e4456b simplify quota logic 2020-03-03 16:55:40 -05:00
ansuz
170aa6d47e clean up a few tasks related to allow lists 2020-03-03 15:52:49 -05:00
ansuz
2f00ff3278 simplify check-in deactivation 2020-02-28 14:24:53 -05:00
ansuz
c26560e3f9 commit stashed admin-rpc refactor 2020-02-27 15:00:31 -05:00
ansuz
59ad80d7f1 support larger upload sizes for users with premium accounts 2020-02-26 13:09:11 -05:00
ansuz
f951951077 merge communities-allow-list and lint compliance 2020-02-25 09:48:41 -05:00
ansuz
021a6b361f remove a bogus XXX note 2020-02-21 14:33:28 -05:00
ansuz
7ee8778210 kick unauthorized users from restricted channels
and update dependencies
2020-02-21 13:05:24 -05:00
ansuz
fe0e02a421 restrict access to metadata 2020-02-21 10:31:10 -05:00
ansuz
791aad53f2 WIP allow list changes 2020-02-21 08:18:05 -05:00
ansuz
f579c9b059 semi-functional allow-list implementation in historyKeeper 2020-02-19 14:22:12 -05:00
ansuz
0edcdcb1b8 oups! fix a missing callback that's been here for a while... 2020-02-18 16:15:05 -05:00
ansuz
27e57e7af0 oups! fix a missing callback that's been here for a while... 2020-02-18 16:07:14 -05:00
ansuz
f86196e40a implement shared environment between historyKeeper and RPC 2020-02-17 09:01:10 -05:00
ansuz
38c1700173 Respond to pinning RPCs as soon as possible
(instead of waiting until you've read an unbounded number
of pin logs while queries back up in memory)

Also replace instances of 'publicKey' with 'safeKey' or 'unsafeKey'
to clearly and correctly indicate their format.
2020-02-14 13:46:40 -05:00
ansuz
3dc789cbca replace pinned.js with low-profile streaming implementation 2020-02-13 17:46:10 -05:00
ansuz
1231cacef1 remove XXX 2020-02-10 11:21:04 -05:00
ansuz
7f22da0f57 fix use of incorrect key format in admin-rpc 2020-02-10 08:07:29 -05:00
ansuz
802034616c centralize historykeeper-rpc interaction in rpc methods 2020-02-07 12:53:12 -05:00
ansuz
b4172676f3 lint compliance 2020-02-05 18:12:09 -05:00
ansuz
f17d14fd99 simpler limit updates 2020-02-05 18:01:48 -05:00
ansuz
bde17a62a1 continue refactoring rpc 2020-02-05 17:31:44 -05:00
ansuz
fa525817ba fix broken removePins RPC 2020-02-05 13:05:06 -05:00
ansuz
5808c534b5 fix function signature for trimHistory RPC 2020-02-05 11:53:47 -05:00
ansuz
9c28941f6c lint compliance 2020-02-05 10:47:43 -05:00
ansuz
46dfa026f0 fix an API change that caused a typeError 2020-02-03 18:47:18 -05:00
ansuz
88be40ede3 standardize some function signatures and factor out a lot of boilerplate 2020-02-03 18:32:21 -05:00
ansuz
65ba85d97b clear historyKeeper cache when we trim a channel 2020-02-03 17:14:42 -05:00
ansuz
43307ffb1a define all server intervals in a map so we can easily clear them all 2020-02-03 17:14:23 -05:00
ansuz
6523974ca2 fix a WRITE_PRIVATE_MESSAGE rpc regression 2020-02-03 15:47:41 -05:00
ansuz
779e817443 stop relying on netflux-server internals
* create RPC module from inside historyKeeper
* stop passing around netflux-server context
  * update to use newer netflux-server's formal APIs
* manage your own cache of indexes instead of storing things in the netflux context
2020-02-03 14:20:05 -05:00
ansuz
b093d3f0d2 WIP massive rpc refactor 2020-01-24 14:45:53 -05:00
ansuz
ceb351326c split out some more rpc functionality and fix broken module paths 2020-01-24 13:36:14 -05:00
ansuz
c1f222dd6c move metadata commands from rpc to their own module 2020-01-24 13:19:40 -05:00
ansuz
bb7e8e4512 move login block functionality into its own rpc module 2020-01-24 13:14:26 -05:00
ansuz
c765362744 move more rpc functionality into modules 2020-01-24 13:06:46 -05:00
ansuz
c93b39c094 separate more rpc functionality into pinning and core submodules 2020-01-24 12:43:11 -05:00
ansuz
ba6e3f33bd move admin commands into their own module 2020-01-24 11:25:48 -05:00