The_Great_Archive/ArchiveBox
1
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-09-14 11:06:13 +05:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,306 Commits 29 Branches 604 Tags 62 MiB
5ec6478b12
Commit Graph

375 Commits

Author SHA1 Message Date
Nick Sweeting
4dd53dc12a
Merge branch 'newchanges' into dev 2024-11-19 05:28:20 -08:00
Nick Sweeting
b852951c58
fix cli loading edge case where setup_django wasnt running when it should 2024-11-19 05:27:35 -08:00
Nick Sweeting
f8e2f7c753
restore missing archivebox_update work 2024-11-19 05:09:19 -08:00
Nick Sweeting
52446b86ba
restore missing archivebox_status work 2024-11-19 05:08:41 -08:00
Nick Sweeting
0f536ff18b
restore missing archivebox_schedule work 2024-11-19 05:07:55 -08:00
Nick Sweeting
fe3320eff0
restore missing archivebox_remove work 2024-11-19 05:07:12 -08:00
Nick Sweeting
230bf34e14
restore missing archivebox_config work 2024-11-19 05:05:06 -08:00
Nick Sweeting
6740202d78
fix cli loading edge case where setup_django wasnt running when it should 2024-11-19 04:20:00 -08:00
Nick Sweeting
f21b86aba8
better cli colors 2024-11-19 04:10:07 -08:00
Nick Sweeting
0f860d40f1
working archivebox_status CLI cmd 2024-11-19 04:05:05 -08:00
Nick Sweeting
292730ebad
working archivebox_schedule cmd 2024-11-19 03:54:47 -08:00
Nick Sweeting
3a64ced697
fix archivebox delete errors 2024-11-19 03:45:44 -08:00
Nick Sweeting
0347b911aa
archivebox add and remove CLI cmds 2024-11-19 03:40:01 -08:00
Nick Sweeting
2595139180
improve statemachine logging and archivebox update CLI cmd 2024-11-19 03:31:05 -08:00
Nick Sweeting
c9a05c9d94
working archivebox update CLI cmd 2024-11-19 02:32:05 -08:00
Nick Sweeting
a0edf218e8
fix archivebox init and archivebox install CLI commands 2024-11-19 01:05:49 -08:00
Nick Sweeting
5f01fc8307
fix archivebox shell and manage CLI commands 2024-11-19 00:48:39 -08:00
Nick Sweeting
328eb98a38
move main funcs into cli files and switch to using click for CLI 2024-11-19 00:18:51 -08:00
Nick Sweeting
569081a9eb
rename abid_utils to base_models 2024-11-18 19:40:05 -08:00
Nick Sweeting
65afd405b1
merge seeds and crawls apps 2024-11-18 19:23:14 -08:00
Nick Sweeting
4a5d607296
move logging_util into archivebox.misc subfolder 2024-11-18 19:08:49 -08:00
Nick Sweeting
e469c5a344
merge queues and actors apps into new workers app 2024-11-18 18:52:48 -08:00
Nick Sweeting
0acd388c02
fix imports and deps 2024-11-18 18:07:34 -08:00
Nick Sweeting
6b83b4c995
leave archivebox running when in archivebox update 2024-11-18 04:27:38 -08:00
Nick Sweeting
eeb2671e4d
API improvements 2024-11-18 04:27:38 -08:00
Nick Sweeting
1e3ce67834
fix API and CLU calls 2024-11-18 04:27:38 -08:00
Nick Sweeting
c8e186f21b
fix plugin loading order, admin, abx-pkg 2024-11-16 06:44:12 -08:00
Nick Sweeting
b4a5da3ffd
update archivebox add CLI command to use new actor system 2024-11-16 02:45:37 -08:00
Nick Sweeting
312e40b95b
finally get rid of config/legacy in favor of configfile.py and django.py 2024-10-21 03:06:19 -07:00
Nick Sweeting
b3107ab830
move final legacy config to plugins and fix archivebox config cmd and add search opt 2024-10-21 02:56:00 -07:00
Nick Sweeting
a211461ffc
fix LIB_DIR and TMP_DIR loading when primary option isnt available 2024-10-21 00:35:56 -07:00
Nick Sweeting
6e7071bd19
add new binproviders and binaries args to install and version, bump pydantic-pkgr version 2024-10-11 00:45:59 -07:00
Nick Sweeting
cf1ea8f80f
improve config loading of TMP_DIR, LIB_DIR, move to separate files 2024-10-07 23:45:11 -07:00
Nick Sweeting
5323953f94
handle Ctrl+C more gracefully 2024-10-04 21:33:46 -07:00
Nick Sweeting
12f32c4690
fix tmp data dir resolution when running help or version outside data dir 2024-10-04 01:40:41 -07:00
Nick Sweeting
f321d25f4c
fallback to reading binaries from filesystem when theres no db 2024-10-04 01:00:09 -07:00
Nick Sweeting
3b9e48ead8
show deprecation warning for archivebox setup command 2024-10-03 03:10:36 -07:00
Nick Sweeting
32167de936
add daemonize flag to archivebox server 2024-10-02 19:46:48 -07:00
Nick Sweeting
035a14b6ea
better help text output 2024-10-02 19:46:31 -07:00
Nick Sweeting
968adf64da
small easter eggs 2024-10-02 14:17:28 -07:00
Nick Sweeting
8fbfa10df3
fix missing check_migrations import 2024-09-30 23:52:48 -07:00
Nick Sweeting
d21bc86075
finish migrating almost all config to new system 2024-09-30 23:21:34 -07:00
Nick Sweeting
8c3342afe5
rename archivebox setup to archivebox install 2024-09-30 23:19:46 -07:00
Nick Sweeting
51fe4c38c2
speed up version command by checking if quiet is passed 2024-09-30 18:33:43 -07:00
Nick Sweeting
b913e6f426
rename OUTPUT_DIR to DATA_DIR 2024-09-30 17:44:18 -07:00
Nick Sweeting
363a499289
move util.py into misc folder 2024-09-30 17:25:15 -07:00
Nick Sweeting
3e5b6ddeae
move config into dedicated global app 2024-09-30 15:59:05 -07:00
Nick Sweeting
8d3f45b720
merge plugantic and abx, all praise be to praise our glorious pluggy gods 2024-09-27 01:26:12 -07:00
Nick Sweeting
64c7100cf9
speed up startup time, add rich startup progressbar, split logging and checks into misc, fix search index import backend bug 2024-09-24 19:04:38 -07:00
Nick Sweeting
60154fba5f
add django_huey, huey_monitor, and replace Threads with huey tasks 2024-09-10 00:05:45 -07:00
Nick Sweeting
c76c50e71f
add HTTP byte range request support to media file serving 2024-09-05 21:41:49 -07:00
Nick Sweeting
b5ad134264
dont wait for ipython history saver thread before shell exit 2024-05-12 19:25:55 -07:00
Nick Sweeting
e1a04729b3
wait for bg threads to finish before exiting 2024-05-12 01:42:20 -07:00
Nick Sweeting
8ab17f0561 add tag support to schedule command 2024-01-03 20:31:14 -08:00
Nick Sweeting
4b8b17e788 add update flag support to archivebox schedule 2022-05-09 20:18:43 -07:00
Nick Sweeting
8cfe6f4afb cleanup update flag handling and show better logging to clarify when its working 2022-05-09 20:15:55 -07:00
Nick Sweeting
8b4f84959a new setup flag on init command to autosetup on init 2021-04-05 21:15:32 -04:00
Nick Sweeting
211bf86f4a fix bug checking value in subcommand_args when None 2021-04-05 21:13:49 -04:00
Nick Sweeting
05c17af4dc fix lint errors 2021-04-05 20:00:16 -04:00
Nick Sweeting
ef76f7c490 require running setup in existing data dir 2021-04-05 19:59:33 -04:00
Nick Sweeting
8c1d12020f fix lint errors 2021-04-05 19:53:42 -04:00
Nick Sweeting
3449bac5bc add new setup command to autoinstall dependencies 2021-04-05 19:51:57 -04:00
Nick Sweeting
0e39a2098d split migrations and data dir check 2021-04-01 14:00:07 -04:00
Nick Sweeting
005da31f8f add overwrite flag option to schedule cmd 2021-03-31 11:29:51 -04:00
Nick Sweeting
36f0646501
Merge pull request #669 from FliegendeWurst/fix-issue-235 
add command: --parser option (fixes #235)
 2021-03-31 00:53:47 -04:00
Nick Sweeting
2656e59215
change list style 2021-03-31 00:47:42 -04:00
FliegendeWurst
60bd9a902e add command: --parser option 2021-03-28 10:09:11 +02:00
Nick Sweeting
fea0b89dbe add tag cli option 2021-03-27 03:57:05 -04:00
Nick Sweeting
bd6d9c165b enforce utf8 on literally all file operations because windows sucks 2021-03-27 01:16:29 -04:00
Nick Sweeting
6aef1dd98d dont check db if init is pending 2021-02-28 22:53:23 -05:00
Nick Sweeting
33df9c1ebe fix after and before in remove tests 2021-02-18 06:21:44 -05:00
Nick Sweeting
106f6adc59 fix lgtm alerts 2021-02-18 04:29:24 -05:00
Nick Sweeting
74c882ef16 add nothreading arg to archivebox server for perf profiling 2021-02-16 15:46:58 -05:00
Nick Sweeting
49939f3eaa only accept stdin if args are not passed, fix stdin hang in docker 2021-02-16 01:20:47 -05:00
Nick Sweeting
9ce3bd5bdc use index.LINK_FILTERS to validate filter-type args instead of hardocding them twice 2021-02-15 20:43:36 -05:00
Nick Sweeting
3c3bae02d2 add quick-init option to skip reimporting all snapshot dirs on init 2021-02-15 14:52:10 -05:00
Nick Sweeting
0407d03b6b add cli tests file back 2021-02-15 13:39:49 -05:00
Nick Sweeting
c4b02be24d remove dead tests code 2021-02-01 05:14:43 -05:00
Nick Sweeting
39ec77e46c add createsuperuser flag to server command 2021-01-28 22:27:02 -05:00
Nick Sweeting
02bdb3bdeb fix DATABASE_NAME posixpath 2021-01-20 18:42:12 -05:00
Cristian
a57a5b6b83 refactor: call setup_django with the check_db attribute for the commands that actually need the database 2020-12-11 18:02:56 -05:00
Nick Sweeting
9fa70b3452 add extractors arg to oneshot command and bump version to v0.5.1 2020-12-11 15:48:46 +02:00
Cristian
275ad22db7 refactor: Remove skip_index from archive related functions 2020-12-08 18:42:25 -05:00
Cristian
9aac09a5e1 feat: Patch setup_django so we can use an inmemory db in specific commands 2020-12-08 18:42:25 -05:00
Cristian
35a5700c73 fix: Move the setup_django command to a place where we already know what the actual subcommand is 2020-12-08 18:42:25 -05:00
JDC
273c9d91c6 Add tag filter to update command 2020-12-06 01:13:39 +02:00
JDC
8484bdb973 Fix add search filter to update 2020-12-06 01:13:39 +02:00
JDC
70cc0c1950 Add search filter-type 2020-12-06 01:13:39 +02:00
JDC
0ed53cc117 Add search filter type for update 2020-12-06 01:13:39 +02:00
JDC
0f7dba07df feat: add search filter-type to list command 2020-12-06 01:13:37 +02:00
Cristian
35389608d1 feat: Add --extract flag to update command 2020-12-05 12:20:47 -05:00
Nick Sweeting
1b22f8eeef
Merge pull request #515 from cdvv7788/POC-setup-django-on-init 2020-11-27 23:56:37 -05:00
Nick Sweeting
fdd4effc92
Merge pull request #535 from cdvv7788/extractors-flag 2020-11-13 14:53:17 -05:00
Nick Sweeting
257d3f2a98
Update archivebox/cli/archivebox_add.py 2020-11-13 14:52:21 -05:00
JDC
d54c3eec9d Add tag filter argument to remove command 2020-11-13 14:16:48 -05:00
Cristian
54df0a035b fix: Move csv split to the add function to avoid optional nullable argument 2020-11-13 13:10:17 -05:00
Cristian
1ec8276514 fix: Use a comma separated input instead of nargs for the extract flag 2020-11-13 13:01:11 -05:00
JDC
cbb3d04c12 Allow list filtering by tag name 2020-11-13 12:06:12 -05:00
Cristian
44eede96e5 feat: Add extract flag to add command 2020-11-13 09:24:34 -05:00
Nick Sweeting
cac3912439 small type fixes 2020-10-30 04:50:14 -04:00
Cristian
a6bee5f111 feat: Move setup_django to an inner module 2020-10-26 08:02:04 -05:00
Angel Rey
3e26ab3ce3 Replaced os.path in clic tests 2020-10-02 15:46:39 -05:00
Angel Rey
fa364ed728 Replaced od.path in init cli 2020-10-02 15:46:39 -05:00
apkallum
1aa7bac85b fix oneshot command type signature 2020-09-17 09:09:52 -05:00
Cristian
fa622d3e14 refactor: Replace --index with --with-headers in the list command to make it more explicit. Change it so it affects the csv output too. 2020-09-15 08:05:46 -05:00
Cristian
404f333e17 feat: Refactor get_invalid_folders to work with a queryset instead of a list of links 2020-09-15 08:05:46 -05:00
Cristian
a77d6dc235 feat: list command fails when --index is used without --json or --html 2020-09-15 08:05:46 -05:00
Cristian
885ff50449 feat: Add html export to list command 2020-09-15 08:05:46 -05:00
Cristian
aab8f96520 feat: Add flag to list command to support index like output 2020-09-15 08:05:46 -05:00
Nick Sweeting
73408fb035 fix version parsing and attempt to npm install during pip post_install 2020-08-18 15:58:05 -04:00
Nick Sweeting
718d39e242 add common code extensions to default blacklist 2020-08-18 08:12:10 -04:00
Nick Sweeting
b681a477ae add overwrite flag to add command to force re-archiving 2020-08-18 04:37:54 -04:00
Nick Sweeting
da671532a4 fix schedule cmd defaulting to day and remove week 2020-08-18 04:35:59 -04:00
Nick Sweeting
494be09bc2 add depth flag to schedule cmd 2020-08-18 01:58:54 -04:00
Nick Sweeting
a0901ba474 use BIND_ADDR config default for runserver 2020-08-13 23:21:37 -04:00
Cristian Vargas
b2a318c5eb
fix: Update error message for oneshot command 
Co-authored-by: Nick Sweeting <git@sweeting.me>
 2020-07-31 10:51:54 -05:00
Cristian
8bcb171e74 fix: Remove support for multiple urls in oneshot command 2020-07-31 09:05:40 -05:00
Cristian
c073ea141d feat: Initial oneshot command proposal 2020-07-29 11:19:06 -05:00
Nick Sweeting
9e7330cc14 add init flag to server and fix SHOW_PROGRESS config being ignored 2020-07-28 05:57:34 -04:00
Cristian
6006b4f93b refactor: Organize code to remove flake8 issues 2020-07-24 12:25:25 -05:00
Cristian
a5550b2105 fix: Rename logging folder to avoid naming conflicts (and circular import issues) 2020-07-22 11:02:13 -05:00
Cristian
f4d1b5121e refactor: Move logging.py to main module to avoid circular import issues 2020-07-17 18:00:04 -05:00
Nick Sweeting
ae208435c9 fix the add links form 2020-07-13 12:21:37 -04:00
Nick Sweeting
d159e674e1 write stderr instead of stdout for version info 2020-07-13 11:41:18 -04:00
Nick Sweeting
d3bfa98a91 fix depth flag and tweak logging 2020-07-13 11:26:34 -04:00
Cristian
4ebf929606 refactor: Change wording on CLI help 2020-07-08 08:30:07 -05:00
Cristian
f12bfeb322 refactor: Change add() to receive url and depth instead of import_str and import_path 2020-07-08 08:17:47 -05:00
Cristian
c1d8a74e4f feat: Make input sent via stdin behave the same as using args 2020-07-07 15:49:40 -05:00
Cristian
b68c13918f feat: Disable stdin from archivebox add 2020-07-07 12:39:36 -05:00
Cristian
a6940092bb feat: Make sure that depth can only be either 1 or 0 2020-07-07 10:25:02 -05:00
Cristian
32e790979e feat: Enable depth=1 functionality 2020-07-07 10:07:44 -05:00
Cristian
2db0324539 feat: depth=0 crawls the current page only 2020-07-07 09:49:28 -05:00
Cristian
8b22a2a7dd feat: Enable --depth flag (still does nothing) 2020-07-07 09:10:36 -05:00
Nick Sweeting
7c428f40c8 fix stdin link importing 2020-07-02 13:31:05 -04:00
Nick Sweeting
0c48449aa6 fix subcommand and args not being passed 2020-07-02 04:00:51 -04:00
Nick Sweeting
322be6b292 move main into cli init and remove circular import layer 2020-07-02 03:53:39 -04:00
Nick Sweeting
df593dea0a fix missing imports 2020-06-30 05:55:34 -04:00
Nick Sweeting
44bd6866ac colorize hints in CLI output more consistently 2020-06-30 01:48:41 -04:00
Nick Sweeting
1ecb94fe09 check PYTHON_ENCODING correclty when lowercase 2020-06-30 01:07:38 -04:00
Nick Sweeting
b9e17fa0d1 rename archivebox-info to archivebox-status 2020-06-25 23:32:01 -04:00
Nick Sweeting
f60b5ed867 better stdin handling 2019-05-01 02:28:12 -04:00
Nick Sweeting
ad3898add6 allow forcing init in dirty directory 2019-05-01 02:27:50 -04:00
Nick Sweeting
cb2dd1ee28 rename model Page to Snapshot 2019-04-30 23:44:51 -04:00
Nick Sweeting
95007d9137 split up utils into separate files 2019-04-30 23:13:04 -04:00
Nick Sweeting
daf5951897 allow passing debug flag to archivebox server 2019-04-30 23:11:41 -04:00
Nick Sweeting
0ba821b351 better return status from list cmd 2019-04-30 23:11:07 -04:00
Nick Sweeting
158f145d9a move docstrings to main.py out of cli files 2019-04-30 23:10:48 -04:00
Nick Sweeting
1b8abc0961 move everything out of legacy folder 2019-04-27 17:26:24 -04:00
Nick Sweeting
1edb14b229 wip config reset command 2019-04-26 14:43:23 -04:00
Nick Sweeting
2b4244bb52 allow setting config via aliases and show side effect changes in stderr 2019-04-26 14:43:13 -04:00
Nick Sweeting
4d6ad7a65d add archivebox config command and move config into sections 2019-04-25 19:03:38 -04:00
Nick Sweeting
df4307c4ff move config loading into function for easier runtime reloading 2019-04-25 19:00:25 -04:00
Nick Sweeting
d248684839 add index-only option to archivebox add and update 2019-04-25 18:59:41 -04:00
Nick Sweeting
56d0b2c088 new methods for detecting valid/invalid data dirs on init 2019-04-24 11:40:10 -04:00
Nick Sweeting
168d253c17 show human readable filesize in version info 2019-04-24 11:39:02 -04:00
Nick Sweeting
b0ace38273 warn user that no admins exist when starting runserver 2019-04-24 11:38:37 -04:00
Nick Sweeting
5ef5415f3f rename archive toggles from FETCH_ to SAVE_ for clarity 2019-04-24 04:09:25 -04:00
Nick Sweeting
71402c3d09 add manage command and shell welcome message 2019-04-22 19:08:01 -04:00
Nick Sweeting
f0f516e853 check for data folder when running most subcommands 2019-04-22 19:06:48 -04:00
Nick Sweeting
2f0dbeebc1 update docstrings and comments 2019-04-22 14:42:04 -04:00
Nick Sweeting
ab68819332 add archivebox info command to scan data dir 2019-04-22 14:34:30 -04:00
Nick Sweeting
168e578ea4 fix bad default in scheduler 2019-04-22 13:21:08 -04:00
Nick Sweeting
f489dd96a9 fix archivebox remove rejecting stdin patterns 2019-04-22 13:19:47 -04:00
Nick Sweeting
39a0ab3013 add pipenv, schedule cmd, logs dir, and lots more 2019-04-18 21:10:00 -04:00
Nick Sweeting
920898e160 working archivebox-server command 2019-04-17 05:42:35 -04:00
Nick Sweeting
88a37bc552 fix json list output 2019-04-17 05:41:41 -04:00
Nick Sweeting
cdb70c73df first working django model with archivebox-shell command and sql exporting 2019-04-17 03:49:18 -04:00
Nick Sweeting
8b4b13b667 accept multiple pattern args for removal 2019-04-17 02:22:29 -04:00
Nick Sweeting
21174da014 better data folder checking on startup 2019-04-17 02:22:19 -04:00
Nick Sweeting
d8d8f7c2cc working consistent list and remove with filtering 2019-04-11 07:00:26 -04:00
Nick Sweeting
4ca9a0beac colorized and better command sorting in help msg 2019-04-11 06:59:30 -04:00
Nick Sweeting
718e25c973 better dependency checking system and guards 2019-04-11 03:42:35 -04:00
Nick Sweeting
eb2b6978c3 comment out uninmplemented args 2019-04-03 03:52:56 -04:00
Nick Sweeting
fd80275856 print import instructions on first run 2019-04-03 01:54:15 -04:00
Nick Sweeting
51ae634ec9 working argparse based CLI with most commands implemented 2019-04-03 00:27:37 -04:00