Commit graph

93 commits

Author SHA1 Message Date
Timothy J. Baek
9cdb5bf9fe feat: frontend integration 2024-04-14 18:31:40 -04:00
Timothy J. Baek
2952e61167 feat: external embeddings support 2024-04-14 17:55:00 -04:00
Timothy Jaeryang Baek
b9cadff16b
Merge pull request #1419 from lainedfles/embedding-model-fix-and-manual-update
feat: improve embedding model update & resolve network dependency
2024-04-10 01:10:07 -07:00
Timothy J. Baek
582d11f191 refac: RAG_EMBEDDING_MODEL_PATH removed 2024-04-10 00:59:05 -07:00
Timothy J. Baek
cb2158a794 fix 2024-04-10 00:51:16 -07:00
Timothy J. Baek
abfcceecef refac 2024-04-10 00:46:09 -07:00
Timothy J. Baek
f4b87ecb23 refac 2024-04-10 00:33:45 -07:00
Steven Kreitzer
0bae789d39
fix: support batching chromadb 2024-04-09 10:13:29 -05:00
lainedfles
506a061387
Merge branch 'dev' into embedding-model-fix-and-manual-update 2024-04-08 14:57:54 -06:00
Jannik S
3b3d0cce1e
Merge branch 'dev' into dockerfile-optimisation 2024-04-08 09:15:00 +02:00
Timothy J. Baek
e61e1b079f fix: file upload issue 2024-04-04 17:38:59 -07:00
Self Denial
9f82f5abba Formatting... 2024-04-04 12:09:48 -06:00
Self Denial
075fbedb02 More format fixes 2024-04-04 12:07:42 -06:00
Self Denial
bcf79c8366 Format fixes 2024-04-04 12:02:48 -06:00
Self Denial
3b66aa55c0 Improve embedding model update & resolve network dependency
* Add config variable RAG_EMBEDDING_MODEL_AUTO_UPDATE to control update behavior
* Add RAG utils embedding_model_get_path() function to output the filesystem path in addition to update of the model using huggingface_hub
* Update and utilize existing RAG functions in main: get_embedding_model() & update_embedding_model()
* Add GUI setting to execute manual update process
2024-04-04 11:01:23 -06:00
Mmx233
947c392f72
fix: manually check the docs' filename 2024-04-03 23:37:13 +08:00
Jannik Streidl
9bcb37ea10 fixes and updates 2024-04-02 14:47:52 +02:00
Jannik S
099b1d066b
Revert "Merge Updates & Dockerfile improvements" (#3)
This reverts commit 9763d885be.
2024-04-02 11:28:04 +02:00
lainedfles
9763d885be
Merge Updates & Dockerfile improvements 2024-04-02 11:25:20 +02:00
Timothy J. Baek
5558514ff1 fix 2024-04-01 15:23:12 -07:00
KoreLogic Disclosures
6c96361402
Suggested mitigation for KL-CAN-2024-002. 2024-04-01 15:55:14 -05:00
Timothy J. Baek
a6c154d839 feat: rag context logging 2024-03-31 14:02:31 -07:00
Self Denial
144c9059a3 Improve logging. Move print() statements to appropiate log().
Add COMFYUI and WEBHOOK logging and associated environment variable
control. Add WEBHOOK payload & request debug logs.
2024-03-31 13:17:29 -06:00
Timothy J. Baek
3688955c77 fix: encoding issue 2024-03-25 23:50:52 -07:00
Timothy J. Baek
6307adfba1 feat: better error handling 2024-03-25 23:47:08 -07:00
Doug Danat
c91a5d8b1f switch to using BeautifulSoup HTML loader so title is also captured 2024-03-25 11:26:18 +01:00
Doug Danat
784a6ec85e include html langchain loader for RAG 2024-03-25 09:50:53 +01:00
Timothy Jaeryang Baek
371dfc1143
Merge branch 'dev' into debug_print 2024-03-24 18:04:03 -05:00
Timothy J. Baek
ff8a55a861 refac: rag api 2024-03-24 00:41:41 -07:00
Timothy J. Baek
7e0ea8f77d feat: RAG text ingestion(store) api 2024-03-24 00:40:27 -07:00
Jannik Streidl
fdef2abdfb cuda fix 2024-03-22 12:48:48 +01:00
Self Denial
e6dd0bfbe0 Migrate to python logging module with env var control. 2024-03-20 17:11:36 -06:00
Jannik Streidl
1f6739337b docker improvements & changed universal device type env for different models used 2024-03-20 08:44:09 +01:00
Timothy J. Baek
91efd6cb63 fix: file upload encoding issue 2024-03-15 23:52:37 -07:00
Timothy J. Baek
072b499a50 fix: backslash rag content issue 2024-03-15 13:34:52 -07:00
Timothy J. Baek
8df6b137cb fix: rag 2024-03-10 18:40:50 -07:00
Timothy J. Baek
98948814fd feat: toggle pdf ocr 2024-03-10 13:32:34 -07:00
Timothy J. Baek
c49491e516 refac: rag to backend 2024-03-08 22:34:47 -08:00
Timothy J. Baek
7e5e2c42c9 refac: rag routes 2024-03-08 19:26:39 -08:00
Timothy J. Baek
b88c64f80e fix: ocr issue 2024-03-06 17:54:42 -08:00
Timothy J. Baek
bb98c10abb revert: ocr feature 2024-03-06 17:04:40 -08:00
Timothy Jaeryang Baek
8fb5f54751
Merge pull request #1050 from jannikstdl/rag-pdf-ocr
feat: added ocr functionality to the pdf loader
2024-03-06 00:45:33 -05:00
Jannik Streidl
089a63e0c6 feat: added ocr functionality to the pdf loader 2024-03-05 22:25:25 +01:00
Firat Birlik
6782e95c75 recreate rag collection is now optional and only used for web requests 2024-03-04 10:00:06 -06:00
Firat Birlik
5d4ff85228 recreate rag collection instead of falling back to stale version 2024-03-03 21:25:00 -06:00
Timothy J. Baek
47a05a47b4 feat: add rag top k value setting 2024-03-02 18:56:57 -08:00
Ased Mammad
b473ad574f fix: RAG scan unsupported mimetype
This fixes an issue with RAG that stops loading documents as soon
as it reaches a file with unsupported mimetype.
2024-02-23 14:27:31 +03:30
Timothy J. Baek
7c127c35fc feat: dynamic embedding model load 2024-02-19 11:05:45 -08:00
Jannik Streidl
acf999013b storing vectordb in project cache folder + device types 2024-02-19 07:51:17 +01:00
Timothy J. Baek
0cb0358485 refac: more descriptive var names 2024-02-18 11:16:10 -08:00