Skip to main content

Today I've been playing around with this
https://github.com/PromtEngineer/localGPT

GitHub - PromtEngineer/localGPT: Chat with your documents on your local device using GPT models. No data leaves your device and 100% private.

Chat with your documents on your local device using GPT models. No data leaves your device and 100% private. - GitHub - PromtEngineer/localGPT: Chat with your documents on your local device using ...
GitHub
Ian Molton
Ian Molton  
any good?
Alastair Cooper
Alastair Cooper  
I fed it the manifestos of the major parties from the last General Election and it was able to answer questions about them. It managed to get that Brexit referred to withdrawal from the European Union and Labour planned to nationalise the railways.
david
david  
How hardware intensive is this to run?
Sounds great!
Alastair Cooper
Alastair Cooper  
It runs on my PC which has a 12GB GeForce RTX 3060 GPU. Not sure what the actual system requirements are. There is another one called privateGPT which seems to only need CPU.
david
david  
Ah cool. Thanks for the info. I have an Nvidia 1070 8GB in my ageing Windows PC. For a while I had it running stable diffusion following this tutorial (with tweaks) but it isn’t working now for some reason.
Alastair Cooper
Alastair Cooper  
I recommend Linux or Mac if you can. If you're using Windows use WSL2. Trying to run it natively on Windows is possible but likely to be more of a hassle.
Alastair Cooper
Alastair Cooper  
I could do a demo of some of these things at Autscape if anyone is interested, since I can remote to my desktop PC.
david
david  
I’d be interested in a demo if it’s not pitched too high. I’m not on your or Ian’s level at all. Far far from it.
Alastair Cooper
Alastair Cooper  
Also if someone has something they want to try but doesn't have the hardware, I may be able to set up a demo. Note that there are services like Google Colab, Amazon Web Services (AWS) and Microsoft Azure where you can rent things. The basic tier of Colab is free and you can use a decent GPU for several hours at a stretch.
Ian Molton
Ian Molton  

... if you're willing to be spied on.

No-one gives out hot free computers.

Alastair Cooper
Alastair Cooper  
Pragmatically I don't think people have much to worry about if Google knows they're doing some Coursera notebooks vs £500+ to buy a GPU.
Alastair Cooper
Alastair Cooper  
I do, if you ask nicely and it's temporary.
Alastair Cooper
Alastair Cooper  
The secondary reason is that people who get really into it will rapidly outgrow the free tier.
Alastair Cooper
Alastair Cooper  
I'm privileged in having been able to save up and buy the hardware needed to run a lot of this stuff myself (I actually already had it, even before the post-GPT boom kicked off). Many people don't have that luxury and I don't think it's unreasonable to use alternatives with different trade-offs.
Alastair Cooper
Alastair Cooper  
Only problem now is I used to not care much about having a premium GPU because "I don't game much". Now it might matter.
⇧