Search the Community

As an experiment, I was considering fine-tuning an LLM (models like 'ChatGPT') on WOBs. They generally fit the training format of question: answer that makes for a good, and in this case, fun training dataset. While I could probably do web scraping and eventually pull in all the text, with the website itself open source I figured there must be a copy of the database somewhere. Anyone know where I could find this?

Hello, all! One topic of conversation on the Discord as-of-late has been an eventual API for the Arcanum library. I've made this thread in hopes of helping our amazing dev team with ideas as to what form this should take. If you can think of any uses for this API, please comment them below so they can know what features are wanted. I'd start off by recommending that the API returns a JSON array to users since that's the easiest method I know of to allow integration with things like bots. For human users, it wouldn't be the best, but people can just use the search anyways. This JSON array would be filled with all relevant WoBs, each of which would be an object containing information such as the date, the event, the speaker(s) (as an array), the tags, the footnotes, and of course the main content. I've seen this strategy used well before in another API whose link I can provide at request The other question is what format the API requests should be made in. I think that each request should be able to fill any number of available fields, which would include those options on the search menu (text, earliest date, latest date, speaker, tags, antitags, and ordering style), as well as those options which can be achieved with special characters such as ampersands for "and" or exclamation marks for "not". While I'm not sure these fields would all be used too often, they'd be better to integrate from the start, and shouldn't be too difficult given they're already worked into queries. I could, of course, be wrong about it being difficult - it's notoriously challenging to predict how resistant to change a system you've never seen before is. I've started "hard" projects and been done in an hour, but sometimes what "should be easy" has taken many hundreds of times as long Finally, a very low priority request that I nonetheless think would be great is giving each user an API key. That API key could be used to query results that have been "favorited" by that keyholder for easy reference. I'm not sure how this would be implemented, but it would likely require modifications to the SQL. Another potential future option would be to let people query WoBs they've gotten with this key, where each user could query the speaker as an API key and be returned all WoBs which the keyholder marked as originating from them. This could further be possibly integrated with existing "speaker" data, where each user could say who they are and have their WoBs automatically synced to their key. This would be a lot of work for a little gain, however, so I'd consider this a very low priority feature