owncast/core/data/persistence.go
Gabe Kangas 045a0a2afd
Social features / ActivityPub federation (#1629)
* Support webfinger requests for the live account. Closes https://github.com/owncast/owncast/issues/1193

* Support for actor requests. Returns response for live actor. Closes https://github.com/owncast/owncast/issues/1203

* Handle follow and unfollow requests. Closes
https://github.com/owncast/owncast/issues/1191 and https://github.com/owncast/owncast/issues/1205 and https://github.com/owncast/owncast/issues/1206 and https://github.com/owncast/owncast/issues/1194

* Add basic support for sending out text activities. For https://github.com/owncast/owncast/issues/1192

* Some error handling and passing of dynamic local account names.

* Add hardcoded example image attachment to test post

* Centralize the map of accounts and inboxes

* No longer disable the preview generator based on YP toggle

* Send a federated message to followers when stream starts. For https://github.com/owncast/owncast/issues/1192

* Placeholder for attaching tags

* Add image description

* Save and get to outbox persistence. Return using outbox endpoint for actor

* Pass payloads to be handled through the gochan

* Handle undo follow requests explitly, not all undo requests

* Add API for manually sending simple federated messages. Closes #1215

* Verify inbox requests. Closes #1321

* Add route to fetch a single AP object by ID. For #1329

* Add responses to fediverse nodeinfo requests

* Set and get federation config values for admin

* Handle host-meta requests

* Do not send out message if disabled. Use saved go live message.

* Require AP-compatible content types for AP-related requests

* Rename ap models to apmodels for clarity

* Change how content type matching takes place.

* io -> ioutil

* Add stub delete activity callback

* Handle likes and announces to surface engagement in chat. Part of #1229

* Append url to go live posts

* Do not require specific content types for nodeinfo requests

* Add follow engagement chat message via AP

* add owncast user-agent to requests

* Set note visibility to public (for now)

* Fix saving/fetching a single object

* Add support for x-nodeinfo2 responses

* Point to the dev admin branch for ap

* Bundle in dev admin for testing

* Add error logging

* Add AP middleware back

* Point to the new external compatible logo endpoint

* Clean up more AP logging to help testing

* Tweak go live text and link hashtags

* Fix bug in fetching init time

* Send update actor activities when server details/profile is updated

* Add federation config overview to web client config

* Add additional actor properties

* Make the AP middleware checking more flexible when looking at types

* First pass at remote fediverse follow flow. For #1371

* Added a basic AP actor followers endpoint

* WIP client followers API

* Add profile-page reference to webfinger response

* Add aliases to webfinger response

* Fix content-type returned to be expected activitypub+json

* First pass at followers api

* Point at local dev copy of go-fed/activity

* Add custom toot Hashtag objects to posts

* Store additional user details to followers table

* Fix AP followers endpoint. Closes #1204

* Add owncast hashtag as an invisible tag to go live posts

* Reject AP requests when it is disabled

* Add actor util for generating full account user from person object

* Verify inbox requests before performing any other work

* Accept actor update requests

* Fix linter errors in federation branch

* Migrate AP SQL to sqlc for type safe queries

* Use the @unclearParadigm REST parameter helper

* Fix verifying post ID on AP engagement

* WIP privacy/request approval

* Style the remote follow modal

* First pass at a followers list component w/ mock data. #1370

* Revert "Use the @unclearParadigm REST parameter helper"

This reverts commit c8af8a413f6f53e7d1a15a7d823ff28be2db3c23.

* Fix get followers API

* Add support for requiring approval. Closes https://github.com/owncast/owncast/issues/1208

* Handle Applications as Actors partly for PeerTube support

* add temp todo list

* check route on load, this might change later

* style followers

* account for just 1 tab case

* Remove mock data. Allow showing follow button even when there are no external actions defined

* Point to actual followers API

* Support fallback img for follower views

* Remove duplicate verification. Add some additional verbose logging

* Bundle dev admin

* Add type to host-meta webfinger template response

* Tweak remote follow modal content

* WIP federation followers refactor

* Do not send pointer to middleware

* Update admin

* Add setting for toggling displaying fediverse engagement. Closes #1404

* Add in-development admin

* Do not enable cors on admin followers api

* Add db migration for updating messages table

* Enable empty string go live messages to disable

* Remove debug messages

* Rework some ActivityPub handling.

Create new Actor->Person handling.
Create new Actor->Service handling.
Add engagement handlers to send chat events and store event objects.
Store inbound activities to new ap_inbound_activities table.

* Support federated engagement events.

Store them in the messages table and surface them via chat events.

* Support federated event engatement in the chat

* Tweak web UI followers handling

* Point go.mod at remote fork instead of local

* Update admin

* Merged in develop. Couple fixes

* Update dev admin

* Update fedi engagement posts.

- Fix incorrect action text.
- Add action icons.

* Set public as to instead of cc for ap msg

* Updated styling for federated actions in chat

* Add support for blocking federated domains. Closes #1209

* Force checking of https in verify step

* Update dev admin

* Return user scopes in chat history api. Closes #1586

* Update dev admin

* Add AP outbound request worker pool. Closes #1571

* Disable (temporarily?) owncast tag on AP posts

* Consolidate creating activity+notes in outbound AP messages

* Add inbox worker pool. Closes #1570

* Update dev admin bundle

* Clean up some logs

* Re-enable inbound verfication

* Save full IRI to outbox instead of path

* Reject if full IRI is not found in outbox

* Use full ActivityPub user account in chat event

* Fix and expand follower APIs

- Add missing IDs to AP follower endpoints
- Split AP follower endpoints into initial request and pages.
- Support pagination in AP requests.

* Include IRI in error message

* Hide chat toggle when chat is hidden. Closes #1606

* Updates to followers pagination

* Set default go live message

* Remove log

* indirect -> direct import

* Updates for inbound federated event handling.

- Keep track of existing events and reject duplicates.
- Change what is sent to chat for surfing federated engagement.
- Keep track if outbound events are automated "go live" events or not.

* Update chat federated engagement.

* Update dev admin.

* Move from being a person to a bot (service). Closes #1619

* Only set server init date if not already set

* Only save notes to outbox able

* Rework private-mode followers/approvals

* API for returning a list of federated actions for #1573

* Fix too-small follower cells and jumpy tabs. Closes #1616 and closes #1516

* Fix shortcuts getting fired on inputs. Fixes #1489 and #1201

* Add spinner, autoclose + other fixes to follow modal. Fixes #1593

* Fix fetching a single object by IRI

* SendFederationMessage -> SendFederatedMessage

* Autolink and create tag objects from manual posts. Closes #1620

* Update dev admin bundle

* Handle engagement from non-automated/live posts

* Reject federated engagement actions if they do not match a local post

* Update dev admin bundle

* A bunch of cleanup

* Fix unused assignments and logic

* Remove unused function

* Add content warning and sentive content flag if stream is NSFW. Closes #1624

* Disable fetching objects by IRI when in private mode. Closes #1623

* Update the error message of the remote follow dialog. closes #1622

* Update dev admin

* Fix NREs throwing in test content

* Fix query that wasn't properly filtering out hidden messages

* Test against user being disabled instead of message visibility

* Fix automated test NRE

* Update comment

* Adjust federated engagement chat views. Closes #1617

* Add additional index to users table

* Add support for removing followers/requests. Closes #1630

* Reject federated actions from blocked actors. #1631

* Use fallback avatar if it fails to load. Closes #1635

* Fix styling of follower list. Closes #1636

* Add basic blurb stating they should follow the server. Closes #1641

* Update dev admin

* Set default go live message in migration. Closes #1642

* Reset the messages table on 0.0.11 schema migration

* Fix js error with moderation actions. Closes #1621

* Add a bit more clarification on follow modal. Closes #1599

* Remove todos

* Split out actor and domain blocking checks

* Check for errors on default values being set

* Clean up actor rejection due to being blocked

* Update dev admin

* Add colon to error to make it easier to read

* Remove markdown rendering of go live message. Reorganize text. Remove content warning. Closes #1645

* Break out the sort+render messages logic so it can be fired on visibility change. Closes #1643

* Do not send profile updates if federation is disabled

* Save follow references to inbound activities table

* Update dev admin

* Add blocked actor test

* Remove the overloaded term of Follow from social links

* Fix test running in memory only

* Remove "just" in engagement messags

* Replace star with heart for like action.

* Update dev admin

* Explicitly set cc as public

* Remove overly using the stream name in fediverse engagement messages

* Some federated/follow UI tweaks

* Remove explicit cc and bcc as they are not required

* Explicitly set the audience

* Remove extra margin

* Add Join Fediverse button to follow modal. Closes #1651

* Do not allow multiple follows to send multiple events. Closes #1650

* Give events a min height

* Do not allow old posts to be liked/shared. Closes #1652

* Remove value from log message

* Alert followers on private mode toggle

* Ignore clicks to follow button if disabled

* Remove underline from action buttons

* Add moderator icon to join message

* Update admin

* Post-merge remove unused var

* Remove pointing at feature branch

Co-authored-by: Ginger Wong <omqmail@gmail.com>
2022-01-12 13:53:10 -08:00

173 lines
3.7 KiB
Go

package data
import (
"bytes"
"database/sql"
"encoding/gob"
"sync"
"time"
// sqlite requires a blank import.
_ "github.com/mattn/go-sqlite3"
"github.com/owncast/owncast/config"
"github.com/owncast/owncast/db"
log "github.com/sirupsen/logrus"
)
// Datastore is the global key/value store for configuration values.
type Datastore struct {
DB *sql.DB
cache map[string][]byte
DbLock *sync.Mutex
}
func (ds *Datastore) warmCache() {
log.Traceln("Warming config value cache")
res, err := ds.DB.Query("SELECT key, value FROM datastore")
if err != nil || res.Err() != nil {
log.Errorln("error warming config cache", err, res.Err())
}
defer res.Close()
for res.Next() {
var rowKey string
var rowValue []byte
if err := res.Scan(&rowKey, &rowValue); err != nil {
log.Errorln("error pre-caching config row", err)
}
ds.cache[rowKey] = rowValue
}
}
// GetQueries will return the shared instance of the SQL query generator.
func (ds *Datastore) GetQueries() *db.Queries {
return db.New(ds.DB)
}
// Get will query the database for the key and return the entry.
func (ds *Datastore) Get(key string) (ConfigEntry, error) {
cachedValue, err := ds.GetCachedValue(key)
if err == nil {
return ConfigEntry{
Key: key,
Value: cachedValue,
}, nil
}
var resultKey string
var resultValue []byte
row := ds.DB.QueryRow("SELECT key, value FROM datastore WHERE key = ? LIMIT 1", key)
if err := row.Scan(&resultKey, &resultValue); err != nil {
return ConfigEntry{}, err
}
result := ConfigEntry{
Key: resultKey,
Value: resultValue,
}
return result, nil
}
// Save will save the ConfigEntry to the database.
func (ds *Datastore) Save(e ConfigEntry) error {
ds.DbLock.Lock()
defer ds.DbLock.Unlock()
var dataGob bytes.Buffer
enc := gob.NewEncoder(&dataGob)
if err := enc.Encode(e.Value); err != nil {
return err
}
tx, err := ds.DB.Begin()
if err != nil {
return err
}
var stmt *sql.Stmt
stmt, err = tx.Prepare("INSERT INTO datastore (key, value) VALUES(?, ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value")
if err != nil {
return err
}
_, err = stmt.Exec(e.Key, dataGob.Bytes())
if err != nil {
return err
}
defer stmt.Close()
if err = tx.Commit(); err != nil {
log.Fatalln(err)
}
ds.SetCachedValue(e.Key, dataGob.Bytes())
return nil
}
// Setup will create the datastore table and perform initial initialization.
func (ds *Datastore) Setup() {
ds.cache = make(map[string][]byte)
ds.DB = GetDatabase()
ds.DbLock = &sync.Mutex{}
createTableSQL := `CREATE TABLE IF NOT EXISTS datastore (
"key" string NOT NULL PRIMARY KEY,
"value" BLOB,
"timestamp" DATE DEFAULT CURRENT_TIMESTAMP NOT NULL
);`
stmt, err := ds.DB.Prepare(createTableSQL)
if err != nil {
log.Fatal(err)
}
defer stmt.Close()
_, err = stmt.Exec()
if err != nil {
log.Fatalln(err)
}
if !HasPopulatedDefaults() {
PopulateDefaults()
}
if !hasPopulatedFederationDefaults() {
if err := SetFederationGoLiveMessage(config.GetDefaults().FederationGoLiveMessage); err != nil {
log.Errorln(err)
}
if err := _datastore.SetBool("HAS_POPULATED_FEDERATION_DEFAULTS", true); err != nil {
log.Errorln(err)
}
}
// Set the server initialization date if needed.
if hasSetInitDate, _ := GetServerInitTime(); hasSetInitDate == nil || !hasSetInitDate.Valid {
_ = SetServerInitTime(time.Now())
}
}
// Reset will delete all config entries in the datastore and start over.
func (ds *Datastore) Reset() {
sql := "DELETE FROM datastore"
stmt, err := ds.DB.Prepare(sql)
if err != nil {
log.Fatalln(err)
}
defer stmt.Close()
if _, err = stmt.Exec(); err != nil {
log.Fatalln(err)
}
PopulateDefaults()
}
// GetDatastore returns the shared instance of the owncast datastore.
func GetDatastore() *Datastore {
return _datastore
}