Clojurians Log v2 - Open Source Funding

👋

I along with my colleagues over at Gaiwan (the team behind Lambda Island) are largely maintaining the current clojurians-log-app (v1), keep the servers running, and help debug issues (#clojureverse-ops channel).

I started work on a Version 2 of the log to experiment with some of my ideas which didn’t make sense to add in the official app. Slowly I realised that the v2 (standing on top of the great work done in v1) could have significant benefits to the community and continued to work on it.

My mains reasons to do so were:

• I feel the amount of knowledge being shared on the clojurians slack server is immense. Conserving and making this discourse complete, easily accessible, and searchable should greatly benefit the community as a whole.
• The current clojurians-log-app (v1) has aged a bit as it’s being maintained over the years. There are lots of operational issues cropping up and there is a huge scope of improvement which I can see (being one of it’s current maintainers).
• The decision to make an independent v2 makes a lot of sense to me personally to get a chance to completely re-architect the solution from all the lessons we’ve learnt over the years. It also sounds much more exciting 🙈🙈
• There is a LOT of potential features I wanted to add, listed on the README page. But those are just my personal dreams, do check it out! 💭
• I wanted to work with Postgres Full text search extention 🤣

Thanks to the awesome team over at Clojurists Together, I was able to receive a funding of $2000 USD to work on this project to bring it to MVP and release it. Super thanks to the amazing folks of the clojure community for all their support 🥳 🌸

Action plan ⚡

I’m writing down a small action plan for the work I intend to do up until December, any feedback / discussion / suggestions / rants are welcome. Let me know if I would be wasting my time on a feature no one would use 🐍 🐌

This is the bare minimum I need to do get an MVP up and running. Like @plexus always says, “Always start with a walking skeleton”, and that’s what I intend to do first.

Walking Skeleton

The walking skeleton 💀🦴 would look something like this:

• I have purchased a Hetzner VPS (4gb ram)
• A bunch of ansible scripts to provision the server and deploy the code
  • setup user, fail2ban, firewall, postgres, java, clojure, git, caddy, SSL, etc.
  • run one-off commands?
  • run migrations
  • deploy & update the code / systemd / caddy services
• 🐿 Ship it live on a dummy staging domain🌏
• [Possibly] GitHub actions to continually deploy changes from master
  • I am not sure how safe GITHUB secrects would be in a public repo as anyone who creates a PR could possibly print/echo them 🤔
  • Low priority, might skip this
• Setup sql migrations 🛰️
  • https://github.com/yogthos/migratus
  • or https://github.com/weavejester/ragtime ?

MVP / critical features

Once the walking skeleton is done, the plan is to work on these:

• slack imports from the beginning of time ⌛
  • I don’t want to start off with a real-time archiving solution
  • first goal is to make clojurians-log-v2 idempotent to data imports (incremental / ad-hoc)
  • given a slack export, and a single repl command (import "path-to-slack-export-dir"), all data should be populated properly
  • this would give us huge flexibility in the future and should be really easy to run up a new instance independently at anytime
  • instance was down for a week? No problem, just export the weeks data from slack admin ui and import it. DONE!
  • this is the BIGGEST task (messages, users, channels, reactions, threads, replies etc)
• real time full text search for entire logs 🔍
  • I’m most excited to work on this, my first attempt is going to be using postgres’ inbuilt FTS
  • 2nd choice would be MeiliSearch vs typsense
  • creating a frontend interface for this search (it would be wise to start off with a server rendered search result, maybe in the future we can do “type as you search")
• mobile responsive and a brand new UI + UX

Once this is done the MVP would be ready and we can discuss further from there! 😁 🦄 🙏

Additional

Once the above critical features are done, 2 very important features would still be pending to really get this off the ground!

• Full SEO compatibility with existing clojurians-log so that Google and others keep indexing it as is (if and when the switch happens)
• Implementing real-time socket API of slack to process events and log messages instantly 🚥
  • making sure this stays compatible and idempotent with our static data import / backfilling

Preliminary screenshot:

Feedback

Am I missing something really important or obvious? Please feel free to discuss and post your thoughts / feedback / suggestions / rant below, I am super excited to hear them! 😁

I will try to record some videos while building this and upload to my youtube channel if possible.

Also do not shy away from creating issues, or contributing! Issues and PR’s are always welcome 🌸

If you’re as excited about this as I am, please leave a star ⭐ ! It would greatly help in keeping me motivated 🌞

Link to Github repo https://github.com/oxalorg/clojurians-log-v2