Show HN: We scaled Git to support 1 TB repos I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here: https://ift.tt/m4FOuZ9 Today, XetHub works for 1 TB repositories, and we plan to scale to 100 TB in the next year. Our implementation is in Rust (client & cache + storage) and our web application is written in Go. XetHub includes a GitHub-like web interface that provides automatic CSV summaries and allows custom visualizations using Vega. Even at 1 TB, we know downloading an entire repository is painful, so we built git-xet mount - which, in seconds, provides a user-mode filesystem view over the repo. XetHub is available today (Linux & Mac today, Windows coming soon) and we would love your feedback! Read more here: - https://ift.tt/TQMmIqP - https://ift.tt/QWFJ4qS https://ift.tt/JWZ3Flp December 13, 2022 at 09:14PM
Show HN: KittenDNS – easy to setup, rule engine, LetsEncrypt compatible My goal was to create a DNS server as simple as the LDAP server I've been contributing to (glauth) and here it is: https://ift.tt/i6S4IHQ A few things to know about it: - It is easy to configure using a Toml file - I am using it to bail my coredns/etcd instances when under attack - (yes, it also works as a service location server) - It comes with a simple, plain English, rule engine that doesn't do much at this point - It can be used with LetsEncrypt. I use it to retrieve certificates for my home servers. - RFC: all the nice ones :) Anyway, feedback is welcome. May 16, 2022 at 03:34AM
Comments