Show HN: We scaled Git to support 1 TB repos I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here: https://ift.tt/m4FOuZ9 Today, XetHub works for 1 TB repositories, and we plan to scale to 100 TB in the next year. Our implementation is in Rust (client & cache + storage) and our web application is written in Go. XetHub includes a GitHub-like web interface that provides automatic CSV summaries and allows custom visualizations using Vega. Even at 1 TB, we know downloading an entire repository is painful, so we built git-xet mount - which, in seconds, provides a user-mode filesystem view over the repo. XetHub is available today (Linux & Mac today, Windows coming soon) and we would love your feedback! Read more here: - https://ift.tt/TQMmIqP - https://ift.tt/QWFJ4qS https://ift.tt/JWZ3Flp December 13, 2022 at 09:14PM
Show HN: StreetComplete, an OpenStreetMap Editor for Humans StreetComplete is an OpenStreetMap[0] editor directed at people who want to contribute and want to do this using their smartphone, without learning how to edit things[1]. It is available as an Android application. It is intended to be used as one walks, with quests appearing as markers on the map. Selecting a marker allows one to answer a simple question. The answer will be added to the OpenStreetMap database, with app handling selecting objects for editing, transforming answer into OSM tags and making edits. OpenStreetMap account is needed to apply edits, but it is possible to start without it, make some edits and login/register later. Note: I am not the main author, but I am one of the active contributors. Github page is at https://ift.tt/2g8lasH and https://ift.tt/3nR9PzS shows what was recently released. [0]OpenStreetMap is a Wikipedia of maps, available on the open licence. This dataset is already used for many interestin...
Comments