Show HN: Capillaries: Distributed data processing with Go and Cassandra I started thinking about this approach after working on a large-scale project for a major financial company where our group developed a distributed in-house data processing solution. On a regular basis, it ingested a few gigabytes of financial data and, within a tight SLA time limit, produced a lot of enriched/aggregated/validated data for a number of customers. Sometimes, source data had errors, so operators with domain knowledge had to verify data validity at some checkpoints, immediately make corrections, and re-run parts of the workflow manually. The solution involved complex web service orchestration, custom database and was very demanding on the infrastructure availability. Capillaries is a built from scratch, open-source Go solution that does just that: ingests data and applies user-defined transforms - Go one-liner expressions, Python formulas, joins, aggregations, denormalization - using Cassandra for intermediate data storage and RabbitMQ for task scheduling. End users just have to provide: - source data in CSV files; - Capillaries script (JSON file) that defines the workflow and the transforms; - Python code that performs complex calculations (only if needed). The whole data processing pipeline can be split into separate runs that can be started independently and re-run by the user if needed. The goal is to build a platform that is tolerant to database and processing node failures, and allows users to focus on data transform logic and data quality control. “Getting started” Docker-based demo calculates ARK funds performance, using EOD holdings and transactions data acquired from public sources. There are also integration tests that use non-financial data. There is a test deploy tool that uses Openstack API for provisioning in the cloud. https://capillaries.io May 16, 2023 at 06:13AM
Show HN: Tape It, iOS recording app for musicians Hello HN, Over the last 15 months, two friends and I developed the music recording app we felt we wanted based on our own needs as musicians. It's called Tape It [1] and has just recently hit the Apple App Store [2]. We put a lot of effort into a good UX to help musicians really focus on playing their instrument instead of pretending to be a recording engineer. The app records in stereo on newer iPhones (although that's a premium feature; the free version only records in standard mono audio quality). I would be really grateful for advice from this community on how to best approach marketing. We had a great TechCrunch article covering our launch [3], and we posted it on various music websites. Turns out advertising on Google or Apple Search is a dark art, though. We have some good ideas for developing a good social media presence, but they will take time. Please hit us with feedback, opinions and advice that you think a young ind...
Comments