So I've Been Trying Other Languages...

68,971
0
2024-06-25に共有
I got nerd sniped by the 1 billion row challenge and ended up spending far too much time playing with file readers in various tools and languages. The results were interesting enough to do a video about

I knew Go would be fast but this was nuts. I was disappointed by the IO perf in Gleam but the response from the team was so good. Overall pretty pumped I took the time to do this

SOURCES
github.com/richard-viney/file_streams/issues/4
x.com/t3dotgg/status/1796689128207368687
gist.github.com/giacomocavalieri/192dd30f4fe5c85b7…
x.com/jarredsumner/status/1797139905937862658

Check out my Twitch, Twitter, Discord more at t3.gg/

S/O Ph4se0n3 for the awesome edit 🙏

コメント (21)
  • @richnz
    Author of the Gleam file_streams package here. Yep a previous version of the library was painfully slow at reading large text files. This was fixed and an update released within 48hrs of the GH issue being opened, several weeks prior to this video being published on YT! Would have been great if that could have been part of the video, but regardless thanks for helping make the Gleam ecosystem better, and everyone’s invited to join the Discord to chat and learn some BEAM & FP 😎
  • @jackhedaya571
    Did I… just hear a TS dev say the DX of Go is bad? Dude there is no setup — that’s the whole point. Consistent, built in formatting. The package manager is incredible, seamless, and global so that you don’t have a massive node_modules. Built-in file embedding into the binary. It doesn’t even matter that much if go build takes a second or two because you spend way less time iterating vs TS. Are you forgetting bundlers, tsconfig, eslint, prettier, type definitions you literally have to describe for your editor…? Wtf is that if not an awful DX? I’ve use TS a lot and have been way happier since moving to Go. (No hard feelings here, just doesn’t make sense to me)
  • @khai96x
    >thumbnail has the Crab >video doesn't have Rust Truly the clickbait of all time
  • @tuna5618
    I really want to learn gleam but I am busy learning rust so it will have to wait 10-15 years when I become intermediate at rust.
  • @scestefan
    "Go's focus is not on the DX around the language" is a wild statement to me
  • @M-xl1vx
    The more i listen to this dude, the more i get affirmed he is a FRONTEND developer, that happens to have a couple backend projects.
  • @Hyperion_21
    Random Gleam library dev here. I remember talking to the author of the file_streams package on Discord about there being no "read to end" function in the library, and that discussion sparked him to implement said function. I also suggested adding a way to read and write to a file without multiple opens, and he completely reworked the entire library to accommodate that! "this is not the most readable thing for developers that were born within the last 40 years" Yeah as someone who's working on a highly Erlang-focused library (Bravo, which ports Erlang ETS to Gleam) I totally agree. I hate the fact that in case statements, you have to suffix the last line of each arm with a semicolon except for the final arm, which must not be suffixed and errors if you put a ; there which causes some headaches. Also had an error once where I was missing a period on a line I didn't think needed a period, and the error message was just "syntax error before" [sic] which is infuriating. Also, every single character in an Erlang string is 16 bytes in size :) (thank god Gleam uses binary arrays for strings instead of linked lists) Great video!
  • To everyone watching the video - the challenge was created for Java, and their leaderboard has sub 3 seconds - that’s with all the parsing and aggregation also
  • @OnFireByte
    Its so weird that theo is bashing on go’s “outside of code” because of no watch flag when even node only adding it this year (as stable feature) despite being interpreted runtime. I know he doesn’t like go but this is just pure bias at this point lmao.
  • @lpil
    Woooo it's the Gleam team!!!!
  • I'm really confused by what the point of these comparisons are because almost all of the solutions presented here use different algorithms so it doesn't really make sense to compare the languages like this. The slow Go version (~15s) doesn't create any string (so no allocations) but still has to write to an internal buffer to make the result of Scan() available to the user. The fast Go version (~0.3s) doesn't allocate anything at all besides the buffer at the very beginning, the only thing it does is count the number of \n. The JS and Gleam versions have to create string object in memory to represent the lines so they have to call the allocator at least a billion times.
  • @deedlefake
    "gofmt's format is no one's favorite, but gofmt is everyone's favorite." -- Go Proverb Also, a clarification: scanner.Text() isn't reading the file. The bufio.Scanner is already reading it into an internal buffer. scanner.Bytes() would give you access to that buffer, while scanner.Text() copies the buffer's contents into a string. It's that allocation and copy that's probably causing the extra second of runtime.
  • @pedroalonsoms
    The JavaScript church won’t forget about this unloyal event !
  • @3DArea
    It's lovely to see Theo so heartbroken when he realizes that numbers doesn't match his personal preferences. We all have been there (hello Haskell devs). I know it hurts, here is a hug (づ。◕‿‿◕。)づ
  • @SnowDaemon
    Gleam is a really attractive language. If youre a complete beginner to programming, i wouldnt suggest it, as it is functional (no "for loops" or "if statements") and also doesnt have great learning material yet. but if youre already a programmer...its lovely.
  • @roguesherlock
    doesn't go ship with lsp, formatter, test suite, cross-arch-compiler, package manager, cache for hosted package, docker images, etc and so much more? I wouldn't say they don't care about dx, but yeah they could definitely add watch mode.
  • @opposite342
    16:48 basically options/results by default Also omg Giacomo being the goat once again (refers to prime's reaction on new gleam features)