from SpookyMulder@twun.io to selfhosted@lemmy.world on 08 Jul 14:07
https://twun.io/post/6565
Background: I’ve been writing a new media server like Jellyfin or Plex, and I’m thinking about releasing it as an OSS project. It’s working really well for me already, so I’ve started polishing up the install process, writing getting started docs, stuff like that.
I’m interested in how other folks have set up their media libraries. Especially the technical details around how files are encoded and organized.
My media library currently has about 1,100 movies and just shy of 200 TV shows. I’ve encoded everything as high quality AV1 video with Opus audio, in a WebM container. Subtitles and chapters are in a separate WebVTT file alongside the video. The whole thing is currently about 9TB. With few exceptions, I sourced everything directly from Blu-ray or DVD using MakeMKV. It’s organized pretty close to how Jellyfin wants it.
What about you?
threaded - newest
My Jellyfin library:
1,152 - Movies
552 - Shows
37, 062 - Episodes
491 - Albums
6,558 - Songs
362 - Music Videos
14 - Concert Films
Files are a mix of 1080p and 4K. 264 and 265. Standard and REMUX.
Total space used is currently 149.90TiB
About the same here, minus the music videos (only a few dozen there for the kids), plus a fitness library, so I’d say it evens out to roughly equal.
Mostly HEVC but I still have some h.264 floating around that I have no interest in reencoding.
No AV1 at all until I get a new Intel GPU or newer Intel CPU to handle transcoding it nicely.
There’s some relatively inexpensive NVIDIA cards now with AV1 hardware encoding. I’m on my third round of re-encoding my whole library (HEVC, then VP9, now AV1). For 1080p NTSC, I get about 13x speeds on NVENC AV1, whereas with VP9 I was CPU-bound at around 4x. Definitely worth the upgrade, in case you’re on the fence.
Yeah I just dont have a need with no devices to handle it natively, while the rest of my library can be. Building a new htpc media player for the living room next, new server after that.
New because I’m using a lenovo tiny as the server, which means either I build a new box completely, or I find the right used workststion tiny/mini/micro that can handle av1. Complete build will do a lot more (well, the t/m/m does too, but not to the extent my big box builds are set up for).
So what card are you using?
I went with a GeForce RTX 4060. Cost was about $300.
Cool, thanks!
You do realize that you lose quality with wach encode, right?
It’s not AS bad when bitrates are high, but it’s still there.
True.
When I migrated off of Jellyfin, I re-encoded everything up to that point directly from the Blu-ray rips wherever possible. Because I’d already started culling those for space, I did end up just doing another pass on the first round of encoding for a portion of the library. There’s some noticable degradation on those, and I’ll want to re-rip those at some point.
Fortunately, I’ve got my process pretty dialed in for ripping and I actually enjoy it, so if I ever have a quality issue, it’s not a huge ordeal to re-rip and encode.
Ahh, I like how you split Concert Films and Music Videos. I’ve been pigeon-holing my Short Films, Mini-Series, and TV Movies into just the two categories: Shows and Movies. Makes way more sense having separate categories.
Kinda unrelated to OOP, but out of curiosity, what does your storage setup look like? Do you keep stuff reasonably backed up with that much data?
Ah yes. My storage system is 2 x Supermicro CSE-846 cases. Only one has a CPU and motherboard, the other is acting as a plain Jane JBOD.
Hard drives I have 21 x 8TB 7200RPM mix of Seagate and Western Digital and 4 x 16TB 7200RPM from Seagate. I use mergerfs and snapraid. Mergerfs presents all the 21 8TB drives as one mount point. Snapraid uses the 4 16TB drives to provide 4 parity drives. Note that snapraid is not live and the parity is only updated after running a “snapraid sync” which I run nightly.
I only backup my songs and music videos. The rest is easy to get again. I have a script that generates a list of every single file I have each night. So if the day comes it wouldn’t take too long to get back to where I was. The other reason I use mergerfs is if 1 drive dies, I only lose the files on that one drive and not the entire array. The truely important stuff such as tax documents, mortgage details, family pictures, will & estate documents are stored on a 2 x 8TB RAID1 and all backed up nice a safe using Proxmox PBS. The PBS datastore is synced to 2 remote locations as well as to external drives that I keep offline and rotate.
Thanks for the detailed write up!
Nice write-up. I thought I had a large library (24TB) and my off site backup is starting to get full. I backup everything though but I have long debated on if there’s a point of keeping movies and TV since they’ll likely always be available. Anyway, I never thought of generating a list of files and eliminating the stuff that’s not particularly important. Good idea.
I used to back everything up before I broke the 50TB mark. Just can’t justify it now. I even looked at LTO drives for backing up the multimedia but they’re still to expensive for the higher capacity drives. And then you need tapes…
All the truely good content will always be out there somewhere on the net.
The script I use to generate the file lists is very very basic. Nothing special no formatting the lists or anything since it’s just for that oh balls, everything is gone scenario.
ls -alR /mnt/volume1/media > /mnt/volume2/backups/file_lists/media.txt
ls -alR /mnt/snapraid/data* > /mnt/volume2/backups/file_lists/snapraid.txt
Those text files are also part of the files backed up with PBS so I can always go back and restore previous versions of them. You may ask why I generate the list twice? The first is just everything inside the media folder on the volume1 mount point. The second let’s me see what files are on each individual drive so if only 1 drive dies I can just grep the text file and output to another text file and show me what is on that 1 drive.
i just got my music server going recently, let me see that i had like 19 thousand songs!
What do you use for your music server if I might ask?
i settled on gonic via docker, for now, with jukebox mode controlled from dsub2000, as the device I’m using as the server is an old autonomic mms2a running debian. (not enough horsepower to run a full DE so it needs to be remotely controllable)
i just about drove myself crazy trying every single version of any subsonic related server and client, jellyfin (still running) ampache, mopidy with plugins, addons for kodi, various MPD iterations and control points, including trying to drive it from home assistant.
i am not sure if I’m completely happy with my setup, but with vpn, i can listen on the phone anywhere, make playlists that are immediately visible on the machine that can play to my speakers in the house (driven off of a matrixing amplifier)
i am giving my brain a rest before i try to look into snapcast
Movies: 7796
TV Series: 1443 (4128 seasons, 49344 episodes)
Music (tracks): 37909
All up its pushing 45TB currently. All legal backups, obviously.
I’m trying to get all 1080p 10bit 5.1 x265 for tv and movies, but am not converting 264 -> 265 myself as it would take forever and is lossy. Sonarr and radarr will take care of it eventually anyway with the way I’ve set up my profiles.
Subtitles are usually SRTs grabbed by Bazarr, stored in a subtitles folder inside each movie folder.
Folder structure is just the standard folder per movie, and folder per tv series with sub folders per season.
Music is 320kbps mp3 where possible, and for the last year or 2 I’ve been trying to get FLAC and then convert to mp3 (automatically) and archive off the FLAC for safe keeping.
Whenever the 265 successor comes out I’ll look at upgrading to 4K if the space requirements are not crazy. With the price of storage and large bay NAS/DAS devices there’s just no way I could do 4K as it stands.
~2000 movies ~200 tv shows
Many English only, many German and English, some German only. A few in different languages, if it’s the original language.
~50TB
Mostly 1080p h264. Lately, due to free space running out, I have started prioritizing and redownloading accordingly. Low bitrate h265 1080p for less important stuff, 4K h265 for important things and normal bitrate h264/265 (preferably the latter) 1080p for everything else.
50TB?
Dang, thought I was doing well at about 5TB,haha
I know, right? I feel like the little guy in the memes next to the Giants in armor.
I thought 20TB of storage would last me forever
I've had low storage warnings for years now
Lol. I feel your pain.
I setup a 2.5TB RAID box in 2011, thought it was going to last a while.
Now my server has a single 8TB data drive, my NAS is 7TB, and I have 2 4TB drives and everything is replicated between them.
Now I need to build another NAS as all this stuff is aging.
Save yourself time on downloading and look into tdarr
1,028 movies
517 shows (20,702 episodes)
Shows are all 1080p or lower except a couple seasons of select shows in 4k. Movies are 4k HDR when it’s available, otherwise best quality I can find.
I use Jellyfin because of the client apps and FOSS nature.
I tend to prefer HEVC/h.265 encodings for the strong trade off between player compatibility and smaller size for the quality level, but h264 and AV1 are also both in my library. I don’t reencode anything except through the Jellyfin server transcoding.
Nice try Universal Studios!
Emby Server
382 Shows 30130 Episodes
1703 Movies
24740 Music Albums
Most are downloaded with *arr apps and are random quality. I shoot for 1080 for shows and movies but for the really good stuff that I personally like I will get the 4K version.
My
pornmedia library is roughly ~500GB right now.~3tb of a jumbled disgusting mess of miscellaneous files. Somehow it all works on jellyfin though.
I’d be down to try something new if you do end up releasing something. Jellyfin works just fine but I’m not in love with it.
80K+ hires music flac - no movies - no episodes, no shows, no podcasts, no youtube
About a gig of books, and one movie called “Lensman: Secret of the Lens”
about 8TB
1911 TV shows (65728 episodes)
2294 Movies
5051 Albums (66644 songs)
65.37 TB total.
I use Tdarr to transcode everything in VP9 (can play in a browser and doesn’t need transcoding from Jellyfin).
Audio is AAC 2 channel (I keep the original audio track and add the new AAC). Subs are in SRT.
Everything is made for play from a browser without issue. I use Infuse on my Apple TV and ether never the web player but when my family watch something form Jellyfin wathever the device no trancode needed.
TV Shows : 172 | Movies : 394 | 7.2 Tib
Actually, not all files are transcoded the process is very slow. All files are stored on my NAS (Synology DS918+) with SHR-1 (hybrid RAID with 1 drive fault).
I use Janitorr, he removes old files when I run low on space. This is why my library is not big.
Feel free to ask if you have questions.
Sorry for my English.
Wow, thanks for suggestion of Tdarr — that project indeed looks very nice. What is. your experience using it? Any quirks?
I second this. Never heard of it but looks great.
I tried tdarr, but have issues using more than one node. I may just wind up installing docker on my more powerful desktop specifically for tdarr, instead of on the proxmox server I have without a real gpu. (It’s a Xeon Supermicro board with their onboard VGA)
Playing files directly in the browser and avoiding the need for transcoding is exactly what the system I’ve built is designed around, so I get the appeal!
Could you share your plugin setup/flow? I’d like to have tdarr do exactly what you have it doing.
4 direct tv boxes worth
cries in broke
I have 4x3TiB drives in a currently-degraded RAIDZ1 due to a hard drive failure. I have a replacement coming, and my fingers are crossed that I don’t lose another drive beforehand.
Just started with YAMS using Plex 2 months ago:
Movies: 241 TV Shows: 30
About 3.5 TB on an 8 TB drive
13200 movies 1200 shows
Over a 1/4 PB of data.
OK Netflix, you don’t count 🤣
Haha. Thanks. I really didn’t want to pay Netflix or any other streaming service. But it might have been cheaper than hdds and electricity.
This is something I’ve been building for over 10 years at this point. I’ve gone through so many iterations of servers and storage architecture. I’ve lost my entire TV and movie library multiple times. (I don’t back it up because a. It’s expensive at this scale and b. this data is easy to rebuild over time.)
It’s been a part of learning about hosting and data management that I’ve brought to/from my work.
I’m in a similar place with books and comics. Of course, nowhere near the Jupiter size collection of media you have, but easily 2.5TB+ of just books and comics.
My wife, my kids and I are all avid readers, so we are always sharing some book or comic arc. We’re all rerunning all of Lobo’s arcs, it was Deadpool a couple of weeks ago (that should tell you all anyone needs to know about our family 😜)
You say that, but some shows and movies are getting very hard to find these days. I lost one show to an external HDD that died, and I have never been able to find it again :(. That’s part of the reason I recently went to a DAS in my setup instead of a NAS - backblaze backs the entire 40TB DAS up for ~$90 a year!
What did you lose?
And there are plenty of shows that I’ve had to manually recover at times. And plenty more that I can’t find a copy of at all. I don’t get too attached to much, and those I’ll usually be seeding on multiple machines myself at that point.
Why start anew instead of forking or contributing to Jellyfin?
The short answer is because it’s a fun project, and I wanted to see if I had it in me to make exactly the media server I want.
The longer answer is that I wanted something dramatically and fundamentally different from what either Jellyfin or Plex have to offer.
I don’t see either of those goals happening with a contribution or fork, because achieving them would require some dramatic feature deprecation.
Okay that’s gotta be radically different!
Does it not need to transcode then if it runs on cheap hardware?
Yep, transcoding is the main reason I had to buy any new hardware when getting my library going with Jellyfin.
For me, the main draw of Jellyfin wasn’t the transcoding. It was being able to browse and stream my library from anywhere. My partner and I would alternate weekends hanging out at each other’s places, and we just wanted access to the library from wherever we were and whatever device we were using.
I was willing to put up with weeks of encoding to get everything into a web-compatible format. But that’s just me and I know it’s not for everyone. I’m curious where the palatibility for that is on the spectrum more broadly.
Sometimes I hear about other people's storage setups and I think, "that is overkill, no one really needs that." According to this thread, I am quite mistaken about that. 😳
I have 2,057 songs, taking up a measly 51 GB, on a Funkwhale server. No movies or TV shows.
That should get a little larger soon. I have about 100 vinyl records that I want to make digital rips of.
I have single movies that are larger than your entire song library.
I'm kind of surprised that it's only 51 GB. They're all FLAC files ripped from CDs -- I was expecting like 300 GB at least.
So apparently this 1TB SSD is going to last me a while. :P
Lawrence of Arabia 4K remix is so fucking crispy.
Movies 1127 TV Shows 96
Nice try FBI Agent.
4TB mostly TV, then movies, then a distant third is music. Novice at all, tried remuxing a few things that didn’t work. Everything works on jellyfin android and PC. Android TV jellyfin is frustrating, some things don’t play so well
12.8TB. Mostly uncompressed rips from Blu-rays, some DVDs, some from iTunes Store. Some from the high seas, but not in a long time because the market solved that problem with streaming.
2.71Tb/515 series for TV, 6.28Tb/1176 titles in Movies.
Almost everything in MKV because that’s what I prefer.
I use Plex so it’s organized according to their requirements.
Everything is stored with a redundant backup on a Synology NAS with 6/9 HDD bays filled, totaling 48Tb in total storage space.
I run two servers (one on the Synology, one on a NUC-type Asus box) along with all my other systems.
Oh, and I have dual antenna tuners connected as well for live TV, DVR and playback.
650 shows, 1400 movies, 1450 anime. Take up like 130TB or something
Since you listed them separately from shows does the count also include anime movies?
Still, if I add up both my shows and movie I’m at 1447. Damn you!
You got them all in uncompressed 8K or something!? How on earth does it take up that much space?
About 18 TB.
Nowhere near as big as yours. I haven’t bothered checking, but probably something like 100 movies and about the same number of TV shows (only a handful of series). It consists pretty much only of what I’ve ripped from physical media, plus a handful of things my SO uploaded. Total storage is about 2TB, and mostly DVDs w/ a handful of Blurays. Rips are full quality, and mostly ripped from MakeMKV, with a handful ripped w/ Handbrake.
We don’t watch a ton, but I do order new stuff periodically, so it slowly grows (most recent addition is Adventure Time).
My Linux ISO collection take up around 12TB, 268 of smaller ISOs, and 751 big boi ISOs.
Mine is sitting around 10TB, mostly podcasts and a few videos like graduations.
~70TB, ~2500 movies, and ~250 series with , varying quality, I’m still trying to replace lower quality stuff with better versions
My library is almost cracking 18TB. Backing up all documents, pictures, videos and profile/settings dumps for apps and laptops. Also have plenty of moving Linux ISOs, 1359 longer isos and 269 smaller iso series.
Like 7
150 shows, 4000 movies. 25k songs. Tightly curated, playlists, mostly 1080p with truly important stuff 4k.