4·
6 days agoThen it’s probably time to switch boards. Anything that will expand drive connections will need a PCIe slot. Using an adapter that goes into an m.2 is probably going to have issues. Using USB based devices is asking for trouble.
- 4 Posts
- 148 Comments
Joined 3 years ago
Cake day: June 18th, 2023
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
Yep. I’m using one myself.
If you are intending to run software raid then you want something that can be flashed to Initiator Target or IT mode. There is a JBOD function for some RAID cards and you DO NOT want that because it does not send all the same data that software raid wants from the dive. You also do not want to waste money on a card with RAM or Battery Backups since they are really designed for hardware RAID.
A SAS3008 card will really be what you are looking for.
https://www.ebay.com/shop/lsi-sas-3008?_nkw=lsi+sas+3008
With a SAS backplane or expanders you can connect 256 drives.
I have so many irons in the fire I can never quit…
Right now I am tinkering with a local only HomeAssistant AI voice assistant. I have an Ubuntu VM running docker on my TrueNAS which has an ARC A310. The reason I’m running a VM is because I’m using A Dual Edge TPU module for FrigateNVR while using the ARC A310 for video decoding. Ollama is running Qwen2.5 3B. I’m also running Wyoming Whisper and open-webui for the voice AI. I’m using a ESP32-S3 AI Smart Speaker Dev Board for voice control.
I found a project on GitHub that looked really good but was designed to always pull files and so I forked that to create a local only version and setup a small docker container for the audio files. Still working on modifications of the repository but the setup is working OK but I’m trying to work out the bugs.
I’m also working on a Halloween project. I have a plastic pumpkin with a big red arcade button for a nose, LED’s for eyes, a speaker, some car horns, proximity sensors, and a temper… I call it my Doom Pumpkin. It will go outside a week before Halloween and complain about people touching it’s buttons or lack there of during the day and stay quiet at night. On Halloween it will keep going till later and when someone presses the button it will randomly honk horns in different ways. When someone steps in front it will say funny things to get them to press the button. If they walk behind it will warn them something bad might happen (like saying it farts fireballs.) It’s all setup in ESPHome and will be controlled by HomeAssistant and NodeRed. There is also a theft prevention mode built in that will cause it to just spazz out and blow the horns.
I’ve still gotta pull another 500 feet or so of ethernet then get those and some others dropped into the server closet and shift some stuff around so it feeds into the rack. I’ve also got to pull my current camera lines from my POE injector and run them into my POE switch.
This is all on top of building and installing racking for solar, getting the solar installed, pulling a wall in a couple inches at the top where it pulled away during an earthquake before we bought the place, and fixing a bunch of other stuff.
Supermicro boards go great in Supermicro 4U cases. I’m using an X10DRH-C in a case that holds 36 X 3.5" drives with a SAS backplane. You can grab a CSE-847 for 350 plus tax and shipping on ebay.
A set of cheap rack rails attached to some 2x4’s with a couple casters on the bottom and you have a rack you can stick in a closet and pull our when you need it.
The cheap rosewill cases are ok. But it you need to swap a drive (I have the 15 bay version that’s about 100 bucks I used for my server to begin with and now use it for a router with only two drives) it’s a pain to deal with. You have to pull the top, then remove screws to pull the cage for 5 drives, then pull the drive that you need to swap. It doesn’t sound like a big deal til a drive goes bad, or one of the 120mm fans that cool the drives dies.
The Supermicro is the bottom one, the Rosewill is above it.
They also make wall brackets you can hang the chassis on and it can sit on a desk as well.
The one built in to Caddy for anything that doesn’t really have authentication. Everything else uses its own and i only have a couple services accessible outside the home.
Most everything is only accessible within my home and my guest wifi is on its own vLAN. Even then I’m the only person who actually uses most things that I setup.
It’s one of the reasons why getting a SAS controller is really nice, backplanes make it even better.
I was preforming a burn with 10 X 8TB drives on a controller that already had a couple operating ZFS pools totaling 18 X 4TB drives and a couple SSD’s and there was no slow downs. A SAS3008 controller can support over 6000 MB/s and has a PCI Express X8 3.0 bus.
I’ve been building my servers and network out of old enterprise gear for a while now. It uses more power but the things you can do are truly amazing. Sadly the ai boom drove prices way up compared to a couple years ago, what I bought for 900 in 2024 is running around 1500 today.
I understand. If you buy a used server then add some drives later you can have a great NAS IMHO. I upgraded from an X8DT6-F with 384GB of RAM and a pair of Xeon X5690’s right before things went sideways. The MoBo has the SAS controller already flashed to IT mode so it’s ready for ZFS. But it’s not exactly light on power and with a 2U chassis and a handful of used 8TB SAS drives you are looking at around 1200.
My current server is a X10DRH-C with dual Xeon E5-2683 v3’s with 128GB of RAM in a 4U chassis with 11 X 8TB SAS drives in a RaidZ3 configuration. Just the MoBo, chassis, cpu’s and 64GB of ram is running about 1150. The drives used are 110 each and before you think you should just get those, SAS drives require a SAS controller and you only get those in enterprise equipment.
But if you can pick up a little here and a little there you can have a nice system. But right now isn’t a great time to get in the game.
I’m not saying that.
What you need to do is decide now if the drive you will buy will be used for a RAID array. If it is a desktop drive won’t be in a RAID array on a NAS system. Many NAS’ will have random writes to the pool. Desktop drives aggressively park the heads, the load and unload of the heads wears them. In a NAS system they can actually wear out.
Over the last 15 years drives have become a bit more specialized. You already found out about surveillance drives not being a good fit for much other than surveillance/DVR. Desktop drives are fine for desktop loads and usage but outside of that or single drive usage they are not useful. NAS drives are meant for NAS usage in RAID arrays. Back when the WD green drives were available years ago you could convert them from a desktop drive to a NAS drive using a tool called wdidle (WD Idle) but that isn’t the case any more.
Using a NAS drive on its own will work in a pinch but if it has an error it won’t try to recover it like a Desktop drive would because it’s made with the idea that it will be in an array that will deal with the issue. Plus once you start loading it up you will have to wipe it to put it into an array unless you go for ZFS mirrors or RAID 1. If the NAS Appliances have some sort of special trickery that allows you to expand one disk at a time and add redundancy I’m completely unaware as I’ve never put much stock in them. I’ve been running FreeNAS/TrueNAS for over 10 years.
If a NAS or Enterprise drive has an error it sends the information to the host to be logged so that the end user can have the information available.
So like an Unrecoverable Read Error (URE) pops up on a sector. A drive that is built for RAID use will just say, “Couldn’t read it” and moves on. A Consumer drive meant for a desktop will try and try and try and try to read that bad sector. In a NAS situation where another drive will be able to fill in the data the controller (hardware or software) will just deal with it by pulling the data from another drive and keep moving.
The drive may not be bad as a whole but it does mean that over time it is more likely that drive will have more errors.
NAS drives are not inherently more reliable, yes they can deal with a bit more vibration and such but it’s the firmware inside that is different. Enterprise drives are another step up again from NAS drives.
What are you using as a router? How are you you trying to setup a reverse proxy?
I will be honest, I would do it for free. Money is nice but helping people is more important to me than that. However my understanding is with OpnSense and using Caddy for reverse proxy with the ACME client for Let’s Encrypt certificates. Beyond that I am outside of my element.
It’s not just about the advanced functions. Many of the older more basic tools are single threaded which will potentially limit the performance. As you figure things out you will want to do more and you may find the current tools more of a limiting factor. But the choice is yours, I have 2 48port gigabit switches and WiFi SSID’s that connect to specific vLAN’s through tagging. I started with some dumb switches and added my 10Gbps backbone switch which I used as a dumb switch for years so I could connect my desktop and server over a faster connection.
In my equipment an untagged port is what a port is where the vLAN is stripped away. A tagged port has the vLAN tag passed to the device. If you can set multiple vlan tags on the same port that port becomes a trunked port. You may also be able to set a vlan as untagged on that port, if a device is plugged into that port it will by default be on the untagged vLAN. If the device is able to handle vLAN tagging it can live on the vLAN’s you set up as well.
It took me a bit to figure it all out and get it working. I spent about 20 hours configuring things before I started making the switch from a single net to multiple vLAN’s. I spent another 8 hours making the change and 5 or 6 more tweaking things.
No. I didn’t use any video’s to set mine up and wouldn’t even know where to find one that is up to date.
Dnsmasq isn’t being deprecated that I know of but when you begin doing more advanced functions some tools work better than others. The “New” rules are fully functional and I suspect the old ones will slowly be removed in later releases. They are revamping some stuff, ISC DHCP used to be the go to but that is being passed out by the creators. https://www.isc.org/kea/ and OPNsense is cleaning up things so it all works with the rest api and is higher performance.
Trunking is when a port can deal with all setup vLAN’s not sure about your switches as they are pretty basic looking. I’m using a pair of Dell PowerConnect 5548’s and a Quanta LB6M which are much more complex. But basically think of a vLAN as a branch of a tree and the Trunk is the base that connects to everything.
Ok, so first of all a TON of things have changed in OPNsense in the last couple updates so you may just want to pull everything you did. It looks like you are using the old firewall rules which while they are going to stick around they are trying to migrate people away from them.
You should be using KEA DHCP as that is the modern and latest and greatest DHCP server that is vLAN aware. Also UnboundDNS is a recursive DNS server and designed for modern networks, with the advantage of being able to use DNS Blocklists to block ads and other “junk.” Just remember that the blocklists live in RAM so if you don’t have much RAM available I wouldn’t recommend using them.
NOTE: If you are using Dnsmasq DNS&DHCP you will have to turn it off before you can enable KEA DHCP and UnboundDNS as it will tie up the ports needed and they will fail to start. You can copy everything over before making the change and if there is an issue you can switch back just by stopping the new ones and enabling the old one.
With that said one of the biggest things is that with a managed switch you will need to trunk the port that OPNsense is plugged in to for the LAN unless you are using multiple ports on your OpnSense install and then those ports will need to be properly tagged and you will need to trunk any ports that are linking switches together you will have to figure that out on your own but I suggest grabbing a copy of your switch’s manual (and if you use a chatbot upload that file to it for help.) Then you can use vLAN tagging for each port that you want to receive an IP address from a particular pool automatically. You can also trunk ports that you want to use for management but set the default the port will use for access so the device can get an ip via DHCP, this really only works well with Linux. If you are using windows you will need to just create firewall rules that allow your device to talk to the other vLAN’s instead.
When you create your firewall rules you have to understand that you can only preform one action per rule. If you want to allow your vLAN 1 network to talk to vLAN 10 that is one rule. To allow vLAN 10 to talk to the internet that is another rule. You can use floating rules to do the work on multiple vLAN’s but that should be limited. If you select more than one network interface a rule will becoming a floating rule and will process before other rules so if you create a rule to block something later on but have the same interface set on a floating rule the block will not work, it’s better to enable piece by piece than to blanket enable and then try to block. With 5 vLAN’s I have 5 floating rules and 34 regular rules plus 40 automatically generated rules (which handle things like allowing DHCP access and basic protections.)
Here is a firewall rule that allows my “Trusted” vLAN to access my “Camera” vLAN as an example. The Categories are not important but make finding what a rule deals with later on a lot easier, they are set under the firewalls - categories. You should also use good descriptions for this reason.
You will also have to explicitly allow access to services like DNS. This is how I am allowing my “Trusted” vLAN to access DNS services on my OPNsense.
This is how I allow my trusted network to access the internet. If you have multiple WAN’s you can choose a specific one or if you have failover configured you would likely select the “group” you created when you setup the failover.
If you need more help let me know. If you have been tinkering with a bunch of stuff you may want to start over, just backup your current configuration and reset everything to defaults. If you can’t figure it out you can reset to defaults and restore your configuration.
Give me a bit and I will be able to help you. I have a multi vlan setup with vLAN tagged switches and wifi ap’s.
Realtek stuff is a major pain it seems. I’m guessing that is why FreeBSD is will known for being fine with nearly anything but realtek.
Not that I really use it by my AMD MoBo runs an Intel NIC for a reason. I started 10G networking a while back. Was a pain to get my Mellanox ConnectX3 working on windows. Linux didn’t even blink.
I tried switching a long time ago. And every 3 or 4 years since. I’m happy I have. Things got a lot better in the last few years.
I was previously using a GTX1080ti with a Ryzen 5800X.
Like you I dabbled with linux but could never get things running to the point where things just worked without constant recurring issues. At one point (during the burning crusade era) I had World of Warcraft running on a Linux system.
I’ve been using Ubuntu Cinnamon 24.04LTS for around 18 months now as a daily driver. I use Ubuntu because it’s stable and I can modify the kernel for my current FrigateNVR docker container. I use Cinnamon because I don’t like searching for stuff and am used to the XP and 7 style desktop.
I still have dual boot setup mainly because I have some crypto wallets that I need to move to a client on Linux but haven’t booted windows in months. I do some gaming, a ton of video editing, and manage my smarthome/network and devices which includes flashing esp devices from my Linux computer. Does a Steam Link work? No clue but I use an old laptop as a thin client with the Steam software allowing me to stream games from my desktop with no issues so as long as it functions with current games etc. I don’t see why not.
For programming… well it’s Linux so yeah I’m sure it will do that and more if you get the right tools.
My stepson uses ChimeraOS for his gaming system which is a Ryzen 1600 that overclocks like a beast (I used it before the 5800X and was putting numbers out that were trading blows with a Ryzen 1700X with it) and a RX7600 with 16GB of RAM. He had used that for about 5 years now and the only issue was support got messed up for the RX480 8GB that I had put in it and the network goofed so I had to remove the connection and then add it again.
The solution is to boycott software with denuvo… if they keep making money they will keep breaking games.
No problem.
I try to explain things as best as I can so everyone can learn. I’ve been using TrueNAS since the FreeNAS 9.2 era so about 13 years now. When I started I had already dealt with a ton of issues and was searching for better than multiple single drives. My first real build was a Dual Xeon X5690 setup in an X8DT6-F so learning this about SAS was really important.
Sadly there are so many acronyms out there (and growing all the time) that is getting hard to just read the news let alone anything else without a little help.