Installing and Running a Usenet News Server with INN and FreeBSD John Kozubik - john@kozubik.com - http://www.kozubik.com June 6, 1999 Introduction This document is intended for users who have a basic understanding of loading, configuring, and maintaining a FreeBSD 3.x or 4.x server, and wish to run such a server as a Usenet news server. These step by step instructions are accompanied by some explanation and background information / theory. However, this document is not a comprehenzive tutorial on the care and feeding of every type of Usenet news server. The reader is encouraged to study the other references listed in Appenix A. These instructions are applicable to the INN 2.2.x news server software, and the FreeBSD 3.x and 4.x operating system revisions. These instructions assume that you have control of the server you plan to run this on (you are root) and that you have an upstream news feed that you are authorized to use. Concepts In the simplest sense, we will be performing the following steps: 1. Configuring a disk subsystem that will handle a news feed 2. Installing the news server software 3. Configuring the news server software to receive news from the upstream newsfeed 4. Configuring the news server to spool news to disk in an acceptable manner 5. Configuring the news server to send articles to news reader clients 6. Instructing the server to run certain maintenance commands on a regular basis This document is in a step by step format, but there are a few concepts that the reader should understand right away: The Disk Subsystem Usenet news fills up disk quickly. It is entirely possible that your newsfeed could send you 4 megabytes of news per minute (or even much more). Therefore, it is essential that you configure a large amount of disk space for news in addition to the amount of disk space that you use for normal operating system installation and use. RAID storage methods are generally not needed - for reasons that will become clear in the next section where we discuss 'Overview vs. Articles'. However, if fault tolerance is important to you (if news is a mission critical appliation in your organization) I would recommend RAID level 1+0 (a stripe of mirrored pairs) over RAID level 5, because of the poor performance of RAID 5. Overview vs. Articles In this document, we will be using something called the CNFS storage method. This means that we will set aside a number of large files (usually each file is the size of an entire disk) and then collecting all of these large files into one large pool of space for articles. This is the reason that RAID is not particularly important. Although our total article storage space might be 100 gigabytes or more, it can be made up of 25 different 4 gigabyte drives, each with a 4 gigabyte spool file on it. When the news server is configured, it is told about each spool file, and when it runs, it spools articles across each of these files in a distributed manner. This is why RAID is not necessary to create a large pool of disk space - the news server aggregates the disks on its own. This large spool space is just for articles though - and articles cannot be given to clients without an accurate index of where those articles are in the spool space. This index is known as the 'overview' and the overview resides on the /var filesystem. Because the overview is simply a collection of pointers to news articles in the spool, it is much smaller - however, it can still grow quite large. When the spool space becomes filled, the earliest articles in the spool are expired and go away to make room for new articles. Therefore, you do not need to worry about running out of spool space. When these articles go away, their corresponding links in overview also go away. There will be a problem, though, if you run out of space in /var for overview before the spool fills up. So the important question is: "how do I know how big to make my /var filesystem to hold overview, given a certain amount of spool space". This is a very hard question to answer. For instance, if your server only receives articles from the alt.binaries heirarchy, then your spool space will fill with these large articles before your overview space will fill with the pointers to them. On the other hand, if you receive articles from everything except the alt.binaries heirarchy, then you may have plenty of spool space for these small articles, but your overview may fill up. It is generally agreed that servers receiving a complete news feed (all available newsgroup heirarchies) should have a 9:1 ratio of spool space to overview space. This means that if you normally like to have one gigabyte of space for your /var file system, and you have 50 gigabytes of spool space, you should atually make your /var filesystem the original one gigabyte _plus_ 5.5 gigabytes (50 divided by 9) for overview. Finally, there are additional files that grow alongside your spool and overview that reside in the /usr filesystem - I would recommend a /usr filesystem that is one half the size of the space reserved for overview (in addition to whatever space you normally set aside for /usr). Our Imaginary System Our imaginary system that we will use throughout this document is an Intel-based PC server with 10 four gigabyte hard drives attached. Our imaginary system is named news.example.com, and has an IP address of 10.10.10.10. Our newsfeed is named news.upstream.com and has an IP address of 20.20.20.20. Our client has a workstation with an IP address of 209.209.209.209. Step By Step Instructions 1. Given that we have 10 disks of 4 gigabytes each, we will set up our file system as follows: Mount Disk Size ------------------------------------ / da0 500 (swap) da0 128 /usr da1 2000 /var da2 4000 /mnt/da3 da3 4000 /mnt/da4 da4 4000 /mnt/da5 da5 4000 /mnt/da6 da6 4000 /mnt/da7 da7 4000 /mnt/da8 da8 4000 /mnt/da9 da9 4000 If you are using the normal FreeBSD installation process (known as /stand/sysinstall) then you can create this collection of mount points by doing the following: After the operating system boots from the 'kern' and 'mfsroot' floppies, you will arrive at the '/stand/sysinstall Main Menu'. Select #4, custom, and press 'enter'. Select choice 2, Partition, and press 'enter'. If all of your disks are correctly attached to your SCSI chain, and are working, you will see a list of all 10 disks, with a checkbox next to each. Select the first disk - you will be taken inside of the 'FDISK Partition Editor'. Delete any existing partitions and then press the 'A' key - you will be asked about 'dangerously dedicated disks' - answer 'No'. Then press 'Q'. Do this for each of the ten disks. You will now be back at the 'Choose Custom Installation Options' screen. Choose 'Label' and press 'enter'. You will now be inside the 'FreeBSD Disklabel Editor'. In this interface, you can choose each disk one at a time, and press the 'C' key to create a partition. First, you will choose da0 and press the 'C' key - you will be asked to enter a size - enter '500M' for 500 megabytes, then press enter. You will be asked if this is a filesystem or a swap partition - choose filesystem and press enter. You will then be asked to enter a mount point. You should enter '/' and press return. You should do this for each partition that is listed in the table above. 2. Install the ports collection. If you are installing from scratch, this is easy to do - if you have already installed, you can also add them. NOTE: It is highly recommended that you install the INN port and install INN using the port - you can, of course, pkg_add the INN package, but you will find that this method does not create all of the directories and file permissions that this document and the instructions in here assume exist. Further, the process of creating all of the structures that the pkg_add method leaves out is long and prone to errors. Installing the INN package via the pkg_add tool is beyond the scope of this document. 3. Install INN. Do this by issuing the commands: (become root) cd /usr/ports/news/inn-stable make install 4. use the 'vipw' command and check the entry for the 'news' user. Make sure the entry looks exactly like this: news:*:8:8::0:0:News Subsystem:/usr/local/news:/bin/sh If it does not, change it so that it does. 5. Place your active file into /usr/local/news/db Note: the file named 'active' is something that you need to get from your upstream news feed. Sometimes they have the file available via ftp on whatever machine the news feed is on, and sometimes you need to contact your feed and request it. Note: You may wish to edit the active file that your news feed gives to you. Note: Remember to use binary mode when transferring the active file via ftp. If you have no way of getting an active file, but would like to get the server up and running, you can just make a file containing the following lines: control 0000000000 0000000001 n junk 0000000000 0000000001 n 6. Issue the command: chown news:news /usr/local/news/db/active 7. Issue the command: chmod 0664 /usr/local/news/db/active 8. Issue the commands: cd /usr/local/news/db /usr/local/news/bin/makeactive -m -o >active.new touch /usr/local/news/lib/history su news -c "/usr/local/news/bin/makehistory -n -f history.n || echo bad" su news -c "/usr/local/news/bin/makehistory -r -i -f history.n && echo good" mv history.n history mv history.n.dir history.dir mv history.n.pag history.pag 9. We now need to make a spool file on each of the spool disks. In our example, the first spool disk is mounted at /mnt/da3 - the following commands will create the spool file for that disk: dd if=/dev/zero of=/mnt/da3/1 bs=1k count=4000000 Next, we make a spool file on /mnt/da4 dd if=/dev/zero of=/mnt/da4/2 bs=1k count=4000000 And so on, until all of our spool disks have a single file that is roughly the size of the disk. Note: There are a few things to keep in mind when creating these files. First, note that although we are operating on 'da3', the name of the file we create is '1'. On 'da4' the name of the file we create is '2'. This is completely arbitrary - we could very well name them 'one' and 'two', or 'green' and 'blue'. Second, I entered 4000000 as the size, but most likely the space available on your disk will not be exactly 4000000 kilobytes - issue the 'df' command and look at the space available on the disk, and then use that number. 10. edit the file /usr/local/news/etc/inn.conf and edit the following lines: organization = (put the name of your organization here) server = news.example.com pathhost = news.example.com moderatormailer = news@example.com domain = example.com fromhost = news.example.com storageapi = true (not all of these values are right next to each other - you may have to search for them) (You can leave all the rest of the values at their defaults) 11. edit the file /usr/local/news/etc/newsfeeds Find the ME\ line and replace it with this: ME\ :*,!junk,!local*,\ /world,usa,na,gnu,bionet,pubnet,u3b,eunet,vmsnet,inet,ddn,k12\ :: Find the innfeed!:\ line and replace it with this: innfeed!:\ !*,\ :Tc,Wnm*,S30000:/usr/local/news/bin/startinnfeed Add the following to the end of the file: news.upstream.com/news.upstream.com\ :\ :Tm:innfeed! Make sure this line exists, or is uncommented if it already exists: overview!:*:Tc,Ao,WhR,S30000:/usr/local/news/bin/overchan Note: If you wish to customize these rather generic values I have for ME\ and innfeed!:\, please read the man page for newsfeeds (there is one - man newsfeeds) 12. edit the file /usr/local/news/etc/motd.news (place whatever you would like in your news server message of the day) 13. edit the file /usr/local/news/etc/incoming.conf Add these lines: peer verio { hostname: "news.upstream.com, 20.20.20.20" } 14. edit the file /usr/local/news/etc/innfeed.conf Add these lines: peer verio { ip-name: news.upstream.com } 15. edit the file /usr/local/news/etc/nnrp.access nnrp.access allows you to specify what hosts and address classes are allowed to talk to your news server. If you would like only a few IPs to be able to read news using your server, then just list those servers like this: 192.168.2.3 192.168.99.34 sdca-dialup.line48.pool2.home.com If there is an entire address class that you would like to allow to use the server, you can enter address classes like this: 192.168.5.0/24 192.168.7.96/27 Or you can mix and match, like this: 192.168.99.34 sdca-dialup.line48.pool2.home.com 192.168.7.96/27 Note: if the news server is currently running, you can add new addresses and address classes to the nnrp.access file without restarting the server. They will take affect immediately after the file is saved. This is not true for most of the configuration files in /usr/local/news/etc 16. edit the file /usr/local/news/etc/cycbuff.conf The cycbuff.conf is where you list all of the spool files that you created in step #9. There are two sets of entries in cycbuff.conf. The first set of entries is a listing of spool files where you define a logical name (alias) for each spool file. The second set of entries is where you define one or more 'meta buffers' (collections of spool files that will work together) So, in our example, we have 7 disks that have spool files on them (that we made in step #9). We define each of these in cycbuff.conf as follows: cycbuff:1:/mnt/da3/1:4000000 cycbuff:2:/mnt/da4/2:4000000 cycbuff:3:/mnt/da4/3:4000000 cycbuff:4:/mnt/da4/4:4000000 cycbuff:5:/mnt/da4/5:4000000 cycbuff:6:/mnt/da4/6:4000000 cycbuff:7:/mnt/da4/7:4000000 In this example, we have given a logical name of '1' to the spool file /mnt/da3/1, and we specified that it has a size of 4000000 kilobytes. (remember what we said about the size of these files - your actual 'dd' commands from step #9 may be with a smaller size, even if you have a 4 gigabyte disk) The next set of entries in cycbuff.conf are the meta buffers. In this example, we are going to dedicate four disks to all news _except_ the alt.binaries heirarchy, and dedicate three disks to _only_ the alt.binaries heirarchy. metacycbuff:NOTBINRY:1,2,3,4 metacycbuff:BINARIES:5,6,7 The logical names 'NOTBINRY' and 'BINARIES' are completely arbitrary - you can name them anything you want - it is in the next step that we will show you how to tell the news server which groups should go in which meta buffer. Note: there is a bit of folklore going around regarding the size of the logical names that you can use for cycbuffs and metacycbuffs in the cycbuff.conf file - apparently there were several buggy versions of INN that misbehaved if these logical names were larger than eight or nine characters. I suspect this is no longer the case, but I have tested the eight character metacycbuff names in the example above and they did work. It is not really an issue for the cycbuff names above, as they are only one character long. I really doubt this bug still exists, but if you want to be safe, make your logical names shorter or equal to the example given here. 17. edit the file /usr/local/news/etc/storage.conf storage.conf is where we specify what groups should be spooled into which meta buffers that we defined (NOTBINRY and BINARIES) in step #16. method cnfs { newsgroups: alt.binaries.* class: 1 size: 0,1000000 options: BINARIES } method cnfs { newsgroups: * class: 2 size: 0,100000 options: NOTBINRY } Both should have a method of 'cnfs' specified, size is in kilobytes, and the options field should have the name of the meta buffer. The class field has a non-zero integer value that specifies in which order these methods should be interpreted. News articles are sifted through these rules like a sieve - if an article goes through the whole list andd matches nothing, it is just dropped and not accepted at all. For instance, in this example, if an article from the alt.binaries heirarchy were larger than 1000000 kilobytes, it would fail the first rule, then move onto the second rule - as far as newsgroups are concerned (*) it passes the first rule, but again, it fails the size test which is even smaller in that rule than in the first rule. The article is not accepted. In this case, the class field is not very important, because of how simple this ruleset is, but there are more complex rulesets where certain articles might be accepted by more than one ruleset, and it is important to make sure they sift through the rulesets in the correct order. 18. For each spool file you created with the 'dd' command in step #9, chown the file to news:news and chmod it to 0664. For example: chown news:news /mnt/da3/1 chmod 0664 /mnt/da3/1 Do this for all of your spool files. 19. Run this command: /usr/local/news/bin/inncheck Make sure to fix any problems that this program reports. 20. Run this command to start your server: /usr/local/news/bin/rc.news If the log reports that there was a problem with initialization, stop the INN programs (innd, inncheck, innwatch, etc.) and run just innd by itself: /usr/local/news/bin/innd then kill it after a while and retry rc.news. After starting it up a few times it should run without complaints. The news server is now running, and should immediately begin collecting news from the upstream news server. At this time you should be able to connect with a news client and read articles. NOTE: Remember that the IP you are running the client on needs to exist in the nnrp.access file. There are certain maintenance activities that must occur on a daily basis to keep the news server running smoothly. These activities are contained in the program /usr/local/news/bin/news.daily The following steps will set this maintenance up: 1. make a shell script called 'news.maintenance' with the following lines in it: /usr/local/news/bin/news.daily /usr/local/news/bin/shrinkfile /var/log/news/news date >> /usr/local/news/news.maintenance.log save this file in /usr/local/news 2. execute the following command: chmod +x /usr/local/news/news.maintenance 3. create the file /usr/local/news/crontab with the following line: 1 1 * * * /usr/local/news/news.maintenance 4. execute the following command: crontab -u news /usr/local/news/crontab Now news.daily will run at 1:00 every day, and will enter a time and date into the file 'news.maintenance.log' when it completes. This is helpful because this log of completion time will let you know how long this daily process is taking - and allow you to make changes if you feel it is taking too long. Appendix A: References (I only list references that I personally have used, so please do look for information sources on your own if you find this list lacking) man pages Almost every file listed in this document, in addition to the commands, have man pages. When you install INN, man pages are added for each configuration file covered by this document (nnrp.acccess, storage.conf, etc.) books Configuration and Capacity Planning for Solaris Servers Brian Wong ISBN: 0133499529 Although this is a book on Solaris servers, there is a chapter that covers capacity planning for news servers. It is a small amount of information, so if you do not have the book, I would not suggest buying it for just this purpose - however if you do have the book, you might check the chapter that deals with Internet Servers - News is one of them. web references http://www.isc.org/products/INN/ This is the official page for the INN package. mailing lists http://www.isc.org/services/public/lists/inn-lists.html Appendix B: Notes We mentioned the 9:1 ratio of spool space to overview space, and although we did not use that ratio in our example (we used a 7:1 ratio which is even better). 9:1 should be fine, given a mixed article set (see the comments in the section Overview vs. Articles). However, if you get the ratio wrong, and you find that your overview (/var filesystem) is about to fill up, you can do the following as an emergency action: 1. kill all of the news processes 2. set a small expiration time in /usr/local/news/etc/expire.ctl It is usually set to /remember/:10 by default - set it to something like /remember/:5 3. run the command: /usr/local/news/bin/news.daily notdaily flags='-v1 -n -N' 4. restart the news processes: /usr/local/news/bin/rc.news This is not a good way of dealing with the problem - it _will_ clean out your overview and give you some more space to work with, but there are some negative effects of this action. Disk space is cheap and plentiful, so if you don't want the headache, set a 7:1 ratio. John Kozubik - john@kozubik.com - http://www.kozubik.com