Teaching Linux Part 1
Recently we got 2 student interns at work. Times have changed since I started working and I decided it would be a good opportunity to pass on some of my knowledge.
I decided to start at zero. In my oppinion this is always a good assumption. Even if you already know something. You are always surprised and learn new things if you read another paper having beginners in mind.
The title of the sessions is “Systems engineering ask me anything” but I prepare myself a bit to fill the gaps. I announced it to the whole development department as well.
In the first session I brought my UNICOMP EnduraPro Keyboard with my laptop. Not because my internal keyboard is broken but to highlight the importance of the keyboard. 95% of the configuration and maintenance on Linux happens on the commandline or in configuration files. A good keyboard and the knowledge of how to use it will speed up things.
So my first advice to anybody starting a career in the computer business is: Learn to touch-type! We don’t have a lot of physical tools. But I feel always sorry how many colleagues are not able to use those few tools properly. I learned it back then with a book and a word processing software. (That was before I heard of Emacs!). Later I got a copy of a DOS touch type trainer. I still have a copy of it in a floppy image ready to use in VirtualBox and friends. But there are others out there:
For sysadmin and programmers I highly recommend a US keyboard layout. On layouts for other languages like german you scratch your head who decided where to place eg brackets and backslash. Don’t worry about speed. There is not much value if you can type 360 keys per minute or 120 words per minute with an error rate of 20%. Speed comes with practice.
Maybe I am wrong but I got the impression that people able to touch type are using keyboard shortcuts more frequently. And this is actually the next area where you can safe time. Every time you move your hand from keyboard to the mouse you loose 1-2 seconds. Not much? Over the course of the day or week this counts up!
These sessions are not only about Linux but as well how to configure your own environment properly. At least the way how it works for me. A few years ago I started to maintain the configuration files in my home directory in git repositories. But more on this later on.
A very nice and powerful feature of Linux and Unix in general is it’s internal documentation. You don’t have to go out to the internet. Even today you might find yourself in a work environment where you can be happy if you have internet. From time to time. So the first most important commands to learn about are:
man pages are manuals you can consult if you already know the name of the program. If not you can use apropos to find man pages to some topics. info brings more details than man. You can even dive into subpages. Here is the sed info as an example:
File: sed.info, Node: Top, Next: Introduction, Up: (dir) sed, a stream editor ******************** This file documents version 4.2.1 of GNU `sed', a stream editor. Copyright (C) 1998, 1999, 2001, 2002, 2003, 2004 Free Software Foundation, Inc. This document is released under the terms of the GNU Free Documentation License as published by the Free Software Foundation; either version 1.1, or (at your option) any later version. You should have received a copy of the GNU Free Documentation License along with GNU `sed'; see the file `COPYING.DOC'. If not, write to the Free Software Foundation, 59 Temple Place - Suite 330, Boston, MA 02110-1301, USA. There are no Cover Texts and no Invariant Sections; this text, along with its equivalent in the printed manual, constitutes the Title Page. * Menu: * Introduction:: Introduction * Invoking sed:: Invocation * sed Programs:: `sed' programs * Examples:: Some sample scripts * Limitations:: Limitations and (non-)limitations of GNU `sed' * Other Resources:: Other resources for learning about `sed' * Reporting Bugs:: Reporting bugs * Extended regexps:: `egrep'-style regular expressions * Concept Index:: A menu with all the topics in this manual. * Command and Option Index:: A menu with all `sed' commands and command-line options. --zz-Info: (sed.info.gz)Top, 70 lines --Top------------------------------------- Welcome to Info version 4.13. Type h for help, m for menu item.
and don’t forget the documentation and examples coming with the packages you have installed:
ls /usr/share/doc/dstat-0.7.0/
AUTHORS cplugins.html dstat-paper.txt README ChangeLog cplugins.txt examples screen.html COPYING dstat.1.html examples.html screen.txt counter-rollovers.html dstat.1.txt examples.txt TODO counter-rollovers.txt dstat-paper.html performance.txt
I find tldr quite useful as well. https://github.com/raylee/tldr is my weapon of choice for using it.
As Linux is part of the unix family some history is a must. I found the timeline on that page quite useful to explain how all evolved from AT&T and BSD Unix and why man pages and commands might be different on your Apple Laptop and your Linux server. A lesson about history would be incomplete without mentioning the unix-history-repo or Richard M. Stallman and the importance of GNU.
If you open a terminal window or start your system in text mode the chances are high that you end up in the bash. Take your time and read the man page for bash! Just a few of the many keyboard shortcuts you will find:
| shortcut | description |
|---|---|
| Ctrl-a | beginning-of-line |
| Ctrl-e | end-of-line |
| Meta-f | forward-word |
| Meta-b | backward-word |
| Ctrl-k | kill text from point to end of the line. |
| Ctrl-y | yank |
| Ctrl-_ | undo |
HINT: Meta is ESC or Alt. If you use ESC you can use one after the other.
Personally I prefer Escape. If you use your shell via a IPMI board over a serial interface this might even be the only way it works. Of course I pointed to my blog about IPMI as well. Especially additional kernel parameters like panic and console.
In remote work the next tool you should learn is either gnu screen or
tmux. Sometimes network connections are not reliable or you have to
run something on a server which will take hours. More time as you are
willing to keep your Laptop up and running. Screen is older and might
be already installed on your server. The most important key combo you
have to remember is Ctrl-a ?. This brings you to the overview of
key bindings. In gnu screen land your mouse scroll wheel might stop
working. Ctrl-a ESC and your page up/down keys will help you. There
are even some search and text selection and copy functions. Plus you
can open multiple windows in that screen. If you detach your session
you can come back later. After you started your laptop again and used
ssh to connect to that server screen -r will bring you right back into
your session. Take some time and read the man page. There are more
command line parameters. You can even share a session with some remote
colleague if you have access to the same user or one of you has root
access to that server. There is no need for bandwith-hogs like
Teamviewer in Linux Land!
On the one hand there is so much more interesting technology out there today but on the other hand it can be quite overwhelming for a rookie. It takes years of learning to master all those tools. Don’t try to learn it all in a month. The most important lesson nowadays if you don’t want to specialize around just one topic: You cannot know everything. Keep notes! Lot’s of them! create your own text based wiki. Nurse it like a treasure. Use git and store it on multiple systems. Websites vanish. You might be without internet when you need those informations. Besides: It helps if you write down the information with your own words. Maybe you have spend a week to understand a piece of software and how to configure it properly. Wouldn’t it be a shame if you have to spend the same time again in a month or a year? I have 2 git repositories for notes: One for work one for myself. From time to time I spend a bit of time to move the information into my personal notes and remove anything related to work. IP addresses, hostnames, ... anything revealing my employer. I haven’t touched Oracle in the last 3 years. If somebody asks me tomorrow if I can configure a Oracle RAC I can pull out my notes. My personal notes files has currently 13752 lines! Of course stored in orgmode. My bash profile contains this snippet:
sn() { if [ -z $1 ]; then echo -e "Please provide a search word\n" return 1 fi grep -ni $1 ~/org/notes.org | less } snw() { if [ -z $1 ]; then echo -e "Please provide a search word\n" return 1 fi grep -Hni $1 ~/workorg/work*.org | less }
A few days ago the question about slow package installs on Ubuntu came up. The answer was just a fingertip away for me:
. ~/.sh-funcs.sh
sn apt-get
5500: /var/log/foo/apt-get-autoremove.log 5739: /var/log/foo/apt-get-autoremove.log 5746: /var/log/foo/apt-get-autoremove.log.1.gz 5750: /var/log/foo/apt-get-autoremove.log.2.gz 10056: sudo apt-get install gnupg pbuilder ubuntu-dev-tools bzr-builddeb apt-file 10069: sudo apt-get install build-essential dh-make bzr-builddeb 10163:| yum check-update | apt-get update; apt-get -s upgrade | 10168:=apt-get -o Acquire::http::Dl-Limit=500000 install= 10198:You have to run =apt-get update= before this configuration kicks in! 10211: apt-get update 10212: apt-get install libc:i386 libncurses5:i386 libstdc++6:i386 10220:=sudo apt-get install debian-keyring debian-archive-keyring= 11635: apt-get install libdbd-odbc-perl unixodbc
Of course if you can touch-type taking notes is not a burdon anymore. orgmode is probably the best choice because you can do literate devops. One of the students used the recent discount and bought Mastering Emacs. A good decision and well spend money. Uppon request I showed examples of magit and my way of working on Jira tickets and keeping notes in Emacs.
While walking through my bash profile we came across finger as I have as a snippet for using finger to get weather:
finger hamburgNOSPAMgraph.no
[graph.no] Trying 178.255.144.27... -= Meteogram for germany/hamburg/hamburg =- 'C Rain 15 14 13 12============ 11 === ^^^ 10 ===^^^^^^^^^ ^^^ 9 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 8 7 6 | | | | | | 1 mm _13_14_15_16_17_18 19 20 21 22 23 00 01 02 03 04 05 06 07_08_09_10 Hour SW SW W NW NW NW NW NW NW NW NW NW NW NW NW W NW NW NW NW NW NW Wind dir. 2 2 1 1 3 3 3 4 5 5 5 5 5 5 5 5 5 6 6 6 6 7 Wind(mps) Legend left axis: - Sunny ^ Scattered = Clouded =V= Thunder # Fog Legend right axis: | Rain ! Sleet * Snow [Vote: Add "feels like" temperature as default or option?]
One question which came up was about rsync and how to use it. Here are some examples straight from tldr:
odin:~ baron$ tldr rsync rsync Transfer files either to or from a remote host (not between two remote hosts). Can transfer single files, or multiple files matching a pattern. - Transfer file from local to remote host: rsync path/to/file remote_host_name:remote_host_location - Transfer file from remote host to local: rsync remote_host_name:remote_file_location local_file_location - Transfer file in archive (to preserve attributes) and compressed (zipped) mode: rsync -az path/to/file remote_host_name:remote_host_location - Transfer a directory and all its children from a remote to local: rsync -r remote_host_name:remote_folder_location local_folder_location - Transfer only updated files from remote host: rsync -ru remote_host_name:remote_folder_location local_folder_location - Transfer file over SSH and show progress per file: rsync -e ssh --progress remote_host_name:remote_file local_file - Transfer file over SSH and show global progress: rsync -e ssh --info=progress2 remote_host_name:remote_file local_file odin:~ baron$
Some solutions build on top of rsync are:
Talking about backups: etckeeper saved me more than once. It keeps a
git repository of your /etc/ directory. Every time a package gets
updated or installed it checks for a clean git and commits modified
files.
The last topic I am going to tackle in this post is ssh. Since 1995 everybody is using it. There is even a complete book from OReilly dedicated to that software.
Some nice defaults for your configuration file:
Host * ForwardAgent yes ForwardX11 yes ForwardX11Timeout 596h ServerAliveInterval 30 ControlMaster auto ControlPath ~/.ssh/sockets/ssh.%C ControlPersist 600
The idea of the Master: You open 1 tcp connection to the server and every additional ssh connection uses that same tcp connection. No additional handshakes necessary. This speeds the process up. One downside: environment variables are used from the first session. Aside from the configuration snippet let me refer to the tldr pages:
odin:~ baron$ tldr ssh ssh Secure Shell is a protocol used to securely log onto remote systems. It can be used for logging or executing commands on a remote server. - Connect to a remote server: ssh username@remote_host - Connect to a remote server with a specific identity (private key): ssh -i path/to/key_file username@remote_host - Connect to a remote server using a specific port: ssh username@remote_host -p 2222 - Run a command on a remote server: ssh remote_host command -with -flags - SSH tunneling: Dynamic port forwarding (SOCKS proxy on localhost:9999): ssh -D 9999 -C username@remote_host - SSH tunneling: Forward a specific port (localhost:9999 to slashdot.org:80) along with disabling pseudo-[t]ty allocation and executio[n] of remote commands: ssh -L 9999:slashdot.org:80 -N -T username@remote_host - SSH jumping: Connect through a jumphost to a remote server (Multiple jump hops may be specified separated by comma characters): ssh -J username@jump_host username@remote_host - Agent forwarding: Forward the authentication information to the remote machine (see `man ssh_config` for available options): ssh -A username@remote_host odin:~ baron$
and ssh-keygen:
odin:~ baron$ tldr ssh-keygen ssh-keygen Generate ssh keys user for authentication, password-less logins, and other things. - Generate a key interactively: ssh-keygen - Specify file in which to save the key: ssh-keygen -f ~/.ssh/filename - Generate an ed25519 key with 100 key derivation function rounds: ssh-keygen -t ed25519 -a 100 - Generate an RSA 4096 bit key with your email as a comment: ssh-keygen -t rsa -b 4096 -C "email" - Retrieve the key fingerprint from a host (useful for confirming the authenticity of the host when first connecting to it via SSH): ssh-keygen -l -F remote_host - Retrieve the fingerprint of a key in MD5 Hex: ssh-keygen -l -E md5 -f ~/.ssh/filename - Change the password of a key: ssh-keygen -p -f ~/.ssh/filename odin:~ baron$
That’s all for today.
